# How the Synthetic Respond Tool Improves Small Model Accuracy in Forge

> Learn how the synthetic respond tool boosts small model accuracy in Forge by forcing tool-calling mode, eliminating ambiguity, and ensuring response validation.

- Repository: [Antoine/forge](https://github.com/antoinezambelli/forge)
- Tags: deep-dive
- Published: 2026-05-22

---

**The synthetic `respond` tool forces small language models to always use tool-calling mode, eliminating ambiguous text output and ensuring Forge's guard-rails validate every response.**

Small local models with approximately 8 billion parameters often struggle to decide between emitting plain text and invoking tools. The **synthetic respond tool** solves this by converting all potential text responses into structured tool calls, allowing Forge's validation and retry mechanisms to protect even low-capacity models. This approach is implemented in the `antoinezambelli/forge` repository to dramatically improve completion rates.

## Why Small Models Fail at Text vs. Tool Decisions

Local models around 8B parameters frequently *mis-choose* between generating plain text and issuing tool calls. When a model should invoke a tool like `get_weather` but returns free-form text instead, Forge's guard-rail stack treats the response as a failure.

This triggers a retry loop that wastes time on additional inference passes and drops the overall completion rate. Evaluations show rates between 4% and 100% completion without the fix, depending on the model's size and the complexity of the task. The root cause is that small models cannot reliably distinguish between conversational text and functional tool calls, causing them to exit tool-calling mode prematurely and bypass validation.

## How the Synthetic Respond Tool Works

Forge solves this problem by **injecting a synthetic `respond` tool** into every request that contains tools, as defined in [`src/forge/tools/respond.py`](https://github.com/antoinezambelli/forge/blob/main/src/forge/tools/respond.py). Instead of allowing the model to emit raw text, the system requires the model to call `respond(message="...")`, keeping it inside validated tool-calling mode.

### Forced Tool-Calling Mode

By converting text generation into a tool invocation, the synthetic `respond` tool ensures the model never accidentally falls back to plain text. This provides three critical benefits:

- **Full validation coverage**: Schema validation and rescue parsing apply to every output, not just explicit tool calls
- **Active retry mechanisms**: Failed calls trigger automatic retries rather than silent failures or ambiguous finish reasons
- **Deterministic output structure**: The model cannot bypass the guard-rail stack by emitting conversational text

### Proxy Injection and Response Stripping

The system handles this transparently through two mechanisms controlled by [`src/forge/proxy/handler.py`](https://github.com/antoinezambelli/forge/blob/main/src/forge/proxy/handler.py). First, the **automatic injection** step adds the `respond` definition to the list of available tools unless the client already provides one, as documented in [`README.md`](https://github.com/antoinezambelli/forge/blob/main/README.md) at lines 1548-1556.

Second, after the model executes the `respond` call, the proxy **strips the synthetic tool frame** and returns normal text to the client. This preserves OpenAI-compatible API behavior while ensuring the model's output passed through validation during generation.

This design replaced the earlier "trust-text-intent" flag approach documented in **ADR-013** ([`docs/decisions/013-text-response-intent.md`](https://github.com/antoinezambelli/forge/blob/main/docs/decisions/013-text-response-intent.md)), which proved unreliable for small models because trusting the model's finish reason led to undetected errors.

## Implementation Examples

### Defining a Workflow with the Synthetic Tool

When building workflows manually using the native API, you must explicitly include the synthetic tool:

```python
from pydantic import BaseModel, Field
from forge import Workflow, ToolDef, ToolSpec, WorkflowRunner, LlamafileClient
from forge.tools import respond_tool, RESPOND_TOOL_NAME

# Example tool – gets the weather

def get_weather(city: str) -> str:
    return f"72°F and sunny in {city}"

class GetWeatherParams(BaseModel):
    city: str = Field(description="City name")

workflow = Workflow(
    name="weather",
    description="Look up weather for a city.",
    tools={
        "get_weather": ToolDef(
            spec=ToolSpec(
                name="get_weather",
                description="Get current weather",
                parameters=GetWeatherParams,
            ),
            callable=get_weather,
        ),
        RESPOND_TOOL_NAME: respond_tool(),          # inject synthetic respond

    },
    terminal_tool="respond",                        # end with a text reply

    system_prompt_template="You are a helpful assistant.",
)

# Run the workflow

client = LlamafileClient(gguf_path="path/to/model.gguf", mode="native")
runner = WorkflowRunner(client=client)
await runner.run(workflow, "What's the weather in Paris?")

```

The model calls `respond(message="...")` once the weather is retrieved, ensuring the guard-rail stack stays active throughout the interaction.

### Automatic Proxy Mode

For simpler integration, the proxy server handles injection automatically without code changes:

```bash
python -m forge.proxy --backend-url http://localhost:8080 --port 8081

```

The proxy reads incoming requests, adds the synthetic `respond` definition to the tool list, and later strips the tool call from outgoing payloads. The client receives plain text while the model enjoyed full validation during generation.

### Response Transformation Logic

Internally, the proxy in [`src/forge/proxy/handler.py`](https://github.com/antoinezambelli/forge/blob/main/src/forge/proxy/handler.py) performs transformation similar to this simplified logic:

```python
def _strip_respond_calls(messages):
    # Remove tool calls named "respond" and replace with TextResponse

    filtered = []
    for msg in messages:
        if msg.role == "assistant" and msg.tool_calls:
            for tc in msg.tool_calls:
                if tc.tool == "respond":
                    filtered.append(TextResponse(content=tc.args["message"]))
                else:
                    filtered.append(msg)   # keep real tool calls

        else:
            filtered.append(msg)
    return filtered

```

This ensures downstream clients receive standard text responses while every model output passed through Forge's validation stack.

## Accuracy Improvements

The **synthetic respond tool** eliminates the ambiguity that causes small models to fail. Without the tool, models choose incorrectly between text and tools, requiring retry loops that reduce completion rates to as low as 4% in some evaluations.

With the synthetic tool, models are forced to use a tool for every response. This ensures the guard-rail stack validates the output and raises completion rates dramatically. The single-pass approach also eliminates extra inference passes that add latency, as the proxy simply rewrites the response format without requiring the model to regenerate content.

## Summary

- **Small models** (≈8B parameters) struggle to distinguish between text generation and tool invocation, causing completion rates as low as 4%.
- The **synthetic `respond` tool** converts all text output into tool calls, keeping models in validated tool-calling mode where guard-rails apply.
- **Proxy injection** automatically adds the tool to requests in [`src/forge/proxy/handler.py`](https://github.com/antoinezambelli/forge/blob/main/src/forge/proxy/handler.py), while **response stripping** preserves OpenAI API compatibility.
- The implementation spans [`src/forge/tools/respond.py`](https://github.com/antoinezambelli/forge/blob/main/src/forge/tools/respond.py) and is documented in [`docs/decisions/013-text-response-intent.md`](https://github.com/antoinezambelli/forge/blob/main/docs/decisions/013-text-response-intent.md) as ADR-013.

## Frequently Asked Questions

### What makes small models different from large models in tool calling?

Small models with approximately 8 billion parameters lack the capacity to reliably interpret finish reasons and distinguish between conversational text and functional tool calls. While large models can correctly signal when they intend to return text versus call a tool, small models frequently emit text when they should invoke tools. This causes validation failures in [`src/forge/proxy/handler.py`](https://github.com/antoinezambelli/forge/blob/main/src/forge/proxy/handler.py) that trigger costly retry loops, degrading completion rates.

### How does the synthetic respond tool affect API compatibility?

The synthetic tool does not break API compatibility. According to the implementation in [`src/forge/proxy/handler.py`](https://github.com/antoinezambelli/forge/blob/main/src/forge/proxy/handler.py), the proxy server strips the `respond` tool frame after execution and converts it back to plain text before sending the response to the client. This maintains OpenAI-compatible API behavior while ensuring the model's output passed through Forge's validation stack during generation.

### Can I disable the synthetic respond tool in Forge?

Forge replaced the "trust-text-intent" flag with the mandatory synthetic `respond` approach documented in ADR-013 ([`docs/decisions/013-text-response-intent.md`](https://github.com/antoinezambelli/forge/blob/main/docs/decisions/013-text-response-intent.md)), because trusting the model's finish reason proved unreliable for small models. While you can define your own workflows without the tool when using the native Python API directly, the proxy mode automatically injects the tool unless explicitly overridden, as the design prioritizes reliability over the minor overhead of tool serialization.

### What performance impact does the synthetic tool have?

The synthetic `respond` tool improves performance by eliminating retry loops. Without it, small models might require multiple inference passes when they incorrectly emit text instead of tools, adding significant latency. With the synthetic tool, the model completes the task in a single pass because it cannot accidentally exit tool-calling mode, and the proxy simply rewrites the response format without additional model inference.