How the Synthetic Respond Tool Improves Small Model Accuracy in Forge

The synthetic respond tool forces small language models to always use tool-calling mode, eliminating ambiguous text output and ensuring Forge's guard-rails validate every response.

Small local models with approximately 8 billion parameters often struggle to decide between emitting plain text and invoking tools. The synthetic respond tool solves this by converting all potential text responses into structured tool calls, allowing Forge's validation and retry mechanisms to protect even low-capacity models. This approach is implemented in the antoinezambelli/forge repository to dramatically improve completion rates.

Why Small Models Fail at Text vs. Tool Decisions

Local models around 8B parameters frequently mis-choose between generating plain text and issuing tool calls. When a model should invoke a tool like get_weather but returns free-form text instead, Forge's guard-rail stack treats the response as a failure.

This triggers a retry loop that wastes time on additional inference passes and drops the overall completion rate. Evaluations show rates between 4% and 100% completion without the fix, depending on the model's size and the complexity of the task. The root cause is that small models cannot reliably distinguish between conversational text and functional tool calls, causing them to exit tool-calling mode prematurely and bypass validation.

How the Synthetic Respond Tool Works

Forge solves this problem by injecting a synthetic respond tool into every request that contains tools, as defined in src/forge/tools/respond.py. Instead of allowing the model to emit raw text, the system requires the model to call respond(message="..."), keeping it inside validated tool-calling mode.

Forced Tool-Calling Mode

By converting text generation into a tool invocation, the synthetic respond tool ensures the model never accidentally falls back to plain text. This provides three critical benefits:

• Full validation coverage: Schema validation and rescue parsing apply to every output, not just explicit tool calls
• Active retry mechanisms: Failed calls trigger automatic retries rather than silent failures or ambiguous finish reasons
• Deterministic output structure: The model cannot bypass the guard-rail stack by emitting conversational text

Proxy Injection and Response Stripping

The system handles this transparently through two mechanisms controlled by src/forge/proxy/handler.py. First, the automatic injection step adds the respond definition to the list of available tools unless the client already provides one, as documented in README.md at lines 1548-1556.

Second, after the model executes the respond call, the proxy strips the synthetic tool frame and returns normal text to the client. This preserves OpenAI-compatible API behavior while ensuring the model's output passed through validation during generation.

This design replaced the earlier "trust-text-intent" flag approach documented in ADR-013 (docs/decisions/013-text-response-intent.md), which proved unreliable for small models because trusting the model's finish reason led to undetected errors.

Implementation Examples

Defining a Workflow with the Synthetic Tool

When building workflows manually using the native API, you must explicitly include the synthetic tool:

from pydantic import BaseModel, Field
from forge import Workflow, ToolDef, ToolSpec, WorkflowRunner, LlamafileClient
from forge.tools import respond_tool, RESPOND_TOOL_NAME

# Example tool – gets the weather

def get_weather(city: str) -> str:
    return f"72°F and sunny in {city}"

class GetWeatherParams(BaseModel):
    city: str = Field(description="City name")

workflow = Workflow(
    name="weather",
    description="Look up weather for a city.",
    tools={
        "get_weather": ToolDef(
            spec=ToolSpec(
                name="get_weather",
                description="Get current weather",
                parameters=GetWeatherParams,
            ),
            callable=get_weather,
        ),
        RESPOND_TOOL_NAME: respond_tool(),          # inject synthetic respond

    },
    terminal_tool="respond",                        # end with a text reply

    system_prompt_template="You are a helpful assistant.",
)

# Run the workflow

client = LlamafileClient(gguf_path="path/to/model.gguf", mode="native")
runner = WorkflowRunner(client=client)
await runner.run(workflow, "What's the weather in Paris?")

The model calls respond(message="...") once the weather is retrieved, ensuring the guard-rail stack stays active throughout the interaction.

Automatic Proxy Mode

For simpler integration, the proxy server handles injection automatically without code changes:

python -m forge.proxy --backend-url http://localhost:8080 --port 8081

The proxy reads incoming requests, adds the synthetic respond definition to the tool list, and later strips the tool call from outgoing payloads. The client receives plain text while the model enjoyed full validation during generation.

Response Transformation Logic

Internally, the proxy in src/forge/proxy/handler.py performs transformation similar to this simplified logic:

def _strip_respond_calls(messages):
    # Remove tool calls named "respond" and replace with TextResponse

    filtered = []
    for msg in messages:
        if msg.role == "assistant" and msg.tool_calls:
            for tc in msg.tool_calls:
                if tc.tool == "respond":
                    filtered.append(TextResponse(content=tc.args["message"]))
                else:
                    filtered.append(msg)   # keep real tool calls

        else:
            filtered.append(msg)
    return filtered

This ensures downstream clients receive standard text responses while every model output passed through Forge's validation stack.

Accuracy Improvements

The synthetic respond tool eliminates the ambiguity that causes small models to fail. Without the tool, models choose incorrectly between text and tools, requiring retry loops that reduce completion rates to as low as 4% in some evaluations.

With the synthetic tool, models are forced to use a tool for every response. This ensures the guard-rail stack validates the output and raises completion rates dramatically. The single-pass approach also eliminates extra inference passes that add latency, as the proxy simply rewrites the response format without requiring the model to regenerate content.

Summary

• Small models (≈8B parameters) struggle to distinguish between text generation and tool invocation, causing completion rates as low as 4%.
• The synthetic respond tool converts all text output into tool calls, keeping models in validated tool-calling mode where guard-rails apply.
• Proxy injection automatically adds the tool to requests in src/forge/proxy/handler.py, while response stripping preserves OpenAI API compatibility.
• The implementation spans src/forge/tools/respond.py and is documented in docs/decisions/013-text-response-intent.md as ADR-013.

Frequently Asked Questions

What makes small models different from large models in tool calling?

Small models with approximately 8 billion parameters lack the capacity to reliably interpret finish reasons and distinguish between conversational text and functional tool calls. While large models can correctly signal when they intend to return text versus call a tool, small models frequently emit text when they should invoke tools. This causes validation failures in src/forge/proxy/handler.py that trigger costly retry loops, degrading completion rates.

How does the synthetic respond tool affect API compatibility?

The synthetic tool does not break API compatibility. According to the implementation in src/forge/proxy/handler.py, the proxy server strips the respond tool frame after execution and converts it back to plain text before sending the response to the client. This maintains OpenAI-compatible API behavior while ensuring the model's output passed through Forge's validation stack during generation.

Can I disable the synthetic respond tool in Forge?

Forge replaced the "trust-text-intent" flag with the mandatory synthetic respond approach documented in ADR-013 (docs/decisions/013-text-response-intent.md), because trusting the model's finish reason proved unreliable for small models. While you can define your own workflows without the tool when using the native Python API directly, the proxy mode automatically injects the tool unless explicitly overridden, as the design prioritizes reliability over the minor overhead of tool serialization.

What performance impact does the synthetic tool have?

The synthetic respond tool improves performance by eliminating retry loops. Without it, small models might require multiple inference passes when they incorrectly emit text instead of tools, adding significant latency. With the synthetic tool, the model completes the task in a single pass because it cannot accidentally exit tool-calling mode, and the proxy simply rewrites the response format without additional model inference.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}