How to Choose Between Native and Prompt-Injected Tool Calling Modes in Forge

Use native mode for FC-enabled backends like llama-server with function-calling templates, prompt-injected mode for generic chat APIs, and auto mode to let Forge probe and select automatically.

Choosing between native and prompt-injected tool calling modes determines how your LLM backend processes function calls. The Forge library supports three distinct strategies that balance performance against compatibility. Understanding when to use each mode ensures your workflows run efficiently across different server configurations.

Understanding the Three Tool-Calling Modes in Forge

Native Mode

Native mode transmits the tools field directly to the backend via the OpenAI-style function-calling API. In src/forge/clients/llamafile.py, the _send_native method (lines 64-115) merges consecutive messages and posts the structured request to /chat/completions. This approach requires a backend compiled with FC (function calling) Jinja templates, such as llama-server with function-calling support.

When the backend receives the request, it parses the tools list natively and returns a structured tool_calls array. This eliminates parsing overhead and reduces token consumption since the model uses a dedicated function-calling channel rather than text generation.

Prompt-Injected Mode

When compatibility matters more than efficiency, prompt-injected mode embeds tool descriptions directly into the system prompt. The _send_prompt method (lines 122-165) calls _downgrade_messages to convert tool-role messages into plain user messages, then prepends a tool description generated by build_tool_prompt from src/forge/prompts/templates.py.

After generation, Forge extracts JSON tool calls using extract_tool_call. This mode works with any chat-compatible backend—even those that reject the tools parameter entirely—but relies on the model correctly formatting its output, making it less reliable than native mode.

Auto Mode (Default)

Auto mode provides intelligent fallback behavior without manual configuration. Upon initialization in LlamafileClient.__init__ (lines 34-45), setting mode="auto" stores the request but leaves self.resolved_mode as None. The _resolve_and_send method (lines 42-61) attempts native transmission first; encountering any httpx.HTTPStatusError or BackendError triggers an immediate switch to prompt-injected mode while caching the result in self.resolved_mode.

Subsequent calls skip the probe and use the resolved mode directly, eliminating overhead after the first request.

Implementation Details and File Structure

The decision logic resides primarily in src/forge/clients/llamafile.py. The constructor accepts the mode parameter and initializes self.resolved_mode for tracking. Public methods send and send_stream dispatch requests based on the resolved state, while private methods handle the specific implementation details for each strategy.

For proxy deployments, ProxyServer in src/forge/proxy/proxy.py defaults to mode="native" during _async_start (lines 60-74), ensuring maximum performance when the backend supports it.

Code Examples for Each Mode

Explicit Native Mode

Use this configuration when your backend supports OpenAI-style function calling:

from forge.clients.llamafile import LlamafileClient
from forge.core.workflow import ToolSpec

client = LlamafileClient(
    gguf_path="mistral-7b.gguf",
    base_url="http://localhost:8080/v1",
    mode="native",                     # force native

)

tools = [ToolSpec(name="search", description="Search the web", parameters={})]

response = await client.send(
    messages=[{"role": "user", "content": "Find the capital of France"}],
    tools=tools,
)

print(response)

Prompt-Injected Mode

Deploy this when working with generic backends that lack FC support:

client = LlamafileClient(
    gguf_path="mistral-7b.gguf",
    mode="prompt",                     # force prompt injection

)

response = await client.send(
    messages=[{"role": "user", "content": "List the top 3 headlines"}],
    tools=tools,
)

print(response)

Auto Mode

Let Forge detect capabilities automatically:

client = LlamafileClient(
    gguf_path="mistral-7b.gguf",
    mode="auto",                       # default; resolves on first call

)

response = await client.send(
    messages=[{"role": "user", "content": "Translate 'hello' to French"}],
    tools=None,
)

print("Resolved mode:", client.resolved_mode)

Proxy Server Configuration

The proxy defaults to native with automatic fallback:

from forge.proxy.proxy import ProxyServer

proxy = ProxyServer(
    backend="llamaserver",
    gguf="mistral-7b.gguf",
    backend_port=8080,
)

proxy.start()
print("Proxy URL:", proxy.url)
proxy.stop()

Performance and Compatibility Trade-offs

Native mode delivers the best performance by leveraging dedicated function-calling endpoints, avoiding prompt token overhead and ensuring structured output parsing. However, it fails with HTTP errors against backends lacking FC support.

Prompt-injected mode maximizes compatibility across Llama-File instances and generic servers but increases token usage and introduces parsing uncertainty. The model must generate valid JSON within its text output, which may require more prompt engineering.

Auto mode balances these concerns by probing the server once, incurring minimal latency overhead on the first call while guaranteeing operation across diverse backend configurations.

Summary

• Native mode transmits the tools parameter directly to FC-enabled backends via _send_native, offering optimal performance when using llama-server with function-calling templates.
• Prompt-injected mode embeds tool descriptions into user messages via _send_prompt and build_tool_prompt, ensuring compatibility with any chat API but requiring robust output parsing.
• Auto mode automatically selects the appropriate strategy through _resolve_and_send, caching the result in resolved_mode after the first probe.
• Configure the mode during LlamafileClient initialization, with ProxyServer defaulting to native for performance-first deployments.

Frequently Asked Questions

What happens if I use native mode with a backend that doesn't support function calling?

The request fails with an HTTP 400 or 500 error raised by httpx.HTTPStatusError or Forge's BackendError. When using mode="auto", Forge catches these exceptions in _resolve_and_send and automatically retries with prompt-injected mode.

How does auto mode affect latency on the first request?

The first request incurs minimal overhead as Forge attempts native transmission, detects the failure, and falls back to prompt injection. Subsequent calls use the cached resolved_mode value, eliminating any probe latency.

Can I switch modes after initializing the client?

While the mode parameter is set during initialization, auto mode allows dynamic resolution. You can check client.resolved_mode to see which strategy is active, but manually toggling between native and prompt requires reinstantiating the LlamafileClient.

Where does Forge store the tool description templates for prompt injection?

The helper functions build_tool_prompt and extract_tool_call reside in src/forge/prompts/templates.py. These utilities generate the textual tool descriptions and parse the JSON responses when operating in prompt-injected mode.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}