# How to Choose Between Native and Prompt-Injected Tool Calling Modes in Forge

> Choosing between native and prompt-injected tool calling modes in Forge? Learn how native mode suits FC-enabled backends and prompt-injected mode works with generic APIs.

- Repository: [Antoine/forge](https://github.com/antoinezambelli/forge)
- Tags: best-practices
- Published: 2026-05-22

---

**Use native mode for FC-enabled backends like llama-server with function-calling templates, prompt-injected mode for generic chat APIs, and auto mode to let Forge probe and select automatically.**

Choosing between native and prompt-injected tool calling modes determines how your LLM backend processes function calls. The Forge library supports three distinct strategies that balance performance against compatibility. Understanding when to use each mode ensures your workflows run efficiently across different server configurations.

## Understanding the Three Tool-Calling Modes in Forge

### Native Mode

Native mode transmits the **`tools`** field directly to the backend via the OpenAI-style function-calling API. In [`src/forge/clients/llamafile.py`](https://github.com/antoinezambelli/forge/blob/main/src/forge/clients/llamafile.py), the `_send_native` method (lines 64-115) merges consecutive messages and posts the structured request to `/chat/completions`. This approach requires a backend compiled with FC (function calling) Jinja templates, such as `llama-server` with function-calling support.

When the backend receives the request, it parses the tools list natively and returns a structured `tool_calls` array. This eliminates parsing overhead and reduces token consumption since the model uses a dedicated function-calling channel rather than text generation.

### Prompt-Injected Mode

When compatibility matters more than efficiency, **prompt-injected** mode embeds tool descriptions directly into the system prompt. The `_send_prompt` method (lines 122-165) calls `_downgrade_messages` to convert tool-role messages into plain user messages, then prepends a tool description generated by `build_tool_prompt` from [`src/forge/prompts/templates.py`](https://github.com/antoinezambelli/forge/blob/main/src/forge/prompts/templates.py).

After generation, Forge extracts JSON tool calls using `extract_tool_call`. This mode works with any chat-compatible backend—even those that reject the `tools` parameter entirely—but relies on the model correctly formatting its output, making it less reliable than native mode.

### Auto Mode (Default)

**Auto** mode provides intelligent fallback behavior without manual configuration. Upon initialization in `LlamafileClient.__init__` (lines 34-45), setting `mode="auto"` stores the request but leaves `self.resolved_mode` as `None`. The `_resolve_and_send` method (lines 42-61) attempts native transmission first; encountering any `httpx.HTTPStatusError` or `BackendError` triggers an immediate switch to prompt-injected mode while caching the result in `self.resolved_mode`.

Subsequent calls skip the probe and use the resolved mode directly, eliminating overhead after the first request.

## Implementation Details and File Structure

The decision logic resides primarily in [`src/forge/clients/llamafile.py`](https://github.com/antoinezambelli/forge/blob/main/src/forge/clients/llamafile.py). The constructor accepts the `mode` parameter and initializes `self.resolved_mode` for tracking. Public methods `send` and `send_stream` dispatch requests based on the resolved state, while private methods handle the specific implementation details for each strategy.

For proxy deployments, `ProxyServer` in [`src/forge/proxy/proxy.py`](https://github.com/antoinezambelli/forge/blob/main/src/forge/proxy/proxy.py) defaults to `mode="native"` during `_async_start` (lines 60-74), ensuring maximum performance when the backend supports it.

## Code Examples for Each Mode

### Explicit Native Mode

Use this configuration when your backend supports OpenAI-style function calling:

```python
from forge.clients.llamafile import LlamafileClient
from forge.core.workflow import ToolSpec

client = LlamafileClient(
    gguf_path="mistral-7b.gguf",
    base_url="http://localhost:8080/v1",
    mode="native",                     # force native

)

tools = [ToolSpec(name="search", description="Search the web", parameters={})]

response = await client.send(
    messages=[{"role": "user", "content": "Find the capital of France"}],
    tools=tools,
)

print(response)

```

### Prompt-Injected Mode

Deploy this when working with generic backends that lack FC support:

```python
client = LlamafileClient(
    gguf_path="mistral-7b.gguf",
    mode="prompt",                     # force prompt injection

)

response = await client.send(
    messages=[{"role": "user", "content": "List the top 3 headlines"}],
    tools=tools,
)

print(response)

```

### Auto Mode

Let Forge detect capabilities automatically:

```python
client = LlamafileClient(
    gguf_path="mistral-7b.gguf",
    mode="auto",                       # default; resolves on first call

)

response = await client.send(
    messages=[{"role": "user", "content": "Translate 'hello' to French"}],
    tools=None,
)

print("Resolved mode:", client.resolved_mode)

```

### Proxy Server Configuration

The proxy defaults to native with automatic fallback:

```python
from forge.proxy.proxy import ProxyServer

proxy = ProxyServer(
    backend="llamaserver",
    gguf="mistral-7b.gguf",
    backend_port=8080,
)

proxy.start()
print("Proxy URL:", proxy.url)
proxy.stop()

```

## Performance and Compatibility Trade-offs

**Native mode** delivers the best performance by leveraging dedicated function-calling endpoints, avoiding prompt token overhead and ensuring structured output parsing. However, it fails with HTTP errors against backends lacking FC support.

**Prompt-injected mode** maximizes compatibility across Llama-File instances and generic servers but increases token usage and introduces parsing uncertainty. The model must generate valid JSON within its text output, which may require more prompt engineering.

**Auto mode** balances these concerns by probing the server once, incurring minimal latency overhead on the first call while guaranteeing operation across diverse backend configurations.

## Summary

- **Native mode** transmits the `tools` parameter directly to FC-enabled backends via `_send_native`, offering optimal performance when using `llama-server` with function-calling templates.
- **Prompt-injected mode** embeds tool descriptions into user messages via `_send_prompt` and `build_tool_prompt`, ensuring compatibility with any chat API but requiring robust output parsing.
- **Auto mode** automatically selects the appropriate strategy through `_resolve_and_send`, caching the result in `resolved_mode` after the first probe.
- Configure the mode during `LlamafileClient` initialization, with `ProxyServer` defaulting to native for performance-first deployments.

## Frequently Asked Questions

### What happens if I use native mode with a backend that doesn't support function calling?

The request fails with an HTTP 400 or 500 error raised by `httpx.HTTPStatusError` or Forge's `BackendError`. When using `mode="auto"`, Forge catches these exceptions in `_resolve_and_send` and automatically retries with prompt-injected mode.

### How does auto mode affect latency on the first request?

The first request incurs minimal overhead as Forge attempts native transmission, detects the failure, and falls back to prompt injection. Subsequent calls use the cached `resolved_mode` value, eliminating any probe latency.

### Can I switch modes after initializing the client?

While the `mode` parameter is set during initialization, `auto` mode allows dynamic resolution. You can check `client.resolved_mode` to see which strategy is active, but manually toggling between `native` and `prompt` requires reinstantiating the `LlamafileClient`.

### Where does Forge store the tool description templates for prompt injection?

The helper functions `build_tool_prompt` and `extract_tool_call` reside in [`src/forge/prompts/templates.py`](https://github.com/antoinezambelli/forge/blob/main/src/forge/prompts/templates.py). These utilities generate the textual tool descriptions and parse the JSON responses when operating in prompt-injected mode.