How to Configure Backend-Specific Native Function Calling in Forge

Configure backend-specific native function calling in Forge by setting mode="native" when initializing Ollama or Llamafile clients, ensuring Llamafile servers start with the --jinja flag, and passing ToolSpec objects to the send() method.

Forge simplifies structured tool usage by supporting native function calling on backends that expose a tools API. This approach lets models return structured JSON tool calls rather than parsing plain text. The framework currently supports native function calling on Ollama and Llamafile backends, while falling back to prompt-injected techniques for other servers.

Supported Backends and Native Capabilities

Forge distinguishes between backends that support native function calling and those that rely on prompt engineering.

Native-capable backends expose a formal tools field in their API:

• Ollama: Uses the /api/chat endpoint with a tools array; no special server flags required.
• Llamafile: Requires the --jinja flag to enable the native template that understands tool schemas.

Prompt-only backends:

• Llama-Server (default): Does not support native function calling; Forge automatically injects tool descriptions into the system prompt when using mode="prompt" or mode="auto".

In src/forge/server.py, the ServerManager class handles backend initialization, while specific client implementations in src/forge/clients/ollama.py and src/forge/clients/llamafile.py manage the native request formatting.

Step-by-Step Configuration Workflow

Configuring native function calling follows a six-stage pipeline from server initialization to response parsing.

1. Select the backend via ServerManager Instantiate the manager with your target backend. In src/forge/server.py, the ServerManager.__init__ method accepts backend="ollama" or backend="llamafile".

2. Start the server in native mode For Llamafile, the ServerManager.start method automatically appends --jinja to the startup flags when mode="native" is specified. Ollama servers do not require additional flags for native support.

3. Initialize the client with mode configuration Create an OllamaClient or LlamafileClient with mode="native". In src/forge/clients/ollama.py, the send method checks this mode to determine whether to include the tools field in the request body. Similarly, LlamafileClient._send_native in src/forge/clients/llamafile.py constructs the native payload.

4. Define tool specifications Create ToolSpec objects describing your functions. The format_tool helper in src/forge/clients/base.py serializes these into the JSON schema required by the backend.

5. Execute the request Pass your tools to client.send(). The client transmits the request and parses the backend's structured response into ToolCall objects (or TextResponse if no tool is invoked).

6. Handle auto-fallback (optional) When using mode="auto", LlamafileClient._resolve_and_send probes the server with a native request first. If the backend returns a 400 error indicating tool calling is unsupported, the client automatically retries in prompt-injected mode.

Backend-Specific Configuration Requirements

Each backend requires distinct configuration to enable native function calling.

Ollama Configuration

Ollama supports native tools without server-side flags. Configure the client explicitly:

from forge.clients.ollama import OllamaClient
from forge.core.workflow import ToolSpec

client = OllamaClient(
    model="mistral",
    base_url="http://localhost:11434",
    mode="native",  # Explicit native mode

    think=True,     # Optional: enable reasoning blocks

)

The OllamaClient.send method (lines 50-92 in src/forge/clients/ollama.py) constructs the request with a tools array and parses the tool_calls field from the JSON response.

Llamafile Configuration

Llamafile requires the --jinja flag for native template support. Use ServerManager to ensure correct startup:

from forge.server import ServerManager

sm = ServerManager(backend="llamafile", port=8080)
await sm.start(
    model="llama3",
    gguf_path="/models/llama3.gguf",
    mode="native",  # Enables --jinja flag automatically

)

The ServerManager.start method in src/forge/server.py (lines 72-78) checks for mode="native" and appends --jinja to the command arguments. Without this flag, the server cannot process the tools field.

Auto-Fallback Configuration

For deployment flexibility, use mode="auto" to let the client detect capabilities:

from forge.clients.llamafile import LlamafileClient

client = LlamafileClient(
    gguf_path="/models/llama3.gguf",
    mode="auto",  # Try native first, fall back to prompt

)

response = await client.send(messages=[...], tools=[...])
print("Resolved mode:", client.resolved_mode)  # "native" or "prompt"

As implemented in src/forge/clients/llamafile.py (lines 84-89), this mode catches HTTP errors indicating unsupported native features and transparently switches to prompt-injected tool calling.

Implementing Tool Specifications

Native function calling requires structured tool definitions using the ToolSpec class from src/forge/core/workflow.py:

from forge.core.workflow import ToolSpec

search_tool = ToolSpec(
    name="web_search",
    description="Search the internet for current information",
    parameters={
        "type": "object",
        "properties": {
            "query": {
                "type": "string",
                "description": "The search query string"
            }
        },
        "required": ["query"]
    },
)

# Use with any native-enabled client

response = await client.send(
    messages=[{"role": "user", "content": "What's the weather in Paris?"}],
    tools=[search_tool],
)

The format_tool function in src/forge/clients/base.py (lines 71-80) marshals these specifications into the exact JSON schema expected by Ollama and Llamafile APIs.

Handling Reasoning Blocks and Error Recovery

Both native clients support a think parameter for reasoning models. When think=True, the client requests the backend to include chain-of-thought reasoning. If the backend rejects this (detected via _is_think_unsupported_error in src/forge/clients/ollama.py), the client automatically retries without the reasoning flag.

For Ollama, the error detection logic in lines 19-27 of src/forge/clients/ollama.py identifies unsupported thinking errors, triggering the retry mechanism in lines 60-66.

Summary

• Forge supports native function calling on Ollama and Llamafile backends, while Llama-Server requires prompt-injected tools.
• Set mode="native" when creating clients or starting servers to enable structured tool APIs.
• Llamafile requires the --jinja flag, which ServerManager.start automatically adds when mode="native" is specified.
• Use ToolSpec objects to define tool schemas; the format_tool helper ensures correct JSON formatting for each backend.
• mode="auto" enables resilient deployments by probing for native support and falling back to prompt mode if unavailable.

Frequently Asked Questions

What is the difference between native and prompt mode in Forge?

Native mode uses the backend's structured API to send tool schemas and receive parsed JSON tool calls, while prompt mode injects tool descriptions into the system prompt and parses the model's text output. Native mode is more reliable and available on Ollama and Llamafile backends, whereas prompt mode works universally but requires careful output parsing.

Why does Llamafile require the --jinja flag for function calling?

The --jinja flag enables the server's native Jinja template engine, which understands how to format tool instructions and parse tool calls according to the ChatML format. Without this flag, Llamafile operates in standard completion mode and cannot process the tools field in API requests, as implemented in ServerManager.start in src/forge/server.py.

How does the auto-fallback mechanism work when using mode="auto"?

When mode="auto" is set, LlamafileClient._resolve_and_send first attempts a native request with the tools field. If the server responds with a 400 error or message indicating tool calling is unsupported, the client catches this error, switches to mode="prompt", and retries the request with injected tool descriptions, storing the resolved mode in client.resolved_mode.

Can I use the same ToolSpec definitions across different backends?

Yes. The ToolSpec class in src/forge/core/workflow.py provides a backend-agnostic way to define tools. The format_tool helper in src/forge/clients/base.py automatically converts these specifications into the specific JSON schema required by Ollama's /api/chat endpoint or Llamafile's native API, ensuring portability across supported backends.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}