How to Implement Tool Calling with FreeLLMAPI

FreeLLMAPI exposes a fully OpenAI-compatible /v1/chat/completions endpoint that routes tool-calling requests only to models proven capable of emitting structured tool_calls, while automatically rescuing plain-text responses into proper JSON payloads.

FreeLLMAPI is an open-source unified gateway that aggregates free-tier LLM providers behind a single OpenAI-compatible API. When you implement tool calling with FreeLLMAPI, the system automatically detects which models support function calling through a capability probe, filters the routing pool to exclude incompatible providers, and normalizes responses to ensure your client receives standard OpenAI-formatted tool_calls objects.

How the Router Handles Tool-Capable Models

The core routing logic in server/src/services/router.ts determines whether a model can handle tool requests before forwarding traffic. When a request includes a tools array or tool_choice parameter, the router checks the supports_tools flag in the model catalog entry.

At lines 2167-2172 of server/src/services/router.ts, the router implements a strict guard that prevents routing to incompatible models:

// Logic from router.ts filtering non-tool-capable entries
if (requireTools && !entry.supports_tools) {
  // Skip models that cannot emit tool_calls
  continue;
}

This ensures that only providers proven capable of structured function calling receive tool requests. The check occurs alongside other capability filters around lines 2000-2008 of the same file, where the router evaluates requireTools && !entry.supports_tools to abort routing to any model that cannot emit structured tool_calls.

Model Discovery and Capability Probing

Before requests hit the router, FreeLLMAPI runs a discovery process to populate the supports_tools column in the model catalog. In server/src/services/model-discovery.ts (lines 498-560), the system executes a dummy tool-probe against each model using a mock get_weather function.

During this probe phase, the discovery service sends a test request with a tool definition and verifies that the model responds with a properly formatted tool_calls payload rather than plain text. Models that pass this test receive supports_tools = 1 in the database, making them eligible for tool-routing logic.

If a provider returns unstructured text instead of JSON during actual request handling, FreeLLMAPI applies a tool-call rescue mechanism. According to docs/architecture.md (line 55), the system parses the plain-text output and restructures it into a valid OpenAI tool_calls object before returning it to the client.

Step-by-Step Implementation

The end-to-end flow for implementing tool calling follows standard OpenAI SDK patterns. FreeLLMAPI requires no proprietary client libraries.

Configure the Client

Point your OpenAI SDK to the FreeLLMAPI base URL:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:3001/v1",
    api_key="freellmapi-your-unified-key",
)

Define Tools and Send Requests

Construct your tool definitions using standard JSON Schema and include them in the chat completion request:

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Fetch current weather for a city",
        "parameters": {
            "type": "object",
            "properties": {
                "city": {"type": "string", "description": "City name"}
            },
            "required": ["city"]
        }
    }
}]

# Initiate tool calling

resp = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "What's the weather in Paris?"}],
    tools=tools,
    tool_choice="auto",
)

Handle Tool Execution and Follow-Up

Extract the function call arguments, execute your local business logic, and return the result as a tool role message:

choice = resp.choices[0]
if choice.message.tool_calls:
    call = choice.message.tool_calls[0]
    args = call.function.arguments  # JSON string

    result = '{"temp_c": 22, "cond": "cloudy"}'  # Your tool execution

    # Continue conversation with tool result

    follow_up = client.chat.completions.create(
        model="auto",
        messages=[
            {"role": "user", "content": "What's the weather in Paris?"},
            {"role": "assistant", "content": None, "tool_calls": [call]},
            {"role": "tool", "tool_call_id": call.id, "content": result},
        ],
        tools=tools,
        tool_choice="auto",
    )
    print(follow_up.choices[0].message.content)

This pattern matches the API examples documented in docs/api.md (lines 12-14), ensuring compatibility with existing OpenAI integrations.

Type Definitions and Data Structures

The TypeScript interfaces governing tool calling are defined in shared/types.ts (lines 385-409). These definitions ensure type safety across the routing layer and include structures for tool_calls, tool_choice, and the tool message role required for multi-turn conversations.

Key interfaces include:

• ChatCompletionTool: The tool definition schema sent in requests
• ChatCompletionMessageToolCall: The structured response object containing function names and arguments
• ChatCompletionToolMessageParam: The message type used when returning tool execution results

Summary

• FreeLLMAPI exposes standard OpenAI-compatible endpoints for tool calling through /v1/chat/completions.
• The router (server/src/services/router.ts) filters models using the supports_tools flag to ensure only capable providers receive tool requests.
• Model discovery (server/src/services/model-discovery.ts) probes each provider with a dummy get_weather request to set the capability flag.
• Tool-call rescue automatically converts plain-text tool responses into proper JSON payloads when providers deviate from the OpenAI specification.
• Client implementations use standard OpenAI SDKs without modification, treating FreeLLMAPI as a drop-in replacement for the official API.

Frequently Asked Questions

What models support tool calling in FreeLLMAPI?

Models receive the supports_tools capability flag only if they pass the dummy tool probe during the discovery phase in server/src/services/model-discovery.ts. The router automatically excludes models lacking this flag from tool requests, ensuring compatibility by routing only to providers that correctly emit structured tool_calls JSON.

How does FreeLLMAPI handle providers that return plain text instead of JSON tool calls?

According to docs/architecture.md, FreeLLMAPI implements a tool-call rescue mechanism that parses plain-text responses and restructures them into valid OpenAI tool_calls objects. This ensures clients receive consistent JSON payloads even when underlying free-tier providers return malformed or text-based function calls.

Can I force a specific tool choice instead of letting the model decide?

Yes. FreeLLMAPI supports the tool_choice parameter defined in shared/types.ts. You can set tool_choice to "auto" (model decides), "none" (no tools), or a specific object like {"type": "function", "function": {"name": "get_weather"}} to force the model to use a particular tool. The router respects this parameter when selecting from the pool of supports_tools flagged models.

Do I need to modify existing OpenAI SDK code to work with FreeLLMAPI?

No. FreeLLMAPI maintains full OpenAI API compatibility as documented in docs/api.md. You only need to change the base_url to your FreeLLMAPI instance (e.g., http://localhost:3001/v1) and use your FreeLLMAPI key. All tool definitions, message roles, and response handling patterns remain identical to the official OpenAI implementation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}