How Tool Calling Works Across LLM Providers in FreeLLMAPI: A Complete Technical Guide

FreeLLMAPI exposes a single OpenAI-compatible endpoint that normalizes tool calling across 16+ free-tier providers by filtering for tool-capable models, forwarding requests transparently, and converting provider-specific dialects back into standard OpenAI tool_calls format.

FreeLLMAPI serves as a unified gateway that simplifies tool calling functionality across heterogeneous large language model providers. The repository implements a thin translation layer that accepts standard OpenAI tool definitions at /v1/chat/completions and automatically manages provider-specific variations, allowing developers to build function-calling applications without worrying about backend inconsistencies.

Request Validation and Model Selection

When a client sends a request containing tools or tool_choice parameters, FreeLLMAPI executes a strict validation and routing pipeline before reaching any upstream provider.

Type Validation and Schema Enforcement

Incoming requests are validated against the shared type definitions in server/src/shared/types.ts (lines 230-251). This module defines the ChatToolDefinition and ChatToolChoice interfaces that enforce structure across the entire system. The validation ensures that tool schemas match the expected OpenAI format before any provider selection occurs.

Tool-Capable Model Filtering

The router logic in server/src/services/router.ts (lines 887-894) filters the model catalog based on the supports_tools flag. When requireTools evaluates to true—meaning the request contains a non-empty tools array—the system exclusively considers models advertising supports_tools = 1. If no tool-capable models are available, the request fails fast rather than attempting execution on incompatible backends.

Provider Dispatch and Request Forwarding

Once a suitable model is selected, FreeLLMAPI forwards the request without modification to the chosen provider.

Transparent OpenAI-Compatible Proxying

The base provider implementation in server/src/providers/openai-compat.ts receives the request payload unchanged, including the verbatim tools array. This design leverages the fact that most modern providers implement the OpenAI chat specification and natively understand tool_calls syntax. FreeLLMAPI does not transform or mangle tool definitions; it acts as a pass-through proxy for providers that already support the standard.

Legacy Endpoint Support

For clients using the legacy /v1/responses endpoint, the system in server/src/routes/responses.ts converts the Responses-API payload into the internal Chat-API format using helpers from server/src/lib/tool-args.ts. The toChatTools and toChatToolChoice functions rebuild the schema map necessary for argument validation later in the pipeline.

The Tool-Call Rescue Pipeline

Not all free-tier providers return tool calls in the standard OpenAI format. FreeLLMAPI includes a sophisticated normalization layer to handle provider-specific dialects.

Detecting Inline Tool Dialects

The server/src/lib/tool-call-rescue.ts module scans responses for known text-based markers. Providers such as Kimi, DeepSeek, Llama-Groq, and Qwen embed tool calls in plain text rather than structured JSON arrays. The rescue system identifies these dialects using the startsWithDialectMarker function, which detects signatures like the <|tool_calls_section_begin|> token used by Kimi.

Extraction and Validation Strategies

When a dialect is detected, the system attempts three parsing strategies in sequence:

• parseTokenDialect: Handles token-delimited formats like <|tool_calls_section_begin|><|tool_call_begin|>functions.get_weather:0<|tool_call_argument_begin|>{"city":"Paris"}<|tool_call_end|><|tool_call_begin|>
• parseFunctionTagDialect: Parses function-specific XML or tag-based wrappers
• parseXmlDialect: Extracts tool calls from generic XML structures

Each extractor returns a RescueResult struct containing the normalized tool_calls array. The extracted JSON is validated against the original tool list provided in the request to ensure the model did not hallucinate undefined functions.

Fallback and Retry Logic

If the dialect cannot be parsed or validation fails, the turn is marked as dead and the router automatically retries the request with the next available model in the fallback chain. This ensures robust execution even when individual providers return malformed tool syntax.

Response Normalization and Argument Repair

Before returning the response to the client, FreeLLMAPI performs final sanitization steps.

Repairing Double-Escaped Arguments

The responses.ts file (lines 191-215) runs the repairToolArguments helper to fix double-escaped JSON strings that some providers return. This ensures that tool arguments arrive at the client as valid, parseable JSON rather than nested string literals.

Preserving Round-Trip Semantics

The system preserves the finish_reason: "tool_calls" marker and maintains the exact tool_call_id values required for the subsequent tool-result message. This allows clients to execute the indicated function locally and feed results back via standard role: "tool" messages, continuing the conversation exactly as with the native OpenAI API.

Implementation Example: Multi-Step Tool Calling

The following Python example demonstrates the complete round-trip workflow using the OpenAI SDK against FreeLLMAPI:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:3001/v1",
    api_key="freellmapi-your-unified-key",
)

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Return current weather for a city.",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"],
        },
    },
}]

# Step 1: Request the tool call

first_response = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "What's the weather in Kyoto?"}],
    tools=tools,
    tool_choice="required",
)

tool_call = first_response.choices[0].message.tool_calls[0]

# Step 2: Execute the function locally

weather_data = {"temp_c": 22, "condition": "clear"}

# Step 3: Feed the result back to the model

final_response = client.chat.completions.create(
    model="auto",
    messages=[
        {"role": "user", "content": "What's the weather in Kyoto?"},
        first_response.choices[0].message,
        {
            "role": "tool",
            "tool_call_id": tool_call.id,
            "content": '{"temp_c":22,"condition":"clear"}'
        },
    ],
    tools=tools,
)

print(final_response.choices[0].message.content)

This identical code works whether the backend resolves to Google Gemini, Groq Llama-3.3, Cloudflare GLM-4.7, or any other supported provider. When the system selects a model that returns text dialects (such as Kimi), the tool-call-rescue.ts module transparently rewrites the response into the standard OpenAI format before it reaches your application.

Summary

• FreeLLMAPI provides a unified OpenAI-compatible endpoint at /v1/chat/completions that works across 16+ free-tier LLM providers.
• The router filters models based on supports_tools = 1 when requests contain tool definitions, ensuring only capable backends receive tool-enabled prompts.
• Provider adapters forward requests transparently without modifying the tools array, leveraging native OpenAI compatibility where available.
• The Tool-Call Rescue system in server/src/lib/tool-call-rescue.ts detects and parses provider-specific dialects (like Kimi's <|tool_calls_section_begin|> tokens) and converts them to standard tool_calls arrays.
• Response normalization repairs double-escaped arguments and preserves finish_reason markers to maintain full round-trip compatibility with the OpenAI SDK.

Frequently Asked Questions

How does FreeLLMAPI handle providers that don't support native tool calling?

FreeLLMAPI filters out models that do not advertise supports_tools = 1 in the model catalog. If a request contains tools but no capable models are available, the router rejects the request immediately rather than attempting execution. For providers that support tools but return non-standard formats (like Kimi or DeepSeek), the tool-call-rescue.ts module parses text dialects and converts them to valid OpenAI tool_calls structures.

Can I use the same tool definitions across different providers?

Yes. FreeLLMAPI accepts standard OpenAI tool schemas and forwards them unmodified to upstream providers. The system handles the translation layer internally, so your client code remains identical whether the backend is Google Gemini, Groq, Cloudflare, or any other supported service. You define tools once using the OpenAI format, and FreeLLMAPI manages provider-specific variations.

What happens if a provider returns a malformed tool call?

If the Tool-Call Rescue module cannot parse a provider's dialect or if the extracted JSON fails validation against the original tool schema, the turn is marked as dead. The router then automatically retries the request with the next available model in the fallback chain. This ensures that transient formatting issues from one provider do not break your application workflow.

Does FreeLLMAPI support the legacy OpenAI Responses API for tools?

Yes. The /v1/responses endpoint is supported via server/src/routes/responses.ts, which converts Responses-API payloads to the internal Chat-API format using helpers from server/src/lib/tool-args.ts. The system runs repairToolArguments on the reply to ensure tool schemas remain consistent, allowing you to use either the Chat Completions or Responses interface for tool calling.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}