# How Tool Calling Works Across LLM Providers in FreeLLMAPI: A Complete Technical Guide

> Learn how FreeLLMAPI unifies tool calling across 16+ LLM providers. Explore a technical guide to its OpenAI-compatible endpoint, provider normalization, and standard tool_calls conversion.

- Repository: [Tashfeen/freellmapi](https://github.com/tashfeenahmed/freellmapi)
- Tags: deep-dive
- Published: 2026-06-29

---

**FreeLLMAPI exposes a single OpenAI-compatible endpoint that normalizes tool calling across 16+ free-tier providers by filtering for tool-capable models, forwarding requests transparently, and converting provider-specific dialects back into standard OpenAI `tool_calls` format.**

FreeLLMAPI serves as a unified gateway that simplifies tool calling functionality across heterogeneous large language model providers. The repository implements a thin translation layer that accepts standard OpenAI tool definitions at `/v1/chat/completions` and automatically manages provider-specific variations, allowing developers to build function-calling applications without worrying about backend inconsistencies.

## Request Validation and Model Selection

When a client sends a request containing `tools` or `tool_choice` parameters, FreeLLMAPI executes a strict validation and routing pipeline before reaching any upstream provider.

### Type Validation and Schema Enforcement

Incoming requests are validated against the shared type definitions in [`server/src/shared/types.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/shared/types.ts) (lines 230-251). This module defines the `ChatToolDefinition` and `ChatToolChoice` interfaces that enforce structure across the entire system. The validation ensures that tool schemas match the expected OpenAI format before any provider selection occurs.

### Tool-Capable Model Filtering

The router logic in [`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts) (lines 887-894) filters the model catalog based on the `supports_tools` flag. When `requireTools` evaluates to true—meaning the request contains a non-empty `tools` array—the system exclusively considers models advertising `supports_tools = 1`. If no tool-capable models are available, the request fails fast rather than attempting execution on incompatible backends.

## Provider Dispatch and Request Forwarding

Once a suitable model is selected, FreeLLMAPI forwards the request without modification to the chosen provider.

### Transparent OpenAI-Compatible Proxying

The base provider implementation in [`server/src/providers/openai-compat.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/providers/openai-compat.ts) receives the request payload unchanged, including the verbatim `tools` array. This design leverages the fact that most modern providers implement the OpenAI chat specification and natively understand `tool_calls` syntax. FreeLLMAPI does not transform or mangle tool definitions; it acts as a pass-through proxy for providers that already support the standard.

### Legacy Endpoint Support

For clients using the legacy `/v1/responses` endpoint, the system in [`server/src/routes/responses.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/routes/responses.ts) converts the Responses-API payload into the internal Chat-API format using helpers from [`server/src/lib/tool-args.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/lib/tool-args.ts). The `toChatTools` and `toChatToolChoice` functions rebuild the schema map necessary for argument validation later in the pipeline.

## The Tool-Call Rescue Pipeline

Not all free-tier providers return tool calls in the standard OpenAI format. FreeLLMAPI includes a sophisticated normalization layer to handle provider-specific dialects.

### Detecting Inline Tool Dialects

The [`server/src/lib/tool-call-rescue.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/lib/tool-call-rescue.ts) module scans responses for known text-based markers. Providers such as **Kimi**, **DeepSeek**, **Llama-Groq**, and **Qwen** embed tool calls in plain text rather than structured JSON arrays. The rescue system identifies these dialects using the `startsWithDialectMarker` function, which detects signatures like the `<|tool_calls_section_begin|>` token used by Kimi.

### Extraction and Validation Strategies

When a dialect is detected, the system attempts three parsing strategies in sequence:

- **`parseTokenDialect`**: Handles token-delimited formats like `<|tool_calls_section_begin|><|tool_call_begin|>functions.get_weather:0<|tool_call_argument_begin|>{"city":"Paris"}<|tool_call_end|><|tool_call_begin|>`
- **`parseFunctionTagDialect`**: Parses function-specific XML or tag-based wrappers
- **`parseXmlDialect`**: Extracts tool calls from generic XML structures

Each extractor returns a `RescueResult` struct containing the normalized `tool_calls` array. The extracted JSON is validated against the original tool list provided in the request to ensure the model did not hallucinate undefined functions.

### Fallback and Retry Logic

If the dialect cannot be parsed or validation fails, the turn is marked as *dead* and the router automatically retries the request with the next available model in the fallback chain. This ensures robust execution even when individual providers return malformed tool syntax.

## Response Normalization and Argument Repair

Before returning the response to the client, FreeLLMAPI performs final sanitization steps.

### Repairing Double-Escaped Arguments

The [`responses.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/responses.ts) file (lines 191-215) runs the `repairToolArguments` helper to fix double-escaped JSON strings that some providers return. This ensures that tool arguments arrive at the client as valid, parseable JSON rather than nested string literals.

### Preserving Round-Trip Semantics

The system preserves the `finish_reason: "tool_calls"` marker and maintains the exact `tool_call_id` values required for the subsequent tool-result message. This allows clients to execute the indicated function locally and feed results back via standard `role: "tool"` messages, continuing the conversation exactly as with the native OpenAI API.

## Implementation Example: Multi-Step Tool Calling

The following Python example demonstrates the complete round-trip workflow using the OpenAI SDK against FreeLLMAPI:

```python
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:3001/v1",
    api_key="freellmapi-your-unified-key",
)

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Return current weather for a city.",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"],
        },
    },
}]

# Step 1: Request the tool call

first_response = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "What's the weather in Kyoto?"}],
    tools=tools,
    tool_choice="required",
)

tool_call = first_response.choices[0].message.tool_calls[0]

# Step 2: Execute the function locally

weather_data = {"temp_c": 22, "condition": "clear"}

# Step 3: Feed the result back to the model

final_response = client.chat.completions.create(
    model="auto",
    messages=[
        {"role": "user", "content": "What's the weather in Kyoto?"},
        first_response.choices[0].message,
        {
            "role": "tool",
            "tool_call_id": tool_call.id,
            "content": '{"temp_c":22,"condition":"clear"}'
        },
    ],
    tools=tools,
)

print(final_response.choices[0].message.content)

```

This identical code works whether the backend resolves to **Google Gemini**, **Groq Llama-3.3**, **Cloudflare GLM-4.7**, or any other supported provider. When the system selects a model that returns text dialects (such as Kimi), the [`tool-call-rescue.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/tool-call-rescue.ts) module transparently rewrites the response into the standard OpenAI format before it reaches your application.

## Summary

- **FreeLLMAPI** provides a unified OpenAI-compatible endpoint at `/v1/chat/completions` that works across 16+ free-tier LLM providers.
- The **router** filters models based on `supports_tools = 1` when requests contain tool definitions, ensuring only capable backends receive tool-enabled prompts.
- **Provider adapters** forward requests transparently without modifying the `tools` array, leveraging native OpenAI compatibility where available.
- The **Tool-Call Rescue** system in [`server/src/lib/tool-call-rescue.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/lib/tool-call-rescue.ts) detects and parses provider-specific dialects (like Kimi's `<|tool_calls_section_begin|>` tokens) and converts them to standard `tool_calls` arrays.
- **Response normalization** repairs double-escaped arguments and preserves `finish_reason` markers to maintain full round-trip compatibility with the OpenAI SDK.

## Frequently Asked Questions

### How does FreeLLMAPI handle providers that don't support native tool calling?

FreeLLMAPI filters out models that do not advertise `supports_tools = 1` in the model catalog. If a request contains tools but no capable models are available, the router rejects the request immediately rather than attempting execution. For providers that support tools but return non-standard formats (like Kimi or DeepSeek), the [`tool-call-rescue.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/tool-call-rescue.ts) module parses text dialects and converts them to valid OpenAI `tool_calls` structures.

### Can I use the same tool definitions across different providers?

Yes. FreeLLMAPI accepts standard OpenAI tool schemas and forwards them unmodified to upstream providers. The system handles the translation layer internally, so your client code remains identical whether the backend is Google Gemini, Groq, Cloudflare, or any other supported service. You define tools once using the OpenAI format, and FreeLLMAPI manages provider-specific variations.

### What happens if a provider returns a malformed tool call?

If the Tool-Call Rescue module cannot parse a provider's dialect or if the extracted JSON fails validation against the original tool schema, the turn is marked as *dead*. The router then automatically retries the request with the next available model in the fallback chain. This ensures that transient formatting issues from one provider do not break your application workflow.

### Does FreeLLMAPI support the legacy OpenAI Responses API for tools?

Yes. The `/v1/responses` endpoint is supported via [`server/src/routes/responses.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/routes/responses.ts), which converts Responses-API payloads to the internal Chat-API format using helpers from [`server/src/lib/tool-args.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/lib/tool-args.ts). The system runs `repairToolArguments` on the reply to ensure tool schemas remain consistent, allowing you to use either the Chat Completions or Responses interface for tool calling.