# How FreeLLMAPI Implements OpenAI-Compatible Tool Calling Across Multiple LLM Providers

> Discover how FreeLLMAPI enables OpenAI-compatible tool calling across LLM providers by normalizing protocols, filtering capable routes, and optimizing for structured tool calls.

- Repository: [Tashfeen/freellmapi](https://github.com/tashfeenahmed/freellmapi)
- Tags: how-to-guide
- Published: 2026-08-31

---

**FreeLLMAPI normalizes OpenAI's function-calling protocol by probing each model for tool support, filtering routes to only capable providers, and short-circuiting fusion panels when structured tool calls are detected.**

FreeLLMAPI is an open-source universal router that aggregates free-tier LLM providers under a single OpenAI-compatible API. To support **OpenAI-compatible tool calling** seamlessly across disparate backends like Groq, Anthropic, Gemini, and Cloudflare, the project implements a four-layer normalization system spanning type definitions, capability discovery, intelligent routing, and parallel-panel handling.

## Unified Type Definitions for OpenAI-Compatible Schemas

All tool-related fields adhere to a canonical OpenAI shape defined in **[`shared/types.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/shared/types.ts)**. This ensures every provider adapter receives and returns identical structures regardless of the underlying API.

The core interfaces include:

- **`ChatToolDefinition`** (lines 59-64): Describes available functions using the standard OpenAI `type`, `function.name`, `function.description`, and `function.parameters` schema.

- **`ChatToolChoice`** (lines 64-73): Controls invocation behavior with `"none"`, `"auto"`, `"required"`, or specific function selection.

- **`ChatToolCall`** and **`ChatToolCallFunction`** (lines 45-50): Carry the model-generated payload containing the target function name and JSON-encoded arguments.

- **`ChatMessage`** (lines 84-90): Extended to include an optional `tool_calls` array, enabling the proxy to forward structured responses back to the client unchanged.

- **`ChatCompletionRequest`** (lines 96-108): The central request body consumed throughout the server, explicitly declaring optional `tools`, `tool_choice`, and `parallel_tool_calls` fields.

By centralizing these definitions in [`shared/types.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/shared/types.ts), FreeLLMAPI guarantees that client code written for the OpenAI SDK works identically against any configured provider.

## Automatic Capability Discovery via Model Probing

Before a model enters the routing pool, **[`server/src/services/model-discovery.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/model-discovery.ts)** executes a lightweight probe to determine if it supports structured tool calls. The service sends a dummy `get_weather` function definition and inspects the response's `finish_reason`:

```typescript
// model-discovery.ts – lines 555-572
const toolMessages: ChatMessage[] = [
  { role: 'user', content: 'What is the weather in Paris? Use the get_weather tool.' },
];
const toolRes = await provider.chatCompletion(apiKey, toolMessages, modelId, {
  max_tokens: 64,
  temperature: 0,
  tools: [TOOL_PROBE],
  tool_choice: 'auto',
});
const finishReason = toolRes.choices?.[0]?.finish_reason;
toolCalls = finishReason === 'tool_calls';

```

If the model returns `tool_calls` as the finish reason, the discovery service persists this capability to the database:

```typescript
// model-discovery.ts – line 487
db.prepare(`UPDATE models SET supports_tools = ? WHERE id = ?`).run(toolCalls ? 1 : 0, modelId);

```

This **`supports_tools`** boolean flag becomes the authoritative signal for all subsequent routing decisions, ensuring that tool-capable requests never reach incompatible models.

## Routing and Request-Level Filtering

The core router in **[`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts)** constructs a fallback chain of enabled models. When an incoming request contains a non-empty `tools` array, the router sets `requireTools = true` and filters the chain accordingly:

```typescript
// router.ts – lines 2003-2008
if (requireTools && !entry.supports_tools) {
  diag.push(`${label}: no tool-calling support`);
  continue;
}

```

The same validation occurs within the Fusion service ([`server/src/services/fusion.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/fusion.ts)) when building parallel panels:

```typescript
// fusion.ts – lines 560-562
const requireTools = (options.tools?.length ?? 0) > 0;
if (requirements.requireTools && !cand.supportsTools) {
  dropped.push(`${id} (no tool-calling support)`);
  continue;
}

```

This dual-layer filtering ensures that **only models with verified `supports_tools = 1` ever receive tool-calling payloads**, preventing errors from providers that lack function-calling capability.

## Fusion Panel Handling of Tool Calls

FreeLLMAPI's Fusion feature executes multiple models in parallel to determine the best response. Because tool calls represent executable actions rather than generative text, the system treats them as high-priority outputs that must not be synthesized or altered.

When processing a Fusion panel, **[`server/src/services/fusion.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/fusion.ts)** checks for structured tool calls immediately after receiving responses:

```typescript
// fusion.ts – lines 640-646
const toolCallWinner = survivors.find(a =>
  (a.toolCalls?.length ?? 0) > 0 && a.rawChoice);
if (toolCallWinner) {
  // Return the raw tool call as the final response
}

```

If any panel member returns a valid `tool_calls` array, Fusion **short-circuits the normal synthesis flow** and returns that exact payload to the client with a `finish_reason` of `"tool_calls"`. This preserves the functional integrity of the tool invocation, ensuring argument fidelity and correct function targeting.

## Provider Adapter Implementation

Each provider adapter receives the normalized `tools` and `tool_choice` fields from the shared types and forwards them to the upstream API. The generic OpenAI-compatible adapter in **[`server/src/providers/openai-compat.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/providers/openai-compat.ts)** demonstrates this passthrough:

```typescript
// providers/openai-compat.ts – lines 204 (non-stream) & 323 (stream)
await fetch(endpoint, {
  method: 'POST',
  body: JSON.stringify({
    model,
    messages,
    tools: options?.tools,
    tool_choice: options?.tool_choice,
    // …other OpenAI params
  })
});

```

Specialized adapters for Cohere, Cloudflare, Gemini, and others implement sanitizing helpers (such as `sanitizeCohereTools` or `ollamaTools`) that map the canonical OpenAI shapes to provider-specific formats while maintaining the same interface contract.

## End-to-End Implementation Example

The following client request works against any free-tier provider configured in FreeLLMAPI, demonstrating the seamless **OpenAI-compatible tool calling** abstraction:

```typescript
import fetch from 'node-fetch';

const body = {
  model: 'auto',                // Router selects a tool-capable model
  messages: [{ role: 'user', content: 'What is the weather in Paris?' }],
  tools: [
    {
      type: 'function',
      function: {
        name: 'get_weather',
        description: 'Get the current weather for a city',
        parameters: {
          type: 'object',
          properties: {
            city: { type: 'string', description: 'Name of the city' }
          },
          required: ['city']
        }
      }
    }
  ],
  tool_choice: 'auto'
};

fetch('https://api.freellmapi.com/v1/chat/completions', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify(body)
})
  .then(r => r.json())
  .then(resp => {
    if (resp.choices?.[0]?.message?.tool_calls) {
      console.log('Tool call received:', resp.choices[0].message.tool_calls);
    } else {
      console.log('Plain text answer:', resp.choices[0].message.content);
    }
  });

```

Under the hood, FreeLLMAPI detects the `tools` array, filters the model fallback chain to only those entries where `supports_tools = 1`, dispatches the request, and returns the raw tool call without modification if generated.

## Summary

- **Unified schemas** in [`shared/types.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/shared/types.ts) enforce OpenAI-compatible `ChatToolDefinition`, `ChatToolCall`, and `ChatCompletionRequest` structures across all providers.
- **Capability discovery** in [`model-discovery.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/model-discovery.ts) probes each model with a dummy function and records tool support via the `supports_tools` database flag.
- **Request filtering** in [`router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/router.ts) and [`fusion.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/fusion.ts) automatically excludes non-tool-capable models when the `tools` array is present.
- **Fusion short-circuiting** returns the first valid `tool_calls` response immediately, bypassing text synthesis to preserve functional accuracy.
- **Provider adapters** forward normalized tool payloads to upstream APIs while handling provider-specific sanitization internally.

## Frequently Asked Questions

### How does FreeLLMAPI determine if a specific model supports tool calling?

The [`model-discovery.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/model-discovery.ts) service sends a probe request containing a dummy `get_weather` tool definition when a model first appears in the catalog. If the model responds with a `finish_reason` of `tool_calls`, the system sets `supports_tools = 1` in the database. This flag is then used by the router to filter eligible models for tool-calling requests.

### Can I use tool calling with the Fusion panel feature enabled?

Yes. When Fusion is active, the system runs multiple models in parallel. If any panel member returns a structured `tool_calls` payload, FreeLLMAPI immediately returns that specific response to the client and skips the normal synthesis process. This ensures tool calls are returned exactly as generated without aggregation or modification.

### What happens if no tool-capable models are available for my request?

If the router cannot find any enabled models with `supports_tools = 1` when the request includes a `tools` array, it logs diagnostic messages indicating "no tool-calling support" for each candidate and returns an error to the client. This prevents sending tool schemas to providers that cannot process them, avoiding runtime failures.

### Do I need to modify my OpenAI SDK code to work with FreeLLMAPI?

No. Because FreeLLMAPI uses the exact type definitions from [`shared/types.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/shared/types.ts) for `tools`, `tool_choice`, and `tool_calls`, standard OpenAI SDK clients work without modification. Simply point the base URL to your FreeLLMAPI instance and ensure your requested model identifier matches or use `auto` to let the router select a capable backend.