# How FreeLLMAPI Handles Tool Calling Across Different LLM Providers

> Discover how FreeLLMAPI unifies tool calling across diverse LLM providers with Zod validation and a rescue layer, simplifying cross-provider integration.

- Repository: [Tashfeen/freellmapi](https://github.com/tashfeenahmed/freellmapi)
- Tags: how-to-guide
- Published: 2026-06-27

---

**FreeLLMAPI unifies tool calling across heterogeneous LLM providers by normalizing requests through Zod validation, synthesizing tool-call IDs for tracking, and implementing a rescue layer that parses inline dialects—such as `<function=...>` tags or `  ` tokens—into standard OpenAI-compatible `tool_calls` arrays.**

FreeLLMAPI is an open-source API gateway that aggregates multiple large language model providers under a single OpenAI-compatible endpoint. The repository implements a sophisticated **tool calling** abstraction layer that bridges the gap between providers offering native structured function calling and those emitting tool invocations as raw text or custom formats.

## Request Validation and ID Synthesis

All **tool calling** requests enter through the `/chat/completions` endpoint defined in [`server/src/routes/proxy.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/routes/proxy.ts). The middleware validates incoming payloads using a Zod schema that normalizes roles, content blocks, and tool definitions. When tools are present, the system generates synthetic IDs and maintains a `pendingToolCallIds` queue to ensure subsequent tool messages can be paired correctly with their initiating calls.

The validation layer also enforces a critical guard at lines 86-92: if a request contains tools, it must route to a model capable of emitting structured `tool_calls`. Otherwise, the router rejects the request early with a 422 error, preventing incompatible providers from receiving tool-bearing payloads they cannot process.

## OpenAI-Compatible Provider Handling

For providers adhering to the OpenAI specification—such as Groq, NVIDIA NIM, and Mistral—FreeLLMAPI applies additional transformations in [`server/src/providers/openai-compat.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/providers/openai-compat.ts).

### Parallel Tool Call Restrictions

Some providers, including NVIDIA NIM, reject requests containing `parallel_tool_calls: true`. The `OpenAICompatProvider` class automatically resolves this conflict through the `resolveParallelToolCalls` method:

```typescript
private resolveParallelToolCalls(options?: CompletionOptions) {
  if (this.forceSingleToolCall && options?.tools?.length) return false;
  return options?.parallel_tool_calls;
}

```

This logic, found at lines 57-60 of [`openai-compat.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/openai-compat.ts), forces the parameter to `false` whenever tools are present and the provider requires single tool invocation.

### Inline Dialect Rescue

When models emit tool calls as inline text—such as `<function=NAME{...}</function>` blocks, `  ` tokens, or bare JSON—rather than structured arrays, FreeLLMAPI activates its rescue mechanism. If the provider returns an error containing `tool_use_failed` or a normal response with inline dialects, the system extracts and parses these into proper OpenAI `tool_calls`:

```typescript
const rescued = this.rescueFailedGeneration(err, options);
if (rescued) {
  // build a synthetic ChatCompletionResponse with structured tool calls
}

```

This rescue logic, implemented at lines 70-88 of [`openai-compat.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/openai-compat.ts), ensures clients receive consistent JSON structures even when the underlying model produces non-compliant text.

## Non-OpenAI Provider Strategies

Providers such as Google Gemini, Anthropic, Cohere, and Cloudflare require distinct handling strategies.

**Google Gemini** returns tool calls in a native `toolCalls` field. The Gemini provider class uses `normalizeChoices` to fold reasoning content into message content while preserving the structured tool calls, converting them to the OpenAI format before returning to the client.

**Anthropic** supports tool calling via `tool_use` messages. The Anthropic provider transforms these native messages into OpenAI-compatible `tool_calls` arrays through conversion logic in [`server/src/providers/anthropic.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/providers/anthropic.ts).

**Cohere and Cloudflare** do not support tool calling in their free tiers. The proxy layer rejects tool-bearing requests to these providers with a `no_tools_model` error, ensuring clients receive immediate feedback rather than failed generations.

## Argument Repair and Schema Enforcement

Even after successful extraction, tool arguments may suffer from double-encoding—common in free-tier models like GLM—or contain schema-irrelevant keys. The `repairToolArguments` helper in [`server/src/lib/tool-args.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/lib/tool-args.ts) walks the JSON payload, unwraps stringified objects, and validates parameters against the tool's defined schema:

```typescript
const schemas = toolSchemaMap(options?.tools);
return rescue.calls.map(c => ({
  id: `call_rescued_${i}`,
  type: 'function',
  function: {
    name: c.name,
    arguments: repairToolArguments(c.arguments, schemas.get(c.name))
  }
}));

```

This function, spanning lines 27-84 of [`tool-args.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/tool-args.ts), ensures that arguments conform to the expected structure before returning to the client.

## Implementation Example

The following example demonstrates making a tool-calling request through FreeLLMAPI:

```typescript
const payload = {
  model: "gpt-oss-120b",
  messages: [
    { role: "user", content: "What is the weather in Paris?" }
  ],
  tools: [
    {
      type: "function",
      function: {
        name: "getWeather",
        description: "Fetch current weather for a city",
        parameters: {
          type: "object",
          properties: {
            city: { type: "string" }
          },
          required: ["city"]
        }
      }
    }
  ],
  tool_choice: "required"
};

const resp = await fetch("http://localhost:3000/v1/chat/completions", {
  method: "POST",
  headers: {
    "Authorization": "Bearer <YOUR_UNIFIED_API_KEY>",
    "Content-Type": "application/json"
  },
  body: JSON.stringify(payload)
});

```

Under the hood, [`proxy.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/proxy.ts) validates the payload, creates synthetic IDs, and routes to an appropriate provider. If the selected provider returns inline dialect text instead of structured JSON, `rescueFailedGeneration` parses the content, `repairToolArguments` sanitizes the parameters, and the system returns a standard OpenAI-compatible response.

## Summary

FreeLLMAPI abstracts provider heterogeneity through a layered **tool calling** architecture:

- **Request normalization** via Zod schemas and synthetic ID generation in [`server/src/routes/proxy.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/routes/proxy.ts)
- **Capability guards** that prevent tool-bearing requests from reaching incompatible providers
- **Parallel tool call handling** that forces single invocation mode for restricted providers like NVIDIA NIM
- **Dialect rescue** that parses inline tool calls from error responses or raw text into structured arrays
- **Argument repair** that fixes double-encoded JSON and validates against tool schemas

## Frequently Asked Questions

### What happens when a model returns tool calls as plain text instead of structured JSON?

FreeLLMAPI detects inline dialect markers—including `<function=...>` tags, `  ` tokens, and bare JSON—through the `rescueFailedGeneration` method in [`server/src/providers/openai-compat.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/providers/openai-compat.ts). The system extracts these patterns from error responses or normal message content and reconstructs them into proper OpenAI `tool_calls` arrays with synthetic IDs.

### Why does FreeLLMAPI disable parallel tool calls for certain providers?

Providers such as NVIDIA NIM reject requests containing `parallel_tool_calls: true`. The `resolveParallelToolCalls` method in [`server/src/providers/openai-compat.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/providers/openai-compat.ts) automatically forces this value to `false` whenever tools are present and the provider requires sequential tool invocation, preventing 400-level errors while maintaining functionality.

### How does the system handle tool calling when switching between different model providers?

When conversations fail over between providers, the new model may inherit the previous provider's output dialect. The rescue layer in [`server/src/lib/tool-call-rescue.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/lib/tool-call-rescue.ts) normalizes these inline formats into standard structures, ensuring that tool calls remain parseable regardless of which backend generated the content or which backend is currently processing the request.

### Which providers in FreeLLMAPI do not support tool calling?

Cohere and Cloudflare Workers AI do not support tool calling in their free-tier implementations. The validation logic in [`server/src/routes/proxy.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/routes/proxy.ts) rejects tool-bearing requests to these providers with a 422 error, routing them only to compatible backends such as OpenAI-compatible services, Google Gemini, or Anthropic.