How FreeLLMAPI Handles Tool Calling Across Different LLM Providers

FreeLLMAPI unifies tool calling across heterogeneous LLM providers by normalizing requests through Zod validation, synthesizing tool-call IDs for tracking, and implementing a rescue layer that parses inline dialects—such as <function=...> tags or tokens—into standard OpenAI-compatible tool_calls arrays.

FreeLLMAPI is an open-source API gateway that aggregates multiple large language model providers under a single OpenAI-compatible endpoint. The repository implements a sophisticated tool calling abstraction layer that bridges the gap between providers offering native structured function calling and those emitting tool invocations as raw text or custom formats.

Request Validation and ID Synthesis

All tool calling requests enter through the /chat/completions endpoint defined in server/src/routes/proxy.ts. The middleware validates incoming payloads using a Zod schema that normalizes roles, content blocks, and tool definitions. When tools are present, the system generates synthetic IDs and maintains a pendingToolCallIds queue to ensure subsequent tool messages can be paired correctly with their initiating calls.

The validation layer also enforces a critical guard at lines 86-92: if a request contains tools, it must route to a model capable of emitting structured tool_calls. Otherwise, the router rejects the request early with a 422 error, preventing incompatible providers from receiving tool-bearing payloads they cannot process.

OpenAI-Compatible Provider Handling

For providers adhering to the OpenAI specification—such as Groq, NVIDIA NIM, and Mistral—FreeLLMAPI applies additional transformations in server/src/providers/openai-compat.ts.

Parallel Tool Call Restrictions

Some providers, including NVIDIA NIM, reject requests containing parallel_tool_calls: true. The OpenAICompatProvider class automatically resolves this conflict through the resolveParallelToolCalls method:

private resolveParallelToolCalls(options?: CompletionOptions) {
  if (this.forceSingleToolCall && options?.tools?.length) return false;
  return options?.parallel_tool_calls;
}

This logic, found at lines 57-60 of openai-compat.ts, forces the parameter to false whenever tools are present and the provider requires single tool invocation.

Inline Dialect Rescue

When models emit tool calls as inline text—such as <function=NAME{...}</function> blocks, tokens, or bare JSON—rather than structured arrays, FreeLLMAPI activates its rescue mechanism. If the provider returns an error containing tool_use_failed or a normal response with inline dialects, the system extracts and parses these into proper OpenAI tool_calls:

const rescued = this.rescueFailedGeneration(err, options);
if (rescued) {
  // build a synthetic ChatCompletionResponse with structured tool calls
}

This rescue logic, implemented at lines 70-88 of openai-compat.ts, ensures clients receive consistent JSON structures even when the underlying model produces non-compliant text.

Non-OpenAI Provider Strategies

Providers such as Google Gemini, Anthropic, Cohere, and Cloudflare require distinct handling strategies.

Google Gemini returns tool calls in a native toolCalls field. The Gemini provider class uses normalizeChoices to fold reasoning content into message content while preserving the structured tool calls, converting them to the OpenAI format before returning to the client.

Anthropic supports tool calling via tool_use messages. The Anthropic provider transforms these native messages into OpenAI-compatible tool_calls arrays through conversion logic in server/src/providers/anthropic.ts.

Cohere and Cloudflare do not support tool calling in their free tiers. The proxy layer rejects tool-bearing requests to these providers with a no_tools_model error, ensuring clients receive immediate feedback rather than failed generations.

Argument Repair and Schema Enforcement

Even after successful extraction, tool arguments may suffer from double-encoding—common in free-tier models like GLM—or contain schema-irrelevant keys. The repairToolArguments helper in server/src/lib/tool-args.ts walks the JSON payload, unwraps stringified objects, and validates parameters against the tool's defined schema:

const schemas = toolSchemaMap(options?.tools);
return rescue.calls.map(c => ({
  id: `call_rescued_${i}`,
  type: 'function',
  function: {
    name: c.name,
    arguments: repairToolArguments(c.arguments, schemas.get(c.name))
  }
}));

This function, spanning lines 27-84 of tool-args.ts, ensures that arguments conform to the expected structure before returning to the client.

Implementation Example

The following example demonstrates making a tool-calling request through FreeLLMAPI:

const payload = {
  model: "gpt-oss-120b",
  messages: [
    { role: "user", content: "What is the weather in Paris?" }
  ],
  tools: [
    {
      type: "function",
      function: {
        name: "getWeather",
        description: "Fetch current weather for a city",
        parameters: {
          type: "object",
          properties: {
            city: { type: "string" }
          },
          required: ["city"]
        }
      }
    }
  ],
  tool_choice: "required"
};

const resp = await fetch("http://localhost:3000/v1/chat/completions", {
  method: "POST",
  headers: {
    "Authorization": "Bearer <YOUR_UNIFIED_API_KEY>",
    "Content-Type": "application/json"
  },
  body: JSON.stringify(payload)
});

Under the hood, proxy.ts validates the payload, creates synthetic IDs, and routes to an appropriate provider. If the selected provider returns inline dialect text instead of structured JSON, rescueFailedGeneration parses the content, repairToolArguments sanitizes the parameters, and the system returns a standard OpenAI-compatible response.

Summary

FreeLLMAPI abstracts provider heterogeneity through a layered tool calling architecture:

• Request normalization via Zod schemas and synthetic ID generation in server/src/routes/proxy.ts
• Capability guards that prevent tool-bearing requests from reaching incompatible providers
• Parallel tool call handling that forces single invocation mode for restricted providers like NVIDIA NIM
• Dialect rescue that parses inline tool calls from error responses or raw text into structured arrays
• Argument repair that fixes double-encoded JSON and validates against tool schemas

Frequently Asked Questions

What happens when a model returns tool calls as plain text instead of structured JSON?

FreeLLMAPI detects inline dialect markers—including <function=...> tags, tokens, and bare JSON—through the rescueFailedGeneration method in server/src/providers/openai-compat.ts. The system extracts these patterns from error responses or normal message content and reconstructs them into proper OpenAI tool_calls arrays with synthetic IDs.

Why does FreeLLMAPI disable parallel tool calls for certain providers?

Providers such as NVIDIA NIM reject requests containing parallel_tool_calls: true. The resolveParallelToolCalls method in server/src/providers/openai-compat.ts automatically forces this value to false whenever tools are present and the provider requires sequential tool invocation, preventing 400-level errors while maintaining functionality.

How does the system handle tool calling when switching between different model providers?

When conversations fail over between providers, the new model may inherit the previous provider's output dialect. The rescue layer in server/src/lib/tool-call-rescue.ts normalizes these inline formats into standard structures, ensuring that tool calls remain parseable regardless of which backend generated the content or which backend is currently processing the request.

Which providers in FreeLLMAPI do not support tool calling?

Cohere and Cloudflare Workers AI do not support tool calling in their free-tier implementations. The validation logic in server/src/routes/proxy.ts rejects tool-bearing requests to these providers with a 422 error, routing them only to compatible backends such as OpenAI-compatible services, Google Gemini, or Anthropic.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}