How FreeLLMAPI Handles Tool Calling: Router, Discovery, and Fusion Implementation

FreeLLMAPI processes tool-calling requests by validating model capabilities against a supports_tools database flag, probing unknown providers with a dummy weather tool, and extracting tool calls from provider responses via the fusion service.

FreeLLMAPI serves as an open-source universal gateway for large language models, treating tool calling as a first-class capability that requires coordination across multiple architectural layers. When a client submits a request containing a tools array or tool_choice parameter, the tashfeenahmed/freellmapi repository executes a validation pipeline to ensure only compatible providers receive the payload. This pipeline involves capability probes, database-backed filtering, and response parsing to reliably route and process tool-calling requests.

Detecting Provider Tool Support

Before routing any request, FreeLLMAPI determines whether a provider actually implements the OpenAI-compatible tool-calling protocol. Rather than relying solely on static configuration, the system actively tests providers and persists the results.

The Tool Probe Mechanism

In server/src/services/model-discovery.ts (lines 617-682), the discovery service sends a synthetic request containing a dummy get_weather tool definition with tool_choice: 'auto'. It then inspects the response's finish_reason to determine if the provider correctly terminated with a tool call:

// Tool probe sent to unverified providers
const toolRes = await provider.chatCompletion(apiKey, toolMessages, modelId, {
  tools: [TOOL_PROBE],  // Dummy get_weather function
  tool_choice: 'auto',
});
const finishReason = toolRes.choices?.[0]?.finish_reason;
const supportsTools = finishReason === 'tool_calls';

If the provider returns tool_calls as the finish reason, the system marks it as capable; otherwise, the probe fails safely and the provider is flagged as incompatible.

Database Flag Persistence

The discovery layer updates the model database with a binary supports_tools flag (0 for no support, 1 for support). This persistent flag eliminates redundant network probes on subsequent requests, allowing the router to make instant eligibility decisions based on cached capability data.

Routing Requests with Tool Requirements

The router in server/src/services/router.ts (lines 996-1004) acts as the gatekeeper, filtering the candidate model pool using the persisted capability flags before forwarding any request containing tool definitions.

Capability-Based Filtering

When a request specifies requireTools, the router iterates through available models and checks the entry.supports_tools property. If a model lacks the flag but the request requires tools, the router immediately discards that candidate:

// Router exclusion logic for tool requirements
if (requireTools && !entry.supports_tools) {
  diag.push(`${label}: no tool-calling support`);
  continue;
}

This prevents incompatible providers from receiving payloads they cannot process, avoiding malformed responses or silent failures.

Exhaustion Diagnostics

If the filtering process eliminates all candidate models, the router's summarizeExhaustion function generates a specific diagnostic message: "model lacks tool-calling". This provides clear visibility into why the request failed, distinguishing between general unavailability and specific capability gaps.

Extracting Tool Calls from Responses

Once a qualified provider processes the request, the fusion service handles response parsing and tool call extraction in server/src/services/fusion.ts (lines 244-250).

Response Parsing Logic

The fusion layer accesses the provider's JSON response through the path choice?.message?.tool_calls. It validates that the field exists and contains an array with length greater than zero:

// Extraction and validation of tool calls
const toolCalls = choice?.message?.tool_calls;
const hasToolCalls = Array.isArray(toolCalls) && toolCalls.length > 0;
if (hasToolCalls) {
  // Forward to tool execution layer
}

Execution Flow

Upon detecting valid tool calls, the fusion service forwards the parsed function definitions and arguments to the appropriate tool handlers. The execution results are then re-injected into the conversation context, allowing the model to generate a final response incorporating the tool output before returning to the client.

Summary

• FreeLLMAPI stores provider capability in a supports_tools database flag to enable instant eligibility checks without repeated network probes
• The model discovery service actively probes unknown providers using a dummy weather tool to verify OpenAI-compatible tool-calling support (lines 617-682 in server/src/services/model-discovery.ts)
• The router filters models based on the supports_tools flag, excluding incompatible providers when tools are required and generating specific "model lacks tool-calling" diagnostics (lines 996-1004 in server/src/services/router.ts)
• The fusion service extracts tool calls by checking choice.message.tool_calls array length before forwarding to executors (lines 244-250 in server/src/services/fusion.ts)

Frequently Asked Questions

How does FreeLLMAPI determine if a provider supports tool calling?

The system sends a probe request containing a dummy get_weather tool and inspects the response's finish_reason. If the value equals tool_calls, the provider is marked with supports_tools = 1 in the database; otherwise, it remains flagged as incompatible.

What happens if no available models support tool calling?

The router generates a "model lacks tool-calling" diagnostic through its exhaustion summary and rejects the request immediately. This prevents tool payloads from reaching providers that would fail to process them correctly.

Where does FreeLLMAPI extract tool calls from provider responses?

The fusion service extracts them from the choice?.message?.tool_calls path within the provider's JSON response, verifying that the field is a non-empty array before passing the calls to the execution layer.

Can FreeLLMAPI handle mixed deployments with partial tool support?

Yes, the per-model supports_tools flag allows the system to route tool requests only to verified capable models while simultaneously sending standard completion requests to any available provider in the same deployment.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}