# How to Implement Tool Calling with FreeLLMAPI

> Implement tool calling with FreeLLMAPI. This guide shows how to leverage OpenAI-compatible endpoints for structured tool calls and automatic JSON payload rescue for your LLM applications.

- Repository: [Tashfeen/freellmapi](https://github.com/tashfeenahmed/freellmapi)
- Tags: how-to-guide
- Published: 2026-08-29

---

**FreeLLMAPI exposes a fully OpenAI-compatible `/v1/chat/completions` endpoint that routes tool-calling requests only to models proven capable of emitting structured `tool_calls`, while automatically rescuing plain-text responses into proper JSON payloads.**

FreeLLMAPI is an open-source unified gateway that aggregates free-tier LLM providers behind a single OpenAI-compatible API. When you implement tool calling with FreeLLMAPI, the system automatically detects which models support function calling through a capability probe, filters the routing pool to exclude incompatible providers, and normalizes responses to ensure your client receives standard OpenAI-formatted `tool_calls` objects.

## How the Router Handles Tool-Capable Models

The core routing logic in [`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts) determines whether a model can handle tool requests before forwarding traffic. When a request includes a `tools` array or `tool_choice` parameter, the router checks the `supports_tools` flag in the model catalog entry.

At lines 2167-2172 of [`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts), the router implements a strict guard that prevents routing to incompatible models:

```typescript
// Logic from router.ts filtering non-tool-capable entries
if (requireTools && !entry.supports_tools) {
  // Skip models that cannot emit tool_calls
  continue;
}

```

This ensures that only providers proven capable of structured function calling receive tool requests. The check occurs alongside other capability filters around lines 2000-2008 of the same file, where the router evaluates `requireTools && !entry.supports_tools` to abort routing to any model that cannot emit structured `tool_calls`.

## Model Discovery and Capability Probing

Before requests hit the router, FreeLLMAPI runs a discovery process to populate the `supports_tools` column in the model catalog. In [`server/src/services/model-discovery.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/model-discovery.ts) (lines 498-560), the system executes a dummy **tool-probe** against each model using a mock `get_weather` function.

During this probe phase, the discovery service sends a test request with a tool definition and verifies that the model responds with a properly formatted `tool_calls` payload rather than plain text. Models that pass this test receive `supports_tools = 1` in the database, making them eligible for tool-routing logic.

If a provider returns unstructured text instead of JSON during actual request handling, FreeLLMAPI applies a **tool-call rescue** mechanism. According to [`docs/architecture.md`](https://github.com/tashfeenahmed/freellmapi/blob/main/docs/architecture.md) (line 55), the system parses the plain-text output and restructures it into a valid OpenAI `tool_calls` object before returning it to the client.

## Step-by-Step Implementation

The end-to-end flow for implementing tool calling follows standard OpenAI SDK patterns. FreeLLMAPI requires no proprietary client libraries.

### Configure the Client

Point your OpenAI SDK to the FreeLLMAPI base URL:

```python
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:3001/v1",
    api_key="freellmapi-your-unified-key",
)

```

### Define Tools and Send Requests

Construct your tool definitions using standard JSON Schema and include them in the chat completion request:

```python
tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Fetch current weather for a city",
        "parameters": {
            "type": "object",
            "properties": {
                "city": {"type": "string", "description": "City name"}
            },
            "required": ["city"]
        }
    }
}]

# Initiate tool calling

resp = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "What's the weather in Paris?"}],
    tools=tools,
    tool_choice="auto",
)

```

### Handle Tool Execution and Follow-Up

Extract the function call arguments, execute your local business logic, and return the result as a `tool` role message:

```python
choice = resp.choices[0]
if choice.message.tool_calls:
    call = choice.message.tool_calls[0]
    args = call.function.arguments  # JSON string

    result = '{"temp_c": 22, "cond": "cloudy"}'  # Your tool execution

    # Continue conversation with tool result

    follow_up = client.chat.completions.create(
        model="auto",
        messages=[
            {"role": "user", "content": "What's the weather in Paris?"},
            {"role": "assistant", "content": None, "tool_calls": [call]},
            {"role": "tool", "tool_call_id": call.id, "content": result},
        ],
        tools=tools,
        tool_choice="auto",
    )
    print(follow_up.choices[0].message.content)

```

This pattern matches the API examples documented in [`docs/api.md`](https://github.com/tashfeenahmed/freellmapi/blob/main/docs/api.md) (lines 12-14), ensuring compatibility with existing OpenAI integrations.

## Type Definitions and Data Structures

The TypeScript interfaces governing tool calling are defined in [`shared/types.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/shared/types.ts) (lines 385-409). These definitions ensure type safety across the routing layer and include structures for `tool_calls`, `tool_choice`, and the `tool` message role required for multi-turn conversations.

Key interfaces include:

- `ChatCompletionTool`: The tool definition schema sent in requests
- `ChatCompletionMessageToolCall`: The structured response object containing function names and arguments
- `ChatCompletionToolMessageParam`: The message type used when returning tool execution results

## Summary

- **FreeLLMAPI** exposes standard OpenAI-compatible endpoints for tool calling through `/v1/chat/completions`.
- The **router** ([`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts)) filters models using the `supports_tools` flag to ensure only capable providers receive tool requests.
- **Model discovery** ([`server/src/services/model-discovery.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/model-discovery.ts)) probes each provider with a dummy `get_weather` request to set the capability flag.
- **Tool-call rescue** automatically converts plain-text tool responses into proper JSON payloads when providers deviate from the OpenAI specification.
- Client implementations use standard OpenAI SDKs without modification, treating FreeLLMAPI as a drop-in replacement for the official API.

## Frequently Asked Questions

### What models support tool calling in FreeLLMAPI?

Models receive the `supports_tools` capability flag only if they pass the dummy tool probe during the discovery phase in [`server/src/services/model-discovery.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/model-discovery.ts). The router automatically excludes models lacking this flag from tool requests, ensuring compatibility by routing only to providers that correctly emit structured `tool_calls` JSON.

### How does FreeLLMAPI handle providers that return plain text instead of JSON tool calls?

According to [`docs/architecture.md`](https://github.com/tashfeenahmed/freellmapi/blob/main/docs/architecture.md), FreeLLMAPI implements a **tool-call rescue** mechanism that parses plain-text responses and restructures them into valid OpenAI `tool_calls` objects. This ensures clients receive consistent JSON payloads even when underlying free-tier providers return malformed or text-based function calls.

### Can I force a specific tool choice instead of letting the model decide?

Yes. FreeLLMAPI supports the `tool_choice` parameter defined in [`shared/types.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/shared/types.ts). You can set `tool_choice` to `"auto"` (model decides), `"none"` (no tools), or a specific object like `{"type": "function", "function": {"name": "get_weather"}}` to force the model to use a particular tool. The router respects this parameter when selecting from the pool of `supports_tools` flagged models.

### Do I need to modify existing OpenAI SDK code to work with FreeLLMAPI?

No. FreeLLMAPI maintains full OpenAI API compatibility as documented in [`docs/api.md`](https://github.com/tashfeenahmed/freellmapi/blob/main/docs/api.md). You only need to change the `base_url` to your FreeLLMAPI instance (e.g., `http://localhost:3001/v1`) and use your FreeLLMAPI key. All tool definitions, message roles, and response handling patterns remain identical to the official OpenAI implementation.