FreeLLM API Usage Examples: Python, Curl, Streaming, and Tool Calling

FreeLLM API provides an OpenAI-compatible endpoint at /v1/* that aggregates free tiers from 18 LLM providers, allowing you to route requests through a single unified key with automatic failover and rate limit management.

The tashfeenahmed/freellmapi repository implements a self-hosted proxy that normalizes access to models from Google, Groq, Cerebras, NVIDIA, Mistral, OpenRouter, and others. Whether you are building a Python application or integrating with VS Code extensions, understanding practical FreeLLM API usage patterns will help you leverage its intelligent routing and fallback capabilities.

Basic FreeLLM API Usage with Python

The most common FreeLLM API usage pattern involves the OpenAI SDK pointed at your local proxy. When you set model="auto", the router in server/src/services/router.ts automatically selects the highest-priority healthy provider based on your configured key chain and rate limit counters stored in SQLite.

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:3001/v1",
    api_key="freellmapi-your-unified-key",
)

resp = client.chat.completions.create(
    model="auto",                     # let the router pick a model

    messages=[{"role": "user", "content": "Summarise the fall of Rome in one sentence."}],
)

print(resp.choices[0].message.content)
print("Routed via:", resp.headers.get("x-routed-via"))

The response headers include x-routed-via, which identifies the actual provider (e.g., "google", "groq") that served the request, enabling you to track which adapters in server/src/providers/*.ts handled the traffic.

Command Line FreeLLM API Usage

For quick testing or shell scripting, you can invoke the endpoint directly with curl. This method uses the same bearer token authentication required by the Express proxy.

curl http://localhost:3001/v1/chat/completions \
  -H "Authorization: Bearer freellmapi-your-unified-key" \
  -H "Content-Type: application/json" \
  -d '{
        "model": "auto",
        "messages": [{"role": "user", "content": "hi"}]
      }'

Streaming FreeLLM API Usage

When handling long-form content, streaming responses reduce perceived latency. The proxy maintains the connection to the upstream provider (selected by the router) and forwards chunks incrementally.

stream = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Stream me a haiku about SQLite."}],
    stream=True,
)

for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

This pattern works across all supported providers, with the router in server/src/services/router.ts handling the provider-specific streaming implementations transparently.

Tool Calling and Function Calling Examples

FreeLLM API supports OpenAI-style tool calling workflows, allowing models to request function executions and return results for final completion. The router preserves the tool calling context across fallback events if a provider fails mid-request.

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get current weather for a city.",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"]
        },
    },
}]

# First request – model asks for a tool call

first = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "What's the weather in Karachi?"}],
    tools=tools,
    tool_choice="required",
)

call = first.choices[0].message.tool_calls[0]

# Execute the tool locally, then feed the result back

final = client.chat.completions.create(
    model="auto",
    messages=[
        {"role": "user", "content": "What's the weather in Karachi?"},
        first.choices[0].message,
        {"role": "tool", "tool_call_id": call.id,
         "content": '{"temp_c": 32, "cond": "sunny"}'},
    ],
    tools=tools,
)

print(final.choices[0].message.content)

The provider adapters translate these tool schemas to each upstream API format, ensuring compatibility across heterogeneous services like Google Gemini and Groq.

Embeddings and Vector Usage

For embedding generation, the router selects from available embedding-capable providers (defaulting to gemini-embedding-001 when model="auto").

resp = client.embeddings.create(
    model="auto",
    input=["the quick brown fox", "pack my box with five dozen liquor jugs"],
)

print(len(resp.data), "vectors of", len(resp.data[0].embedding), "dims")

Usage counters for embeddings are tracked in-memory and persisted to server/src/db/model-pricing.ts alongside chat completion metrics.

Anthropic Client Compatibility

FreeLLM API also exposes a native Anthropic-compatible endpoint at /v1/messages, allowing Claude-specific clients to connect without modification.

curl http://localhost:3001/v1/messages \
  -H "x-api-key: freellmapi-your-unified-key" \
  -H "anthropic-version: 2023-06-01" \
  -H "Content-Type: application/json" \
  -d '{
        "model": "claude-sonnet-4-5",
        "max_tokens": 256,
        "messages": [{"role": "user", "content": "hi"}]
      }'

Integration with Development Tools

You can configure FreeLLM API as a custom endpoint in VS Code extensions like Continue for AI-powered autocomplete.

models:
  - name: FreeLLMAPI Autocomplete
    provider: openai
    model: auto
    apiBase: http://localhost:3001/v1
    apiKey: freellmapi-your-unified-key
    useLegacyCompletionsEndpoint: true
    roles:
      - autocomplete

How FreeLLM API Routes Your Requests

Understanding the architecture behind FreeLLM API usage helps optimize your configuration. When a request arrives:

1. Authentication: The unified bearer token is validated against the encrypted key store in server/src/db/model-pricing.ts.
2. Model Selection: The router evaluates health status (from server/src/services/health.ts) and rate limits (managed in server/src/services/ratelimit.ts) to select the best provider.
3. Provider Adaptation: The request is translated by the appropriate adapter in server/src/providers/*.ts (e.g., google.ts, groq.ts, openrouter.ts).
4. Fallback Handling: If a provider returns 429, 5xx, or timeouts, the router applies a cooldown to that key and retries with the next model in the chain.
5. Response Streaming: The adapter streams the response back to your client with x-routed-via headers injected.

The admin UI in client/src/pages/KeysPage.tsx allows you to reorder the fallback chain and monitor key health, while client/src/pages/PlaygroundPage.tsx provides an interactive environment to test different models.

Summary

• FreeLLM API usage requires only changing the base_url and api_key in your OpenAI SDK client to point to http://localhost:3001/v1 with your unified token.
• The router in server/src/services/router.ts automatically handles provider selection, health checks, and rate-limit cooldowns across 18 integrated services.
• Streaming, tool calling, and embeddings follow standard OpenAI patterns, with the proxy handling provider-specific translations.
• All provider keys are AES-256-GCM encrypted in SQLite (server/src/db/model-pricing.ts) and decrypted in-memory only during request forwarding.
• The system supports fallback chains, automatically retrying failed requests with alternative providers when rate limits or errors occur.

Frequently Asked Questions

How does FreeLLM API handle authentication?

FreeLLM API uses a single unified bearer token (format: freellmapi-…) for client authentication, while storing all upstream provider keys encrypted with AES-256-GCM in a local SQLite database. When you add keys through the React admin interface (client/src/pages/KeysPage.tsx), they are encrypted before persistence and only decrypted in-memory during the request lifecycle in server/src/services/router.ts.

What happens when a provider rate limits my request?

If a provider returns a 429 error or 5xx status, or if the request times out, the router in server/src/services/router.ts automatically falls back to the next available model in your configured priority chain. The failing key enters a short cooldown period managed by server/src/services/ratelimit.ts, and rate-limit counters (RPM, RPD, TPM, TPD) are updated in-memory and persisted to server/src/db/model-pricing.ts to prevent immediate retry.

Can I use FreeLLM API with Claude/Anthropic clients?

Yes, FreeLLM API exposes an Anthropic-compatible endpoint at /v1/messages that accepts x-api-key and anthropic-version headers. This allows you to use Claude-specific SDKs or tools by pointing them to http://localhost:3001/v1 instead of Anthropic's official API, with the router managing provider selection and fallback across Google, Groq, and other services that offer Claude models.

How do I monitor which provider routed my request?

Every response includes an x-routed-via header indicating the specific provider adapter (e.g., "google", "groq", "cerebras") that handled the request. You can also inspect the admin dashboard at client/src/pages/PlaygroundPage.tsx to see real-time routing decisions, or check the health status of individual keys in server/src/services/health.ts to understand which providers are currently active in your rotation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}