# FreeLLM API Usage Examples: Python, Curl, Streaming, and Tool Calling

> Explore FreeLLM API usage examples in Python, Curl, streaming, and tool calling. Discover how to leverage 18 free LLM providers with a single unified key and automatic failover.

- Repository: [Tashfeen/freellmapi](https://github.com/tashfeenahmed/freellmapi)
- Tags: usage-examples
- Published: 2026-07-02

---

**FreeLLM API provides an OpenAI-compatible endpoint at `/v1/*` that aggregates free tiers from 18 LLM providers, allowing you to route requests through a single unified key with automatic failover and rate limit management.**

The `tashfeenahmed/freellmapi` repository implements a self-hosted proxy that normalizes access to models from Google, Groq, Cerebras, NVIDIA, Mistral, OpenRouter, and others. Whether you are building a Python application or integrating with VS Code extensions, understanding practical FreeLLM API usage patterns will help you leverage its intelligent routing and fallback capabilities.

## Basic FreeLLM API Usage with Python

The most common FreeLLM API usage pattern involves the OpenAI SDK pointed at your local proxy. When you set `model="auto"`, the router in [`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts) automatically selects the highest-priority healthy provider based on your configured key chain and rate limit counters stored in SQLite.

```python
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:3001/v1",
    api_key="freellmapi-your-unified-key",
)

resp = client.chat.completions.create(
    model="auto",                     # let the router pick a model

    messages=[{"role": "user", "content": "Summarise the fall of Rome in one sentence."}],
)

print(resp.choices[0].message.content)
print("Routed via:", resp.headers.get("x-routed-via"))

```

The response headers include `x-routed-via`, which identifies the actual provider (e.g., "google", "groq") that served the request, enabling you to track which adapters in `server/src/providers/*.ts` handled the traffic.

## Command Line FreeLLM API Usage

For quick testing or shell scripting, you can invoke the endpoint directly with curl. This method uses the same bearer token authentication required by the Express proxy.

```bash
curl http://localhost:3001/v1/chat/completions \
  -H "Authorization: Bearer freellmapi-your-unified-key" \
  -H "Content-Type: application/json" \
  -d '{
        "model": "auto",
        "messages": [{"role": "user", "content": "hi"}]
      }'

```

## Streaming FreeLLM API Usage

When handling long-form content, streaming responses reduce perceived latency. The proxy maintains the connection to the upstream provider (selected by the router) and forwards chunks incrementally.

```python
stream = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Stream me a haiku about SQLite."}],
    stream=True,
)

for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

```

This pattern works across all supported providers, with the router in [`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts) handling the provider-specific streaming implementations transparently.

## Tool Calling and Function Calling Examples

FreeLLM API supports OpenAI-style tool calling workflows, allowing models to request function executions and return results for final completion. The router preserves the tool calling context across fallback events if a provider fails mid-request.

```python
tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get current weather for a city.",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"]
        },
    },
}]

# First request – model asks for a tool call

first = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "What's the weather in Karachi?"}],
    tools=tools,
    tool_choice="required",
)

call = first.choices[0].message.tool_calls[0]

# Execute the tool locally, then feed the result back

final = client.chat.completions.create(
    model="auto",
    messages=[
        {"role": "user", "content": "What's the weather in Karachi?"},
        first.choices[0].message,
        {"role": "tool", "tool_call_id": call.id,
         "content": '{"temp_c": 32, "cond": "sunny"}'},
    ],
    tools=tools,
)

print(final.choices[0].message.content)

```

The provider adapters translate these tool schemas to each upstream API format, ensuring compatibility across heterogeneous services like Google Gemini and Groq.

## Embeddings and Vector Usage

For embedding generation, the router selects from available embedding-capable providers (defaulting to `gemini-embedding-001` when `model="auto"`).

```python
resp = client.embeddings.create(
    model="auto",
    input=["the quick brown fox", "pack my box with five dozen liquor jugs"],
)

print(len(resp.data), "vectors of", len(resp.data[0].embedding), "dims")

```

Usage counters for embeddings are tracked in-memory and persisted to [`server/src/db/model-pricing.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/db/model-pricing.ts) alongside chat completion metrics.

## Anthropic Client Compatibility

FreeLLM API also exposes a native Anthropic-compatible endpoint at `/v1/messages`, allowing Claude-specific clients to connect without modification.

```bash
curl http://localhost:3001/v1/messages \
  -H "x-api-key: freellmapi-your-unified-key" \
  -H "anthropic-version: 2023-06-01" \
  -H "Content-Type: application/json" \
  -d '{
        "model": "claude-sonnet-4-5",
        "max_tokens": 256,
        "messages": [{"role": "user", "content": "hi"}]
      }'

```

## Integration with Development Tools

You can configure FreeLLM API as a custom endpoint in VS Code extensions like Continue for AI-powered autocomplete.

```yaml
models:
  - name: FreeLLMAPI Autocomplete
    provider: openai
    model: auto
    apiBase: http://localhost:3001/v1
    apiKey: freellmapi-your-unified-key
    useLegacyCompletionsEndpoint: true
    roles:
      - autocomplete

```

## How FreeLLM API Routes Your Requests

Understanding the architecture behind FreeLLM API usage helps optimize your configuration. When a request arrives:

1. **Authentication**: The unified bearer token is validated against the encrypted key store in [`server/src/db/model-pricing.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/db/model-pricing.ts).
2. **Model Selection**: The router evaluates health status (from [`server/src/services/health.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/health.ts)) and rate limits (managed in [`server/src/services/ratelimit.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/ratelimit.ts)) to select the best provider.
3. **Provider Adaptation**: The request is translated by the appropriate adapter in `server/src/providers/*.ts` (e.g., [`google.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/google.ts), [`groq.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/groq.ts), [`openrouter.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/openrouter.ts)).
4. **Fallback Handling**: If a provider returns 429, 5xx, or timeouts, the router applies a cooldown to that key and retries with the next model in the chain.
5. **Response Streaming**: The adapter streams the response back to your client with `x-routed-via` headers injected.

The admin UI in [`client/src/pages/KeysPage.tsx`](https://github.com/tashfeenahmed/freellmapi/blob/main/client/src/pages/KeysPage.tsx) allows you to reorder the fallback chain and monitor key health, while [`client/src/pages/PlaygroundPage.tsx`](https://github.com/tashfeenahmed/freellmapi/blob/main/client/src/pages/PlaygroundPage.tsx) provides an interactive environment to test different models.

## Summary

- **FreeLLM API usage** requires only changing the `base_url` and `api_key` in your OpenAI SDK client to point to `http://localhost:3001/v1` with your unified token.
- The **router** in [`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts) automatically handles provider selection, health checks, and rate-limit cooldowns across 18 integrated services.
- **Streaming**, **tool calling**, and **embeddings** follow standard OpenAI patterns, with the proxy handling provider-specific translations.
- All provider keys are **AES-256-GCM encrypted** in SQLite ([`server/src/db/model-pricing.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/db/model-pricing.ts)) and decrypted in-memory only during request forwarding.
- The system supports **fallback chains**, automatically retrying failed requests with alternative providers when rate limits or errors occur.

## Frequently Asked Questions

### How does FreeLLM API handle authentication?

FreeLLM API uses a single unified bearer token (format: `freellmapi-…`) for client authentication, while storing all upstream provider keys encrypted with AES-256-GCM in a local SQLite database. When you add keys through the React admin interface ([`client/src/pages/KeysPage.tsx`](https://github.com/tashfeenahmed/freellmapi/blob/main/client/src/pages/KeysPage.tsx)), they are encrypted before persistence and only decrypted in-memory during the request lifecycle in [`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts).

### What happens when a provider rate limits my request?

If a provider returns a 429 error or 5xx status, or if the request times out, the router in [`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts) automatically falls back to the next available model in your configured priority chain. The failing key enters a short cooldown period managed by [`server/src/services/ratelimit.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/ratelimit.ts), and rate-limit counters (RPM, RPD, TPM, TPD) are updated in-memory and persisted to [`server/src/db/model-pricing.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/db/model-pricing.ts) to prevent immediate retry.

### Can I use FreeLLM API with Claude/Anthropic clients?

Yes, FreeLLM API exposes an Anthropic-compatible endpoint at `/v1/messages` that accepts `x-api-key` and `anthropic-version` headers. This allows you to use Claude-specific SDKs or tools by pointing them to `http://localhost:3001/v1` instead of Anthropic's official API, with the router managing provider selection and fallback across Google, Groq, and other services that offer Claude models.

### How do I monitor which provider routed my request?

Every response includes an `x-routed-via` header indicating the specific provider adapter (e.g., "google", "groq", "cerebras") that handled the request. You can also inspect the admin dashboard at [`client/src/pages/PlaygroundPage.tsx`](https://github.com/tashfeenahmed/freellmapi/blob/main/client/src/pages/PlaygroundPage.tsx) to see real-time routing decisions, or check the health status of individual keys in [`server/src/services/health.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/health.ts) to understand which providers are currently active in your rotation.