# FreeLLM API Documentation: Complete Guide to the Self-Hosted LLM Aggregation Proxy

> Explore the FreeLLM API documentation. Get a complete guide to the self-hosted LLM aggregation proxy, including OpenAI-compatible endpoints, integrations, and code examples for Python and curl.

- Repository: [Tashfeen/freellmapi](https://github.com/tashfeenahmed/freellmapi)
- Tags: documentation
- Published: 2026-07-02

---

**FreeLLM API provides comprehensive documentation covering its OpenAI-compatible `/v1` endpoint, modular TypeScript architecture, and practical code examples for Python, curl, and streaming integrations.**

FreeLLM API is an open-source, self-hosted proxy that unifies the free tiers of 18 LLM providers into a single endpoint. According to the tashfeenahmed/freellmapi repository, the documentation spans the main **README**, inline source comments, and structured examples demonstrating how the router selects healthy models and handles automatic fail-over. This guide distills the core architecture, key source files, and implementation patterns you need to deploy and consume the API.

## Architecture Overview

The system follows a modular design where each component handles a specific responsibility in the request lifecycle. All traffic flows through the Express server entry point at [`server/src/index.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/index.ts).

### Core Components

**Router** – Located in [`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts), this component chooses the optimal model per request based on health status, rate limits, and fallback priority. It implements OpenAI, Anthropic, and other wire-formats to ensure compatibility with existing SDKs.

**Rate-Limit Ledger** – The [`server/src/services/ratelimit.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/ratelimit.ts) file maintains in-memory counters for RPM (requests per minute), RPD (requests per day), TPM (tokens per minute), and TPD (tokens per day) for each `(platform, model, key)` tuple. It automatically imposes cooldown periods when upstream providers return 429 or 5xx responses.

**Health Service** – Defined in [`server/src/services/health.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/health.ts), this service periodically probes each stored provider key and marks it as `healthy`, `rate_limited`, `invalid`, or `error`.

**Provider Adapters** – Individual files in `server/src/providers/` (such as [`google.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/google.ts), [`groq.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/groq.ts), and [`openrouter.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/openrouter.ts)) implement `chatCompletion()` and `streamChatCompletion()` methods to normalize requests for each upstream service.

**Model Catalog** – The [`server/src/services/model-listing.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/model-listing.ts) syncs a cryptographically signed model list twice daily from freellmapi.co, storing free and premium tier information in SQLite.

**Storage Layer** – [`server/src/db/index.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/db/index.ts) implements SQLite persistence using `better-sqlite3` with **AES-256-GCM** envelope encryption for provider keys, requiring the `ENCRYPTION_KEY` environment variable.

**Dashboard UI** – The React + Vite admin panel resides in [`client/src/App.tsx`](https://github.com/tashfeenahmed/freellmapi/blob/main/client/src/App.tsx), providing key management, fallback chain ordering, analytics, and a playground interface.

**Desktop Wrapper** – An Electron tray application in [`desktop/src/window.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/desktop/src/window.ts) runs the router and UI locally without requiring separate credentials.

### Request Lifecycle

Each inbound request traverses the following stages:

1. **Authentication** – Validates the Bearer token format `freellmapi-…`.
2. **Routing** – Queries the router to select the highest-priority model passing health and quota checks.
3. **Key Decryption** – Decrypts the stored provider key just-in-time using the configured encryption key.
4. **Provider Call** – Invokes the appropriate adapter to forward the request to the upstream service.
5. **Fail-Over** – On 429 or 5xx errors, the system applies a cooldown to the failed key and attempts the next model in the fallback chain (up to 20 attempts).
6. **Response** – Streams the upstream response unchanged, injecting `X-Routed-Via` and `X-Fallback-Attempts` headers for observability.

## Key Source Files and Implementation Details

The documentation is embedded throughout the codebase, with critical logic concentrated in these locations:

- **[`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts)** – Central request routing and fallback chain logic.
- **[`server/src/services/ratelimit.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/ratelimit.ts)** – In-memory rate-limit ledger implementation.
- **[`server/src/services/health.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/health.ts)** – Periodic health check orchestration.
- **[`server/src/services/model-listing.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/model-listing.ts)** – Free and premium model catalog synchronization.
- **`server/src/providers/*.ts`** – Provider-specific adapters (e.g., [`google.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/google.ts), [`groq.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/groq.ts)).
- **[`server/src/db/index.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/db/index.ts)** – SQLite schema and encryption handling.
- **[`client/src/App.tsx`](https://github.com/tashfeenahmed/freellmapi/blob/main/client/src/App.tsx)** – React dashboard entry point.
- **[`desktop/src/window.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/desktop/src/window.ts)** – Electron main process for desktop deployment.
- **`.env.example`** – Required environment variables including `ENCRYPTION_KEY`.
- **[`docker/README.md`](https://github.com/tashfeenahmed/freellmapi/blob/main/docker/README.md)** – Container deployment instructions.

## Usage Examples

### Python OpenAI SDK

```python
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:3001/v1",
    api_key="freellmapi-your-unified-key",
)

resp = client.chat.completions.create(
    model="auto",  # let the router pick the best free model

    messages=[{"role": "user", "content": "Summarise the fall of Rome in one sentence."}],
)

print(resp.choices[0].message.content)
print("Routed via:", resp.headers.get("x-routed-via"))

```

### Curl Requests

```bash
curl http://localhost:3001/v1/chat/completions \
  -H "Authorization: Bearer freellmapi-your-unified-key" \
  -H "Content-Type: application/json" \
  -d '{
        "model": "auto",
        "messages": [{"role": "user", "content": "hi"}]
      }'

```

### Streaming Responses

```python
stream = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Write a haiku about SQLite."}],
    stream=True,
)

for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

```

### Tool Calling

```python
tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get current weather for a city.",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"],
        },
    },
}]

first = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "What's the weather in Karachi?"}],
    tools=tools,
    tool_choice="required",
)

call = first.choices[0].message.tool_calls[0]

# Simulate executing the tool

weather = '{"temp_c": 32, "cond": "sunny"}'

final = client.chat.completions.create(
    model="auto",
    messages=[
        {"role": "user", "content": "What's the weather in Karachi?"},
        first.choices[0].message,
        {"role": "tool", "tool_call_id": call.id, "content": weather},
    ],
    tools=tools,
)

print(final.choices[0].message.content)

```

### Anthropic Claude Compatibility

```bash
curl http://localhost:3001/v1/messages \
  -H "x-api-key: freellmapi-your-unified-key" \
  -H "anthropic-version: 2023-06-01" \
  -H "Content-Type: application/json" \
  -d '{
        "model": "claude-sonnet-4-5",
        "max_tokens": 256,
        "messages": [{"role": "user", "content": "hi"}]
      }'

```

### Embeddings with Family-Aware Fail-Over

```bash
curl http://localhost:3001/v1/embeddings \
  -H "Authorization: Bearer freellmapi-your-unified-key" \
  -H "Content-Type: application/json" \
  -d '{"model": "auto", "input": ["the quick brown fox", "jumps over"]}'

```

## Advanced Configuration Features

**Context Handoff** – When `FREELLMAPI_CONTEXT_HANDOFF=on_model_switch` is enabled, the system injects a compact system message informing the new model of the conversation handoff during fallback events.

**Vision Support** – Requests containing image blocks are automatically routed only to models advertising the `vision` capability in the model catalog.

**Tool Calling** – The proxy fully supports OpenAI-style `tools` and `tool_choice` parameters, injecting and relaying `tool_calls` between the client and upstream providers.

**Embeddings Fail-Over** – Unlike chat completions, embedding fail-over is **family-aware**, ensuring all vectors in a batch remain in the same embedding space when switching providers.

## Summary

- FreeLLM API aggregates 18 free-tier LLM providers into a single OpenAI-compatible `/v1` endpoint, with full support for streaming, tool calling, and embeddings.
- The modular architecture separates routing ([`router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/router.ts)), rate limiting ([`ratelimit.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/ratelimit.ts)), health checks ([`health.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/health.ts)), and provider adapters (`providers/*.ts`) for maintainability.
- Security relies on AES-256-GCM encryption for provider keys stored in SQLite, decrypted just-in-time during request processing.
- The system supports up to 20 automatic fallback attempts per request, with cooldown management for rate-limited or erroring keys.
- Documentation exists in the repository README, inline code comments, and the `.env.example` file, alongside the React dashboard in [`client/src/App.tsx`](https://github.com/tashfeenahmed/freellmapi/blob/main/client/src/App.tsx).

## Frequently Asked Questions

### Where is the FreeLLM API documentation located?

The documentation is distributed across the repository's **README.md**, inline source comments in TypeScript files, and the [`docker/README.md`](https://github.com/tashfeenahmed/freellmapi/blob/main/docker/README.md) for deployment guidance. The `.env.example` file documents required environment variables including `ENCRYPTION_KEY`, while the React dashboard provides interactive documentation for key management and fallback chains.

### How does FreeLLM API handle authentication and security?

The API validates Bearer tokens prefixed with `freellmapi-…` at the entry point. Provider keys are encrypted at rest using **AES-256-GCM** in the SQLite database ([`server/src/db/index.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/db/index.ts)) and decrypted just-in-time during the request lifecycle. No plaintext keys are stored in memory longer than necessary for the active request.

### Can I use FreeLLM API with the Anthropic Claude SDK?

Yes. The router implements the Anthropic wire-format at the `/v1/messages` endpoint, making it compatible with Claude Code and the official Anthropic SDK. Send requests with the `x-api-key` header and `anthropic-version` header as shown in the curl examples, and the system will route to available Claude models.

### What happens when a provider returns a 429 or 5xx error?

The rate-limit ledger in [`server/src/services/ratelimit.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/ratelimit.ts) immediately marks the key as rate-limited or erroring and applies a cooldown period. The router then automatically selects the next model in the fallback chain, retrying up to 20 times until a healthy provider responds. Response headers include `X-Fallback-Attempts` indicating how many retries occurred.