FreeLLM API Documentation: Complete Guide to the Self-Hosted LLM Aggregation Proxy

FreeLLM API provides comprehensive documentation covering its OpenAI-compatible /v1 endpoint, modular TypeScript architecture, and practical code examples for Python, curl, and streaming integrations.

FreeLLM API is an open-source, self-hosted proxy that unifies the free tiers of 18 LLM providers into a single endpoint. According to the tashfeenahmed/freellmapi repository, the documentation spans the main README, inline source comments, and structured examples demonstrating how the router selects healthy models and handles automatic fail-over. This guide distills the core architecture, key source files, and implementation patterns you need to deploy and consume the API.

Architecture Overview

The system follows a modular design where each component handles a specific responsibility in the request lifecycle. All traffic flows through the Express server entry point at server/src/index.ts.

Core Components

Router – Located in server/src/services/router.ts, this component chooses the optimal model per request based on health status, rate limits, and fallback priority. It implements OpenAI, Anthropic, and other wire-formats to ensure compatibility with existing SDKs.

Rate-Limit Ledger – The server/src/services/ratelimit.ts file maintains in-memory counters for RPM (requests per minute), RPD (requests per day), TPM (tokens per minute), and TPD (tokens per day) for each (platform, model, key) tuple. It automatically imposes cooldown periods when upstream providers return 429 or 5xx responses.

Health Service – Defined in server/src/services/health.ts, this service periodically probes each stored provider key and marks it as healthy, rate_limited, invalid, or error.

Provider Adapters – Individual files in server/src/providers/ (such as google.ts, groq.ts, and openrouter.ts) implement chatCompletion() and streamChatCompletion() methods to normalize requests for each upstream service.

Model Catalog – The server/src/services/model-listing.ts syncs a cryptographically signed model list twice daily from freellmapi.co, storing free and premium tier information in SQLite.

Storage Layer – server/src/db/index.ts implements SQLite persistence using better-sqlite3 with AES-256-GCM envelope encryption for provider keys, requiring the ENCRYPTION_KEY environment variable.

Dashboard UI – The React + Vite admin panel resides in client/src/App.tsx, providing key management, fallback chain ordering, analytics, and a playground interface.

Desktop Wrapper – An Electron tray application in desktop/src/window.ts runs the router and UI locally without requiring separate credentials.

Request Lifecycle

Each inbound request traverses the following stages:

1. Authentication – Validates the Bearer token format freellmapi-….
2. Routing – Queries the router to select the highest-priority model passing health and quota checks.
3. Key Decryption – Decrypts the stored provider key just-in-time using the configured encryption key.
4. Provider Call – Invokes the appropriate adapter to forward the request to the upstream service.
5. Fail-Over – On 429 or 5xx errors, the system applies a cooldown to the failed key and attempts the next model in the fallback chain (up to 20 attempts).
6. Response – Streams the upstream response unchanged, injecting X-Routed-Via and X-Fallback-Attempts headers for observability.

Key Source Files and Implementation Details

The documentation is embedded throughout the codebase, with critical logic concentrated in these locations:

• server/src/services/router.ts – Central request routing and fallback chain logic.
• server/src/services/ratelimit.ts – In-memory rate-limit ledger implementation.
• server/src/services/health.ts – Periodic health check orchestration.
• server/src/services/model-listing.ts – Free and premium model catalog synchronization.
• server/src/providers/*.ts – Provider-specific adapters (e.g., google.ts, groq.ts).
• server/src/db/index.ts – SQLite schema and encryption handling.
• client/src/App.tsx – React dashboard entry point.
• desktop/src/window.ts – Electron main process for desktop deployment.
• .env.example – Required environment variables including ENCRYPTION_KEY.
• docker/README.md – Container deployment instructions.

Usage Examples

Python OpenAI SDK

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:3001/v1",
    api_key="freellmapi-your-unified-key",
)

resp = client.chat.completions.create(
    model="auto",  # let the router pick the best free model

    messages=[{"role": "user", "content": "Summarise the fall of Rome in one sentence."}],
)

print(resp.choices[0].message.content)
print("Routed via:", resp.headers.get("x-routed-via"))

Curl Requests

curl http://localhost:3001/v1/chat/completions \
  -H "Authorization: Bearer freellmapi-your-unified-key" \
  -H "Content-Type: application/json" \
  -d '{
        "model": "auto",
        "messages": [{"role": "user", "content": "hi"}]
      }'

Streaming Responses

stream = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Write a haiku about SQLite."}],
    stream=True,
)

for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

Tool Calling

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get current weather for a city.",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"],
        },
    },
}]

first = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "What's the weather in Karachi?"}],
    tools=tools,
    tool_choice="required",
)

call = first.choices[0].message.tool_calls[0]

# Simulate executing the tool

weather = '{"temp_c": 32, "cond": "sunny"}'

final = client.chat.completions.create(
    model="auto",
    messages=[
        {"role": "user", "content": "What's the weather in Karachi?"},
        first.choices[0].message,
        {"role": "tool", "tool_call_id": call.id, "content": weather},
    ],
    tools=tools,
)

print(final.choices[0].message.content)

Anthropic Claude Compatibility

curl http://localhost:3001/v1/messages \
  -H "x-api-key: freellmapi-your-unified-key" \
  -H "anthropic-version: 2023-06-01" \
  -H "Content-Type: application/json" \
  -d '{
        "model": "claude-sonnet-4-5",
        "max_tokens": 256,
        "messages": [{"role": "user", "content": "hi"}]
      }'

Embeddings with Family-Aware Fail-Over

curl http://localhost:3001/v1/embeddings \
  -H "Authorization: Bearer freellmapi-your-unified-key" \
  -H "Content-Type: application/json" \
  -d '{"model": "auto", "input": ["the quick brown fox", "jumps over"]}'

Advanced Configuration Features

Context Handoff – When FREELLMAPI_CONTEXT_HANDOFF=on_model_switch is enabled, the system injects a compact system message informing the new model of the conversation handoff during fallback events.

Vision Support – Requests containing image blocks are automatically routed only to models advertising the vision capability in the model catalog.

Tool Calling – The proxy fully supports OpenAI-style tools and tool_choice parameters, injecting and relaying tool_calls between the client and upstream providers.

Embeddings Fail-Over – Unlike chat completions, embedding fail-over is family-aware, ensuring all vectors in a batch remain in the same embedding space when switching providers.

Summary

• FreeLLM API aggregates 18 free-tier LLM providers into a single OpenAI-compatible /v1 endpoint, with full support for streaming, tool calling, and embeddings.
• The modular architecture separates routing (router.ts), rate limiting (ratelimit.ts), health checks (health.ts), and provider adapters (providers/*.ts) for maintainability.
• Security relies on AES-256-GCM encryption for provider keys stored in SQLite, decrypted just-in-time during request processing.
• The system supports up to 20 automatic fallback attempts per request, with cooldown management for rate-limited or erroring keys.
• Documentation exists in the repository README, inline code comments, and the .env.example file, alongside the React dashboard in client/src/App.tsx.

Frequently Asked Questions

Where is the FreeLLM API documentation located?

The documentation is distributed across the repository's README.md, inline source comments in TypeScript files, and the docker/README.md for deployment guidance. The .env.example file documents required environment variables including ENCRYPTION_KEY, while the React dashboard provides interactive documentation for key management and fallback chains.

How does FreeLLM API handle authentication and security?

The API validates Bearer tokens prefixed with freellmapi-… at the entry point. Provider keys are encrypted at rest using AES-256-GCM in the SQLite database (server/src/db/index.ts) and decrypted just-in-time during the request lifecycle. No plaintext keys are stored in memory longer than necessary for the active request.

Can I use FreeLLM API with the Anthropic Claude SDK?

Yes. The router implements the Anthropic wire-format at the /v1/messages endpoint, making it compatible with Claude Code and the official Anthropic SDK. Send requests with the x-api-key header and anthropic-version header as shown in the curl examples, and the system will route to available Claude models.

What happens when a provider returns a 429 or 5xx error?

The rate-limit ledger in server/src/services/ratelimit.ts immediately marks the key as rate-limited or erroring and applies a cooldown period. The router then automatically selects the next model in the fallback chain, retrying up to 20 times until a healthy provider responds. Response headers include X-Fallback-Attempts indicating how many retries occurred.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}