FreeLLM API Features: OpenAI-Compatible Proxy for Free Tier LLMs

FreeLLM API is an open-source, self-hosted proxy that aggregates dozens of free-tier LLM providers into a single OpenAI-compatible /v1 endpoint, featuring intelligent routing, automatic failover, encrypted key storage, and comprehensive analytics.

FreeLLM API eliminates the complexity of managing multiple AI provider accounts by exposing a unified interface that works with any OpenAI-compatible client. This Node.js-based gateway, available in the tashfeenahmed/freellmapi repository, automatically handles provider selection, rate-limit protection, and quota management while giving you full control through a built-in React dashboard.

Core Architecture Components

Request Router and Provider Selection

The heart of the system resides in server/src/services/router.ts, where the routeRequest methodacts as a smart load balancer. It selects the optimal provider based on model availability, key health, and specific requirements like vision or tool support. The router maintains a fallback chain of up to 20 alternative models, automatically retrying when a provider returns 429 or 5xx errors.

Rate Limiting and Cooldown Protection

To prevent quota exhaustion, server/src/services/ratelimit.ts implements sophisticated tracking for requests per minute (RPM), requests per day (RPD), tokens per minute (TPM), and tokens per day (TPD). The canMakeRequest and canUseTokens functions query the SQLite rate_limit_usage table to enforce limits. When thresholds are exceeded, the setCooldown function temporarily removes that provider from rotation, protecting your free-tier allocations from hard locks.

Web Dashboard and Analytics

The React-based frontend (client/src/pages/Keys.tsx) provides a visual interface for managing encrypted provider keys and monitoring real-time metrics. A background health service (server/src/services/health.ts) pings each configured key and updates its status field (healthy, error, etc.), while request logs—including latency, token counts, and per-provider breakdowns—are stored in the requests table for dashboard visualization.

OpenAI-Compatible Endpoints

Chat Completions and Legacy Support

The Express server exposes standard endpoints including /v1/chat/completions, /v1/completions, and /v1/models. In router.ts, legacy completion requests are automatically translated into the modern chat completion format before routing to maintain backward compatibility.

Streaming and Server-Sent Events

When stream: true is requested, the router returns Server-Sent Events (SSE) for real-time token delivery. The implementation handles both streaming and non-streaming responses transparently, requiring no client-side changes beyond standard OpenAI SDK parameters.

Anthropic Messages API Compatibility

FreeLLM API recognizes anthropic-version headers and serves /v1/messages for Claude models, using the same provider adapters and routing logic as OpenAI-format requests. This allows you to use both OpenAI and Anthropic SDKs against the same endpoint.

Advanced Routing Capabilities

Automatic Failover and Retry Logic

The system implements aggressive automatic failover: upon receiving rate-limit errors or server failures, the router immediately marks the provider with setCooldown (via the ratelimit service) and attempts the next model in the chain. This process repeats for up to 20 attempts to ensure high availability across volatile free tiers.

Sticky Sessions for Conversation Continuity

For maintaining context across turns, the router supports 30-minute model stickiness via the X-Session-Id header. The system stores a preferredModelDbId for each session, ensuring subsequent requests hit the same model unless it becomes unavailable, preventing jarring mid-conversation provider switches.

Vision and Multimodal Support

When the requireVision guard detects image inputs in router.ts, the system filters for models with supports_vision capability before routing. This enables image understanding requests to reach vision-capable providers automatically, while also supporting /v1/images/generations for image generation and /v1/audio/speech for text-to-speech.

Tool Calling Preservation

Requests containing tools or tool_choice parameters pass through unchanged, with the proxy preserving tool_calls across different provider implementations. The requireTools guard in router.ts ensures only providers supporting function calling receive these requests, maintaining compatibility with agent frameworks.

Security and Key Management

AES-256-GCM Encryption

Provider keys are never stored in plaintext. The system uses AES-256-GCM encryption implemented in server/src/lib/crypto.ts, storing encrypted_key, iv, and auth_tag in SQLite. Keys are decrypted only in-memory immediately before provider calls, minimizing exposure.

Unified API Key Authentication

Instead of managing multiple provider tokens, you generate a single freellmapi-... bearer token from the dashboard. Clients authenticate with this unified key while the server internally selects and decrypts the appropriate provider credentials.

Self-Updating Infrastructure

Dynamic Model Catalog

The system periodically downloads a signed model catalog from freellmapi.co via scripts in server/src/scripts/, updating the SQLite database with new free-tier models without requiring manual configuration or restarts.

Context Hand-Off

When model switching occurs mid-conversation, the router can optionally inject a system message describing the hand-off, controlled via the FREELLMAPI_CONTEXT_HANDOFF environment variable.

Usage Examples

Basic Chat Completion with Python

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:3001/v1",
    api_key="freellmapi-your-unified-key",
)

resp = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Summarise the fall of Rome."}],
)

print(resp.choices[0].message.content)
print("Routed via:", resp.headers.get("x-routed-via"))

Streaming Responses

stream = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Write a haiku about SQLite."}],
    stream=True,
)

for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

Tool Calling Example

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get current weather for a city.",
        "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]},
    },
}]

first = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "What's the weather in Paris?"}],
    tools=tools,
    tool_choice="required",
)

call = first.choices[0].message.tool_calls[0]

# Execute tool, then send result back

final = client.chat.completions.create(
    model="auto",
    messages=[
        {"role": "user", "content": "What's the weather in Paris?"},
        first.choices[0].message,
        {"role": "tool", "tool_call_id": call.id, "content": '{"temp_c":12,"cond":"sunny"}'},
    ],
    tools=tools,
)

curl Request

curl http://localhost:3001/v1/chat/completions \
  -H "Authorization: Bearer freellmapi-your-unified-key" \
  -H "Content-Type: application/json" \
  -d '{
        "model": "auto",
        "messages": [{"role": "user", "content": "hi"}]
      }'

Deployment Options

FreeLLM API requires Node.js 20+ and runs on x64 or ARM architectures. The repository provides a multi-architecture Docker image (docker/) for server deployments and an Electron desktop wrapper (desktop/) for single-user installations.

Summary

• Unified OpenAI-compatible endpoint (/v1/chat/completions) for dozens of free-tier providers
• Intelligent routing with automatic failover (up to 20 retries) and 30-minute sticky sessions
• Comprehensive rate limiting tracking RPM, RPD, TPM, and TPD with automatic cooldown protection
• AES-256-GCM encryption for all provider keys with unified freellmapi-... bearer token authentication
• Built-in dashboard for key management, health monitoring, and analytics
• Self-updating model catalog and support for vision, tools, streaming, and image generation
• Flexible deployment via Docker, desktop app, or direct Node.js installation

Frequently Asked Questions

What providers does FreeLLM API support?

FreeLLM API supports any provider offering an OpenAI-compatible endpoint, including Google, Groq, OpenRouter, and Claude via Anthropic's Messages API. The self-updating model catalog in server/src/scripts/ automatically syncs available free-tier models from freellmapi.co, ensuring you always have access to current offerings without manual configuration.

How does FreeLLM API handle rate limits?

The system tracks usage per key in the rate_limit_usage SQLite table using server/src/services/ratelimit.ts. It monitors requests per minute/day and tokens per minute/day. When limits approach, the setCooldown function temporarily removes that provider from the rotation, triggering automatic failover to the next available model in the chain.

Is FreeLLM API compatible with existing OpenAI SDK clients?

Yes. FreeLLM API is fully compatible with the official OpenAI Python and JavaScript SDKs, as well as any HTTP client that speaks the OpenAI REST format. Simply change the base_url to your FreeLLM API instance and use the unified key as the api_key. The proxy handles translation for Anthropic and legacy completion formats internally.

How secure is the API key storage?

All provider keys are encrypted using AES-256-GCM before storage in the database, with separate fields for encrypted_key, iv, and auth_tag. Decryption occurs only in-memory immediately before making provider requests, and keys are never logged or exposed in the dashboard. The unified authentication token (freellmapi-...) acts as a secure proxy to these encrypted credentials.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}