# FreeLLM API Features: OpenAI-Compatible Proxy for Free Tier LLMs

> Explore FreeLLM API features: an open-source, self-hosted OpenAI-compatible proxy for free LLMs. Enjoy intelligent routing, failover, and analytics.

- Repository: [Tashfeen/freellmapi](https://github.com/tashfeenahmed/freellmapi)
- Tags: features
- Published: 2026-07-02

---

**FreeLLM API is an open-source, self-hosted proxy that aggregates dozens of free-tier LLM providers into a single OpenAI-compatible `/v1` endpoint, featuring intelligent routing, automatic failover, encrypted key storage, and comprehensive analytics.**

FreeLLM API eliminates the complexity of managing multiple AI provider accounts by exposing a unified interface that works with any OpenAI-compatible client. This Node.js-based gateway, available in the tashfeenahmed/freellmapi repository, automatically handles provider selection, rate-limit protection, and quota management while giving you full control through a built-in React dashboard.

## Core Architecture Components

### Request Router and Provider Selection

The heart of the system resides in [`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts), where the `routeRequest` methodacts as a smart load balancer. It selects the optimal provider based on model availability, key health, and specific requirements like vision or tool support. The router maintains a fallback chain of up to 20 alternative models, automatically retrying when a provider returns 429 or 5xx errors.

### Rate Limiting and Cooldown Protection

To prevent quota exhaustion, [`server/src/services/ratelimit.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/ratelimit.ts) implements sophisticated tracking for **requests per minute (RPM)**, **requests per day (RPD)**, **tokens per minute (TPM)**, and **tokens per day (TPD)**. The `canMakeRequest` and `canUseTokens` functions query the SQLite `rate_limit_usage` table to enforce limits. When thresholds are exceeded, the `setCooldown` function temporarily removes that provider from rotation, protecting your free-tier allocations from hard locks.

### Web Dashboard and Analytics

The React-based frontend ([`client/src/pages/Keys.tsx`](https://github.com/tashfeenahmed/freellmapi/blob/main/client/src/pages/Keys.tsx)) provides a visual interface for managing encrypted provider keys and monitoring real-time metrics. A background health service ([`server/src/services/health.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/health.ts)) pings each configured key and updates its `status` field (healthy, error, etc.), while request logs—including latency, token counts, and per-provider breakdowns—are stored in the `requests` table for dashboard visualization.

## OpenAI-Compatible Endpoints

### Chat Completions and Legacy Support

The Express server exposes standard endpoints including `/v1/chat/completions`, `/v1/completions`, and `/v1/models`. In [`router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/router.ts), legacy completion requests are automatically translated into the modern chat completion format before routing to maintain backward compatibility.

### Streaming and Server-Sent Events

When `stream: true` is requested, the router returns **Server-Sent Events (SSE)** for real-time token delivery. The implementation handles both streaming and non-streaming responses transparently, requiring no client-side changes beyond standard OpenAI SDK parameters.

### Anthropic Messages API Compatibility

FreeLLM API recognizes `anthropic-version` headers and serves `/v1/messages` for Claude models, using the same provider adapters and routing logic as OpenAI-format requests. This allows you to use both OpenAI and Anthropic SDKs against the same endpoint.

## Advanced Routing Capabilities

### Automatic Failover and Retry Logic

The system implements aggressive **automatic failover**: upon receiving rate-limit errors or server failures, the router immediately marks the provider with `setCooldown` (via the ratelimit service) and attempts the next model in the chain. This process repeats for up to 20 attempts to ensure high availability across volatile free tiers.

### Sticky Sessions for Conversation Continuity

For maintaining context across turns, the router supports **30-minute model stickiness** via the `X-Session-Id` header. The system stores a `preferredModelDbId` for each session, ensuring subsequent requests hit the same model unless it becomes unavailable, preventing jarring mid-conversation provider switches.

### Vision and Multimodal Support

When the `requireVision` guard detects image inputs in [`router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/router.ts), the system filters for models with `supports_vision` capability before routing. This enables image understanding requests to reach vision-capable providers automatically, while also supporting `/v1/images/generations` for image generation and `/v1/audio/speech` for text-to-speech.

### Tool Calling Preservation

Requests containing `tools` or `tool_choice` parameters pass through unchanged, with the proxy preserving `tool_calls` across different provider implementations. The `requireTools` guard in [`router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/router.ts) ensures only providers supporting function calling receive these requests, maintaining compatibility with agent frameworks.

## Security and Key Management

### AES-256-GCM Encryption

Provider keys are never stored in plaintext. The system uses **AES-256-GCM** encryption implemented in [`server/src/lib/crypto.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/lib/crypto.ts), storing `encrypted_key`, `iv`, and `auth_tag` in SQLite. Keys are decrypted only in-memory immediately before provider calls, minimizing exposure.

### Unified API Key Authentication

Instead of managing multiple provider tokens, you generate a single `freellmapi-...` bearer token from the dashboard. Clients authenticate with this unified key while the server internally selects and decrypts the appropriate provider credentials.

## Self-Updating Infrastructure

### Dynamic Model Catalog

The system periodically downloads a signed model catalog from `freellmapi.co` via scripts in `server/src/scripts/`, updating the SQLite database with new free-tier models without requiring manual configuration or restarts.

### Context Hand-Off

When model switching occurs mid-conversation, the router can optionally inject a system message describing the hand-off, controlled via the `FREELLMAPI_CONTEXT_HANDOFF` environment variable.

## Usage Examples

### Basic Chat Completion with Python

```python
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:3001/v1",
    api_key="freellmapi-your-unified-key",
)

resp = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Summarise the fall of Rome."}],
)

print(resp.choices[0].message.content)
print("Routed via:", resp.headers.get("x-routed-via"))

```

### Streaming Responses

```python
stream = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Write a haiku about SQLite."}],
    stream=True,
)

for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

```

### Tool Calling Example

```python
tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get current weather for a city.",
        "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]},
    },
}]

first = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "What's the weather in Paris?"}],
    tools=tools,
    tool_choice="required",
)

call = first.choices[0].message.tool_calls[0]

# Execute tool, then send result back

final = client.chat.completions.create(
    model="auto",
    messages=[
        {"role": "user", "content": "What's the weather in Paris?"},
        first.choices[0].message,
        {"role": "tool", "tool_call_id": call.id, "content": '{"temp_c":12,"cond":"sunny"}'},
    ],
    tools=tools,
)

```

### curl Request

```bash
curl http://localhost:3001/v1/chat/completions \
  -H "Authorization: Bearer freellmapi-your-unified-key" \
  -H "Content-Type: application/json" \
  -d '{
        "model": "auto",
        "messages": [{"role": "user", "content": "hi"}]
      }'

```

## Deployment Options

FreeLLM API requires **Node.js 20+** and runs on x64 or ARM architectures. The repository provides a multi-architecture Docker image (`docker/`) for server deployments and an Electron desktop wrapper (`desktop/`) for single-user installations.

## Summary

- **Unified OpenAI-compatible endpoint** (`/v1/chat/completions`) for dozens of free-tier providers
- **Intelligent routing** with automatic failover (up to 20 retries) and 30-minute sticky sessions
- **Comprehensive rate limiting** tracking RPM, RPD, TPM, and TPD with automatic cooldown protection
- **AES-256-GCM encryption** for all provider keys with unified `freellmapi-...` bearer token authentication
- **Built-in dashboard** for key management, health monitoring, and analytics
- **Self-updating model catalog** and support for vision, tools, streaming, and image generation
- **Flexible deployment** via Docker, desktop app, or direct Node.js installation

## Frequently Asked Questions

### What providers does FreeLLM API support?

FreeLLM API supports any provider offering an OpenAI-compatible endpoint, including Google, Groq, OpenRouter, and Claude via Anthropic's Messages API. The self-updating model catalog in `server/src/scripts/` automatically syncs available free-tier models from `freellmapi.co`, ensuring you always have access to current offerings without manual configuration.

### How does FreeLLM API handle rate limits?

The system tracks usage per key in the `rate_limit_usage` SQLite table using [`server/src/services/ratelimit.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/ratelimit.ts). It monitors requests per minute/day and tokens per minute/day. When limits approach, the `setCooldown` function temporarily removes that provider from the rotation, triggering automatic failover to the next available model in the chain.

### Is FreeLLM API compatible with existing OpenAI SDK clients?

Yes. FreeLLM API is fully compatible with the official OpenAI Python and JavaScript SDKs, as well as any HTTP client that speaks the OpenAI REST format. Simply change the `base_url` to your FreeLLM API instance and use the unified key as the `api_key`. The proxy handles translation for Anthropic and legacy completion formats internally.

### How secure is the API key storage?

All provider keys are encrypted using **AES-256-GCM** before storage in the database, with separate fields for `encrypted_key`, `iv`, and `auth_tag`. Decryption occurs only in-memory immediately before making provider requests, and keys are never logged or exposed in the dashboard. The unified authentication token (`freellmapi-...`) acts as a secure proxy to these encrypted credentials.