How to Get Started with FreeLLM API: A Self-Hosted OpenAI-Compatible Proxy

FreeLLM API is a self-hosted, OpenAI-compatible proxy that aggregates free LLM tiers into a single /v1 endpoint with automatic routing and fallback handling.

FreeLLM API (freellmapi) is an open-source project by tashfeenahmed that unifies dozens of LLM providers behind one OpenAI-style interface. The system is built on Node.js/TypeScript and handles provider selection, rate limit management, and encrypted credential storage automatically.

Architecture Overview

The system centers on a router that intelligently distributes requests across available free-tier providers.

Core Components

Express Server – The entry point defined in server/src/app.ts receives OpenAI-style requests at /v1/* and mounts the admin dashboard.

Router Service – Implemented in server/src/services/router.ts, this component selects the highest-priority healthy model, injects context-handoff messages when switching providers, and retries up to 20 fall-over attempts during failures.

Rate-Limit Ledger – Located in server/src/services/ratelimit.ts, this maintains in-memory counters backed by SQLite to enforce per-platform RPM/RPD/TPM/TPD limits and manages cooldowns after HTTP 429 or 5xx responses.

Provider Adapters – Files in server/src/providers/*.ts (such as the Google adapter) implement chatCompletion() and streamChatCompletion() for each vendor, translating wire-format differences like Anthropic's message structure.

Health Service – The server/src/services/health.ts module performs periodic probes of stored keys, marking them as healthy, rate_limited, invalid, or error for the router to filter.

Encrypted Storage – Database operations in server/src/db/model-pricing.ts use AES-256-GCM encryption for API keys, with the encryption key supplied via environment variables.

Dashboard UI – A React + Vite admin panel in client/src/App.tsx provides key management, analytics, and token generation.

Desktop Wrapper – An Electron application in desktop/src/main.ts runs the server locally with a tray pop-over for live statistics.

The router syncs a signed model catalog from freellmapi.co twice daily, ensuring new free models appear automatically. Responses include X-Routed-Via headers identifying the provider and X-Fallback-Attempts headers when fall-over occurs.

Installation Methods

Server Installation

Clone the repository and install dependencies:

git clone https://github.com/tashfeenahmed/freellmapi.git
cd freellmapi
npm install

Configure your environment using the template in .env.example:

# Required: 32-byte encryption key for AES-256-GCM

ENCRYPTION_KEY=your-32-byte-hex-key-here

# Optional: Server port (default 3001)

PORT=3001

Start the server:

npm run server

Desktop Application

For a zero-install experience on macOS or Windows, build the Electron wrapper:

# macOS

npm run desktop:dist

# Windows

npm run desktop:dist:win

The desktop application runs the same Node.js server locally while providing a system tray interface for monitoring live stats.

Using the API

FreeLLM API exposes an OpenAI-compatible endpoint at http://localhost:3001/v1. Use the unified API key generated in the dashboard (format: freellmapi-your-unified-key).

Python OpenAI SDK

Configure the OpenAI client to point to your local instance:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:3001/v1",
    api_key="freellmapi-your-unified-key",
)

resp = client.chat.completions.create(
    model="auto",  # Let the router choose the best available model

    messages=[{"role": "user", "content": "Summarise the fall of Rome in one sentence."}],
)

print(resp.choices[0].message.content)
print("Routed via:", resp.headers.get("x-routed-via"))

cURL Requests

Send requests directly via HTTP:

curl http://localhost:3001/v1/chat/completions \
  -H "Authorization: Bearer freellmapi-your-unified-key" \
  -H "Content-Type: application/json" \
  -d '{
        "model": "auto",
        "messages": [{"role": "user", "content": "hi"}]
      }'

Streaming Responses

Enable streaming for real-time token delivery:

stream = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Stream me a haiku about SQLite."}],
    stream=True,
)

for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

VS Code Integration

Configure the Continue extension for ghost-text autocomplete:

models:
  - name: FreeLLMAPI Autocomplete
    provider: openai
    model: auto
    apiBase: http://localhost:3001/v1
    apiKey: freellmapi-your-unified-key
    useLegacyCompletionsEndpoint: true
    roles:
      - autocomplete

Key Source Files Reference

Understanding these files helps customize routing behavior:

• server/src/services/router.ts – Core routing logic, fallback chain management, and context handoff injection when switching models mid-conversation.

• server/src/services/ratelimit.ts – In-memory rate-limit counters and cooldown handling for provider-specific quotas.

• server/src/providers/*.ts – Individual adapter implementations for each LLM vendor (Google, Groq, Cerebras, OpenRouter) handling translation between FreeLLM API wire format and vendor SDKs.

• server/src/services/health.ts – Periodic health checks that update key status to prevent routing to failed providers.

• server/src/app.ts – Express server setup, route mounting for /v1/* endpoints, and admin UI serving.

• server/src/db/model-pricing.ts – Encrypted SQLite storage handling using AES-256-GCM envelopes.

• client/src/ – React/Vite dashboard source for key management and analytics visualization.

• desktop/src/main.ts – Electron main process that wraps the server for standalone desktop execution.

Summary

• FreeLLM API aggregates free-tier LLM providers into one OpenAI-compatible endpoint hosted at localhost:3001/v1.

• The router in server/src/services/router.ts automatically selects healthy models and handles up to 20 fallback attempts when providers fail or hit rate limits.

• Provider adapters in server/src/providers/*.ts normalize requests across different vendor APIs, while the health service continuously monitors key validity.

• Encrypted storage uses AES-256-GCM via server/src/db/model-pricing.ts, requiring a 32-byte ENCRYPTION_KEY in your environment configuration.

• Available as a Node.js server for production deployments or an Electron desktop app for local use with system tray monitoring.

Frequently Asked Questions

What is the difference between the server and desktop versions?

The server version is a standard Node.js/Express application designed for production deployments or custom hosting setups. The desktop version packages the same server inside an Electron application, providing a native system tray interface and automatic local execution without requiring manual Node.js installation. Both use identical routing logic from server/src/services/router.ts.

How does FreeLLM API handle rate limits from individual providers?

The system maintains an in-memory rate-limit ledger in server/src/services/ratelimit.ts that tracks requests per minute (RPM), requests per day (RPD), tokens per minute (TPM), and tokens per day (TPD) for each provider. When a provider returns HTTP 429 or 5xx errors, the service marks that key as rate-limited in the health status and enforces a cooldown period before retrying, automatically routing subsequent requests to the next available provider in the priority chain.

Can I force requests to use a specific provider instead of "auto"?

While the documentation emphasizes using "model": "auto" to leverage the intelligent routing system, you can specify individual models if configured in your provider adapters. The router in server/src/services/router.ts selects from available models based on the request parameters and health status. To use a specific provider, you would need to configure that provider's key in the dashboard and potentially modify the model name to match the specific provider's model identifier in your request.

Is my API key data secure when using FreeLLM API?

Yes. According to the source code in server/src/db/model-pricing.ts, all provider API keys are stored using AES-256-GCM encryption in a local SQLite database. The encryption key is never stored in the database itself; you must supply it via the ENCRYPTION_KEY environment variable defined in .env.example. This ensures that even if the database file is compromised, the keys remain encrypted without access to your environment configuration.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}