Understanding the FreeLLMAPI Request Flow: From Client Call to Provider Response

FreeLLMAPI processes OpenAI-compatible requests through a 10-stage pipeline that routes them to the best available free-tier LLM provider using intelligent fallback chains, sticky sessions, and encrypted key rotation, entirely within a local Express server.

FreeLLMAPI is an open-source, self-hosted API gateway that unifies access to free-tier LLM providers behind a single OpenAI-compatible endpoint. Understanding the FreeLLMAPI request flow reveals how it intelligently selects providers, manages_failures, and preserves privacy by keeping all data—including encrypted API keys—on your local machine.

Request Entry and Validation

The FreeLLMAPI request flow begins at the Express server entry point defined in server/src/index.ts. When a client sends a POST request to /v1/chat/completions, the server validates the unified API key, extracts the requested model name from the model field, and identifies special flags such as vision requirements or tool-calling support. This initial layer ensures only authenticated requests proceed to the routing engine while parsing the payload parameters needed for downstream decisions.

Routing Chain Resolution

Once validated, the request enters the core routing logic in server/src/services/router.ts. The resolveRoutingChain() function determines which fallback chain to apply—whether using the current user profile, a named profile, or a global sorting strategy like auto:fast. This decision pulls enabled models from the local SQLite database tables fallback_config and models, establishing the candidate pool for this specific request.

The orderChain() function then ranks these candidates according to the selected routing strategy. Available strategies include priority, balanced, smartest, fastest, and reliable, or a custom weight vector defined by the user. Scoring incorporates recent success and failure statistics, latency measurements, intelligence rankings, and available token headroom to calculate the optimal attempt order.

Session Pinning and Model Filtering

To maintain conversation consistency, FreeLLMAPI implements sticky-session logic within routeRequest() in server/src/services/router.ts. When a request includes an X-Session-Id header, the router attempts to reuse the same model (preferredModelDbId) used in previous turns of that conversation, preventing hallucination spikes that occur when switching models mid-dialogue.

Before attempting a provider, the router applies strict model-level filtering around lines 880–1000 of router.ts. This validation checks:

• Vision support: Models without vision capabilities are excluded when images are present (requireVision)
• Tool calling: Models unable to handle function calls are filtered when tools are specified (requireTools)
• Context window: The model must accommodate the token count of the request within its context_window
• Rate limits: The request must fit within the model's per-minute token budget (tpm_limit)

Encrypted Key Selection

For each viable model, selectKeyForModel() selects a single enabled and healthy API key. This function performs several critical checks:

• Provider registration verification via hasProvider
• Round-robin rotation across multiple keys using roundRobinIndex
• Health checks ensuring the key is not on cooldown (isOnCooldown) and can make requests (canMakeRequest, canUseTokens)
• Decryption using AES-256-GCM as implemented in server/src/lib/crypto.ts

This architecture ensures that raw provider keys never exist in memory unencrypted and distributes load while respecting rate limits.

Provider Adapter Dispatch

With a concrete API key selected, the router instantiates the appropriate BaseProvider subclass and calls either chatCompletion() or streamChatCompletion(). Provider adapters located in server/src/providers/*.ts—such as google.ts for Gemini or openai-compat.ts for custom endpoints—translate the OpenAI-style payload into the upstream provider's native SDK calls. These adapters handle request transformation, response normalization, and streaming event formatting, ensuring the client receives a consistent interface regardless of which provider serves the request.

Automatic Failover and Penalty Tracking

If the upstream provider returns a 429 (rate limit), 5xx error, or times out, FreeLLMAPI's resilience mechanism activates. The recordRateLimitHit() function in server/src/services/router.ts applies a temporary penalty to the model, demoting it in future rankings, and places the specific API key on cooldown. The router immediately retries the next model in the ordered chain. Successful calls trigger recordSuccess(), which clears accumulated penalties for that provider.

This failover loop continues until a provider responds successfully or the chain is exhausted, with each attempt logged via the X-Fallback-Attempts response header.

Response Composition and Headers

After receiving a valid response from the upstream provider, the Express middleware in server/src/index.ts adds diagnostic headers before sending data back to the client:

• X-Routed-Via: <platform>/<model> identifies the specific provider and model that served the request
• X-Fallback-Attempts indicates how many additional models were tried before success

For streaming requests, Server-Sent Events (SSE) are forwarded in real-time with these headers attached to the initial HTTP response. The response body remains unchanged from the provider's format, maintaining OpenAI compatibility.

Local Analytics and Statistics

Every request populates the requests table in the local SQLite database with status codes, token usage, latency, and time-to-first-byte (TTFB) metrics. The scoring system periodically refreshes a decay-weighted statistics cache via refreshStatsCache() in server/src/services/scoring.ts, feeding updated reliability scores (calculated using Beta-posterior distributions) and speed metrics back into the routing algorithm. This closed-loop system ensures routing decisions improve based on actual provider performance without transmitting analytics to external services.

Implementation Examples

The following examples demonstrate how clients interact with the FreeLLMAPI request flow.

Python (OpenAI SDK)

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:3001/v1",
    api_key="freellmapi-xxxxxxxxxxxxxxxxxxxx",
)

resp = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Explain free-tier LLMs in one sentence."}],
)

print(resp.choices[0].message.content)
print("Routed via:", resp.headers.get("x-routed-via"))

cURL

curl http://localhost:3001/v1/chat/completions \
  -H "Authorization: Bearer freellmapi-xxxxxxxxxxxxxxxxxxxx" \
  -H "Content-Type: application/json" \
  -d '{
        "model": "auto",
        "messages": [{"role": "user", "content": "What is the capital of France?"}]
      }'

Image Generation with Session Pinning

resp = client.images.generations.create(
    model="auto",
    prompt="A cat riding a skateboard",
    extra_headers={"X-Session-Id": "conversation-123"}
)
print(resp.data[0].url)

Summary

• FreeLLMAPI operates as a local Express server that proxies OpenAI-compatible requests to free-tier providers without external data exposure.
• Routing decisions rely on resolveRoutingChain() and orderChain() in server/src/services/router.ts, which score models using real-time latency, reliability, and intelligence metrics.
• Session pinning via X-Session-Id headers prevents model switching mid-conversation by preferring preferredModelDbId for existing sessions.
• Security is maintained through AES-256-GCM encryption of provider keys in SQLite and round-robin key rotation with health checks.
• Automatic failover triggers recordRateLimitHit() on 429/5xx errors, immediately attempting the next model in the ranked chain and exposing routing metadata via X-Routed-Via headers.
• Privacy is preserved by storing all configuration, keys, and analytics in local SQLite, ensuring no external service sees your prompts or provider credentials.

Frequently Asked Questions

How does FreeLLMAPI decide which provider to use for a request?

FreeLLMAPI uses the orderChain() function in server/src/services/router.ts to rank available models based on your selected routing strategy. Strategies like fastest, smartest, or balanced combine recent telemetry—including latency measurements, success rates calculated via Beta-posterior distributions, and token headroom—to produce an ordered list of candidates. The router then attempts each model in sequence until one succeeds.

What happens if a provider returns a rate limit error?

When a provider returns a 429 or 5xx error, the recordRateLimitHit() function applies a temporary penalty to that model and places the specific API key on cooldown. The router immediately retries the request with the next model in the ordered chain. Successful requests trigger recordSuccess() to clear penalties. The number of fallback attempts is returned in the X-Fallback-Attempts header.

How does FreeLLMAPI keep my API keys secure?

All provider API keys are stored in SQLite using AES-256-GCM encryption implemented in server/src/lib/crypto.ts. The selectKeyForModel() function decrypts keys only when needed for a specific request, and they never exist in plaintext in memory longer than necessary. Additionally, the system supports round-robin rotation across multiple keys per provider to distribute load and limit exposure.

Can I force FreeLLMAPI to use a specific model instead of auto-routing?

Yes. While you can specify "model": "auto" to let the router decide based on current statistics, you can also request a specific model name (e.g., "model": "gemini-pro" or a custom profile name). The resolveRoutingChain() function will respect your selection and either use that specific model if available or return an error if it does not meet the request requirements (vision, tool support, or context window).

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}