# Understanding the FreeLLMAPI Request Flow: From Client Call to Provider Response

> Explore the 10 stage FreeLLMAPI request flow. Learn how it intelligently routes OpenAI compatible requests to free LLM providers with fallback chains and secure key rotation.

- Repository: [Tashfeen/freellmapi](https://github.com/tashfeenahmed/freellmapi)
- Tags: architecture
- Published: 2026-07-01

---

**FreeLLMAPI processes OpenAI-compatible requests through a 10-stage pipeline that routes them to the best available free-tier LLM provider using intelligent fallback chains, sticky sessions, and encrypted key rotation, entirely within a local Express server.**

FreeLLMAPI is an open-source, self-hosted API gateway that unifies access to free-tier LLM providers behind a single OpenAI-compatible endpoint. Understanding the FreeLLMAPI request flow reveals how it intelligently selects providers, manages_failures, and preserves privacy by keeping all data—including encrypted API keys—on your local machine.


## Request Entry and Validation

The FreeLLMAPI request flow begins at the Express server entry point defined in [`server/src/index.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/index.ts). When a client sends a `POST` request to `/v1/chat/completions`, the server validates the unified API key, extracts the requested model name from the `model` field, and identifies special flags such as vision requirements or tool-calling support. This initial layer ensures only authenticated requests proceed to the routing engine while parsing the payload parameters needed for downstream decisions.


## Routing Chain Resolution

Once validated, the request enters the core routing logic in [`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts). The `resolveRoutingChain()` function determines which fallback chain to apply—whether using the current user profile, a named profile, or a global sorting strategy like `auto:fast`. This decision pulls enabled models from the local SQLite database tables `fallback_config` and `models`, establishing the candidate pool for this specific request.

The `orderChain()` function then ranks these candidates according to the selected routing strategy. Available strategies include `priority`, `balanced`, `smartest`, `fastest`, and `reliable`, or a custom weight vector defined by the user. Scoring incorporates recent success and failure statistics, latency measurements, intelligence rankings, and available token headroom to calculate the optimal attempt order.


## Session Pinning and Model Filtering

To maintain conversation consistency, FreeLLMAPI implements sticky-session logic within `routeRequest()` in [`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts). When a request includes an `X-Session-Id` header, the router attempts to reuse the same model (`preferredModelDbId`) used in previous turns of that conversation, preventing hallucination spikes that occur when switching models mid-dialogue.

Before attempting a provider, the router applies strict model-level filtering around lines 880–1000 of [`router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/router.ts). This validation checks:
- **Vision support**: Models without vision capabilities are excluded when images are present (`requireVision`)
- **Tool calling**: Models unable to handle function calls are filtered when `tools` are specified (`requireTools`)
- **Context window**: The model must accommodate the token count of the request within its `context_window`
- **Rate limits**: The request must fit within the model's per-minute token budget (`tpm_limit`)


## Encrypted Key Selection

For each viable model, `selectKeyForModel()` selects a single enabled and healthy API key. This function performs several critical checks:
- **Provider registration verification** via `hasProvider`
- **Round-robin rotation** across multiple keys using `roundRobinIndex`
- **Health checks** ensuring the key is not on cooldown (`isOnCooldown`) and can make requests (`canMakeRequest`, `canUseTokens`)
- **Decryption** using AES-256-GCM as implemented in [`server/src/lib/crypto.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/lib/crypto.ts)

This architecture ensures that raw provider keys never exist in memory unencrypted and distributes load while respecting rate limits.


## Provider Adapter Dispatch

With a concrete API key selected, the router instantiates the appropriate `BaseProvider` subclass and calls either `chatCompletion()` or `streamChatCompletion()`. Provider adapters located in `server/src/providers/*.ts`—such as [`google.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/google.ts) for Gemini or [`openai-compat.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/openai-compat.ts) for custom endpoints—translate the OpenAI-style payload into the upstream provider's native SDK calls. These adapters handle request transformation, response normalization, and streaming event formatting, ensuring the client receives a consistent interface regardless of which provider serves the request.


## Automatic Failover and Penalty Tracking

If the upstream provider returns a **429** (rate limit), **5xx** error, or times out, FreeLLMAPI's resilience mechanism activates. The `recordRateLimitHit()` function in [`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts) applies a temporary penalty to the model, demoting it in future rankings, and places the specific API key on cooldown. The router immediately retries the next model in the ordered chain. Successful calls trigger `recordSuccess()`, which clears accumulated penalties for that provider.

This failover loop continues until a provider responds successfully or the chain is exhausted, with each attempt logged via the `X-Fallback-Attempts` response header.


## Response Composition and Headers

After receiving a valid response from the upstream provider, the Express middleware in [`server/src/index.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/index.ts) adds diagnostic headers before sending data back to the client:
- `X-Routed-Via: <platform>/<model>` identifies the specific provider and model that served the request
- `X-Fallback-Attempts` indicates how many additional models were tried before success

For streaming requests, Server-Sent Events (SSE) are forwarded in real-time with these headers attached to the initial HTTP response. The response body remains unchanged from the provider's format, maintaining OpenAI compatibility.


## Local Analytics and Statistics

Every request populates the `requests` table in the local SQLite database with status codes, token usage, latency, and time-to-first-byte (TTFB) metrics. The scoring system periodically refreshes a decay-weighted statistics cache via `refreshStatsCache()` in [`server/src/services/scoring.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/scoring.ts), feeding updated reliability scores (calculated using Beta-posterior distributions) and speed metrics back into the routing algorithm. This closed-loop system ensures routing decisions improve based on actual provider performance without transmitting analytics to external services.


## Implementation Examples

The following examples demonstrate how clients interact with the FreeLLMAPI request flow.

**Python (OpenAI SDK)**

```python
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:3001/v1",
    api_key="freellmapi-xxxxxxxxxxxxxxxxxxxx",
)

resp = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Explain free-tier LLMs in one sentence."}],
)

print(resp.choices[0].message.content)
print("Routed via:", resp.headers.get("x-routed-via"))

```

**cURL**

```bash
curl http://localhost:3001/v1/chat/completions \
  -H "Authorization: Bearer freellmapi-xxxxxxxxxxxxxxxxxxxx" \
  -H "Content-Type: application/json" \
  -d '{
        "model": "auto",
        "messages": [{"role": "user", "content": "What is the capital of France?"}]
      }'

```

**Image Generation with Session Pinning**

```python
resp = client.images.generations.create(
    model="auto",
    prompt="A cat riding a skateboard",
    extra_headers={"X-Session-Id": "conversation-123"}
)
print(resp.data[0].url)

```


## Summary

- **FreeLLMAPI** operates as a local Express server that proxies OpenAI-compatible requests to free-tier providers without external data exposure.
- **Routing decisions** rely on `resolveRoutingChain()` and `orderChain()` in [`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts), which score models using real-time latency, reliability, and intelligence metrics.
- **Session pinning** via `X-Session-Id` headers prevents model switching mid-conversation by preferring `preferredModelDbId` for existing sessions.
- **Security** is maintained through AES-256-GCM encryption of provider keys in SQLite and round-robin key rotation with health checks.
- **Automatic failover** triggers `recordRateLimitHit()` on 429/5xx errors, immediately attempting the next model in the ranked chain and exposing routing metadata via `X-Routed-Via` headers.
- **Privacy** is preserved by storing all configuration, keys, and analytics in local SQLite, ensuring no external service sees your prompts or provider credentials.


## Frequently Asked Questions

### How does FreeLLMAPI decide which provider to use for a request?

FreeLLMAPI uses the `orderChain()` function in [`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts) to rank available models based on your selected routing strategy. Strategies like `fastest`, `smartest`, or `balanced` combine recent telemetry—including latency measurements, success rates calculated via Beta-posterior distributions, and token headroom—to produce an ordered list of candidates. The router then attempts each model in sequence until one succeeds.

### What happens if a provider returns a rate limit error?

When a provider returns a **429** or **5xx** error, the `recordRateLimitHit()` function applies a temporary penalty to that model and places the specific API key on cooldown. The router immediately retries the request with the next model in the ordered chain. Successful requests trigger `recordSuccess()` to clear penalties. The number of fallback attempts is returned in the `X-Fallback-Attempts` header.

### How does FreeLLMAPI keep my API keys secure?

All provider API keys are stored in SQLite using AES-256-GCM encryption implemented in [`server/src/lib/crypto.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/lib/crypto.ts). The `selectKeyForModel()` function decrypts keys only when needed for a specific request, and they never exist in plaintext in memory longer than necessary. Additionally, the system supports round-robin rotation across multiple keys per provider to distribute load and limit exposure.

### Can I force FreeLLMAPI to use a specific model instead of auto-routing?

Yes. While you can specify `"model": "auto"` to let the router decide based on current statistics, you can also request a specific model name (e.g., `"model": "gemini-pro"` or a custom profile name). The `resolveRoutingChain()` function will respect your selection and either use that specific model if available or return an error if it does not meet the request requirements (vision, tool support, or context window).