# How FreeLLMAPI's Routing Engine Works: Architecture & Source Code Deep Dive

> Explore FreeLLMAPI's routing engine architecture and source code. Learn how its multi-armed bandit algorithm optimizes free-tier LLM availability, ensuring speed and intelligence.

- Repository: [Tashfeen/freellmapi](https://github.com/tashfeenahmed/freellmapi)
- Tags: architecture
- Published: 2026-08-27

---

**FreeLLMAPI's routing engine implements a multi-armed bandit algorithm that dynamically scores candidate models on intelligence, speed, and available headroom while applying real-time penalties for rate limits and automatic fallback chains to maximize free-tier availability.**

FreeLLMAPI (tashfeenahmed/freellmapi) is an open-source gateway that aggregates multiple free LLM providers behind a unified API. At its core lies a sophisticated routing engine in [`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts) that decides which provider and model should handle each incoming request. This system continuously balances model quality against quota constraints, applying dynamic penalties and intelligent fallbacks to maintain high availability without incurring costs.

## Core Architecture and Request Lifecycle

The routing engine operates as a stateful selector that treats each provider-model combination as an arm in a multi-armed bandit problem. When a request arrives, the engine executes a ten-step evaluation pipeline to determine the optimal endpoint.

### 1. Token Reservation and Budget Protection

Before evaluating candidates, the router reserves a safety buffer of output tokens to prevent large `max_tokens` requests from exhausting the free pool prematurely. The [`routingReserveTokens`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts#L47-L50) function calculates this reserved amount:

```typescript
export function routingReserveTokens(requestedMaxTokens: number | null | undefined): number

```

This reservation ensures that routing decisions account for actual token consumption before committing to a specific provider.

### 2. Candidate Chain Construction

The engine queries the internal catalog for all models matching the request requirements (platform, size label, vision/tools support). Each candidate is represented as a [`ChainRow`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts#L41-L72) interface containing static metadata including priority tiers, limits, and associated `key_id` values.

### 3. Multi-Factor Scoring Algorithm

Scoring combines intelligence ratings, latency benchmarks, and available quota headroom. The [[`scoring.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/scoring.ts)](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/scoring.ts) module exports several key functions:

- **`intelligenceScore`** – Ranks models by capability tiers
- **`speedScore`** – Factors in historical response latency  
- **`headroomFactor`** – Calculates remaining daily budget ratios
- **`combineScore`** – Aggregates metrics into a final selection weight

### 4. Dynamic Penalty Application

The router maintains a penalty system that decays over time to track provider health. According to the source code in [[`router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/router.ts)](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts#L71-L98):

- **Rate-limit penalties** (`PENALTY_PER_429`) are applied when providers return HTTP 429 errors
- **Failure penalties** (`PENALTY_PER_FAIL`) handle non-limit errors (5xx, timeouts)
- **Decay interval** (`DECAY_INTERVAL_MS`) reduces penalties every two minutes

Functions like [`recordRateLimitHit`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts#L71-L98) and [`recordModelFailure`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts#L71-L98) update these weights in real-time.

### 5. Rate Limit and Cooldown Validation

Before selection, the engine validates per-key quotas using [[`ratelimit.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/ratelimit.ts)](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/ratelimit.ts). The [`canMakeRequest`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/ratelimit.ts) function checks RPM, RPD, TPM, and TPD limits, while [`isOnCooldown`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/ratelimit.ts) verifies temporary exclusion periods following failures.

### 6. Selection and Intelligent Fallback

The primary entry point [`routeRequest`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts#L1881-L1890) executes the selection logic:

```typescript
export function routeRequest(
  estimatedTokens: number,
  skipKeys?: Set<string>,
  preferredModelDbId?: string,
  requireVision?: boolean,
  requireTools?: boolean
): RouteResult

```

If the selected provider returns a 429, 5xx error, or times out, the router implements an automatic fallback chain. As documented in [[`architecture.md`](https://github.com/tashfeenahmed/freellmapi/blob/main/architecture.md)](https://github.com/tashfeenahmed/freellmapi/blob/main/docs/architecture.md#L48-L52), the system retries with the next best candidate for up to 20 attempts, bounded by a wall-clock retry budget to prevent infinite loops.

### 7. Headroom and Peak Hour Adjustments

The engine applies dynamic modifiers based on remaining quotas. When daily limits approach exhaustion, [`getHeadroomThresholds`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts#L59-L68) reduces a model's score according to configurable `routing_headroom_ramp_start` and `routing_headroom_floor` parameters. Additional peak-hour settings can demote specific models during high-traffic periods.

### 8. Exploration Mechanism

To ensure new models receive traffic for training data, the router includes an exploration mode. When enabled, [`EXPLORE_CHANCE`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts#L86-L94) (default 0.1) forces a random unmeasured model selection 10% of the time, provided the model has fewer than `EXPLORE_MIN_SAMPLES` recorded requests.

### 9. Custom Weight Overrides

Users can persist custom routing strategies via [`CUSTOM_WEIGHTS_KEY`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts#L42-L47) and [`STRATEGY_KEY`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts#L42-L47) settings. The `applyModelWeightOverride` function applies these user-defined preferences to the base scoring algorithm.

## Key Implementation Files

The routing system spans several specialized modules:

- **[`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts)** – Core engine, penalty tracking, and [`routeRequest`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts#L1881) implementation
- **[`server/src/services/scoring.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/scoring.ts)** – Intelligence, speed, and headroom calculations  
- **[`server/src/services/ratelimit.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/ratelimit.ts)** – RPM/RPD validation, cooldown checks, and lease management via [`acquireLease`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/ratelimit.ts)
- **[`server/src/services/provider-quota.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/provider-quota.ts)** – Quota pool derivation and daily limit tracking
- **[`server/src/services/model-groups.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/model-groups.ts)** – Model grouping logic and unified routing flags

## Practical Code Examples

### Direct Router Invocation

To programmatically route a request within a custom handler:

```typescript
import { routeRequest, routingReserveTokens } from './services/router.js';

async function handleChatCompletion(requestBody: any) {
  // Reserve tokens for routing calculation
  const reserve = routingReserveTokens(requestBody.max_tokens);
  
  // Route 1500 tokens + reserve buffer
  const result = routeRequest(
    1500 + reserve,
    undefined,  // skipKeys
    undefined,  // preferredModelDbId
    false,      // requireVision
    false       // requireTools
  );

  // Use the selected provider
  const response = await result.provider.callChat({
    model: result.modelId,
    messages: requestBody.messages,
    apiKey: result.apiKey
  });

  // Release the key lease for reuse
  result.release?.();
  return response;
}

```

### Handling Exhaustion Errors

When all providers are unavailable, inspect diagnostics:

```typescript
import { summarizeExhaustion, RouteError } from './services/router.js';

try {
  const route = routeRequest(2000);
} catch (error) {
  if (error instanceof RouteError) {
    const diagnosis = summarizeExhaustion(error.diagnostics, error.status);
    console.error('Routing failure:', diagnosis);
    // Output: "All 12 providers exhausted: 8 rate-limited, 4 unhealthy"
  }
}

```

## Summary

- **FreeLLMAPI's routing engine** implements a bandit-style selection algorithm that continuously learns from provider performance
- **Token reservation** prevents premature quota exhaustion by reserving output buffers before routing decisions
- **Multi-factor scoring** weighs intelligence, latency, and available headroom to rank candidates  
- **Dynamic penalties** track 429 errors and failures, decaying every two minutes to reflect recovered providers
- **Automatic fallback** chains retry failed requests across up to 20 alternative models with wall-clock timeouts
- **Exploration mode** ensures new models receive 10% of traffic for performance benchmarking
- **Custom weights** allow users to override default scoring via persisted strategy configurations

## Frequently Asked Questions

### How does FreeLLMAPI handle rate limit errors from providers?

When a provider returns HTTP 429, the router triggers [`recordRateLimitHit`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts#L71-L98) to apply `PENALTY_PER_429` to that model's score. The system simultaneously places the API key on cooldown using [`isOnCooldown`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/ratelimit.ts) and immediately retries the request with the next highest-scoring candidate in the chain.

### What happens when all available models are exhausted?

If the router depletes its fallback chain (up to 20 attempts), it throws a [`RouteError`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts#L1881-L1890) containing detailed diagnostics. The [`summarizeExhaustion`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts#L80-L100) function generates a human-readable summary indicating how many providers were rate-limited versus unhealthy, allowing clients to implement circuit breaker patterns.

### Can users customize which models the router prefers?

Yes. Users can define custom weight presets via `BANDIT_PRESETS` and per-model overrides through `applyModelWeightOverride`. These configurations persist in the settings table under [`CUSTOM_WEIGHTS_KEY`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts#L42-L47) and modify the base scoring algorithm without requiring code changes.

### How does the exploration mechanism ensure new models get tested?

The router checks [`EXPLORE_CHANCE`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts#L86-L94) (default 0.1) before final selection. If triggered and the candidate model has fewer than `EXPLORE_MIN_SAMPLES` recorded requests, the system overrides the normal scoring to select that model. This ensures new providers accumulate sufficient latency and success metrics for accurate future scoring.