How FreeLLMAPI's Routing Engine Works: Architecture & Source Code Deep Dive

FreeLLMAPI's routing engine implements a multi-armed bandit algorithm that dynamically scores candidate models on intelligence, speed, and available headroom while applying real-time penalties for rate limits and automatic fallback chains to maximize free-tier availability.

FreeLLMAPI (tashfeenahmed/freellmapi) is an open-source gateway that aggregates multiple free LLM providers behind a unified API. At its core lies a sophisticated routing engine in server/src/services/router.ts that decides which provider and model should handle each incoming request. This system continuously balances model quality against quota constraints, applying dynamic penalties and intelligent fallbacks to maintain high availability without incurring costs.

Core Architecture and Request Lifecycle

The routing engine operates as a stateful selector that treats each provider-model combination as an arm in a multi-armed bandit problem. When a request arrives, the engine executes a ten-step evaluation pipeline to determine the optimal endpoint.

1. Token Reservation and Budget Protection

Before evaluating candidates, the router reserves a safety buffer of output tokens to prevent large max_tokens requests from exhausting the free pool prematurely. The routingReserveTokens function calculates this reserved amount:

export function routingReserveTokens(requestedMaxTokens: number | null | undefined): number

This reservation ensures that routing decisions account for actual token consumption before committing to a specific provider.

2. Candidate Chain Construction

The engine queries the internal catalog for all models matching the request requirements (platform, size label, vision/tools support). Each candidate is represented as a ChainRow interface containing static metadata including priority tiers, limits, and associated key_id values.

3. Multi-Factor Scoring Algorithm

Scoring combines intelligence ratings, latency benchmarks, and available quota headroom. The [scoring.ts](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/scoring.ts) module exports several key functions:

• intelligenceScore – Ranks models by capability tiers
• speedScore – Factors in historical response latency
• headroomFactor – Calculates remaining daily budget ratios
• combineScore – Aggregates metrics into a final selection weight

4. Dynamic Penalty Application

The router maintains a penalty system that decays over time to track provider health. According to the source code in [router.ts](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts#L71-L98):

• Rate-limit penalties (PENALTY_PER_429) are applied when providers return HTTP 429 errors
• Failure penalties (PENALTY_PER_FAIL) handle non-limit errors (5xx, timeouts)
• Decay interval (DECAY_INTERVAL_MS) reduces penalties every two minutes

Functions like recordRateLimitHit and recordModelFailure update these weights in real-time.

5. Rate Limit and Cooldown Validation

Before selection, the engine validates per-key quotas using [ratelimit.ts](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/ratelimit.ts). The canMakeRequest function checks RPM, RPD, TPM, and TPD limits, while isOnCooldown verifies temporary exclusion periods following failures.

6. Selection and Intelligent Fallback

The primary entry point routeRequest executes the selection logic:

export function routeRequest(
  estimatedTokens: number,
  skipKeys?: Set<string>,
  preferredModelDbId?: string,
  requireVision?: boolean,
  requireTools?: boolean
): RouteResult

If the selected provider returns a 429, 5xx error, or times out, the router implements an automatic fallback chain. As documented in [architecture.md](https://github.com/tashfeenahmed/freellmapi/blob/main/docs/architecture.md#L48-L52), the system retries with the next best candidate for up to 20 attempts, bounded by a wall-clock retry budget to prevent infinite loops.

7. Headroom and Peak Hour Adjustments

The engine applies dynamic modifiers based on remaining quotas. When daily limits approach exhaustion, getHeadroomThresholds reduces a model's score according to configurable routing_headroom_ramp_start and routing_headroom_floor parameters. Additional peak-hour settings can demote specific models during high-traffic periods.

8. Exploration Mechanism

To ensure new models receive traffic for training data, the router includes an exploration mode. When enabled, EXPLORE_CHANCE (default 0.1) forces a random unmeasured model selection 10% of the time, provided the model has fewer than EXPLORE_MIN_SAMPLES recorded requests.

9. Custom Weight Overrides

Users can persist custom routing strategies via CUSTOM_WEIGHTS_KEY and STRATEGY_KEY settings. The applyModelWeightOverride function applies these user-defined preferences to the base scoring algorithm.

Key Implementation Files

The routing system spans several specialized modules:

• server/src/services/router.ts – Core engine, penalty tracking, and routeRequest implementation
• server/src/services/scoring.ts – Intelligence, speed, and headroom calculations
• server/src/services/ratelimit.ts – RPM/RPD validation, cooldown checks, and lease management via acquireLease
• server/src/services/provider-quota.ts – Quota pool derivation and daily limit tracking
• server/src/services/model-groups.ts – Model grouping logic and unified routing flags

Practical Code Examples

Direct Router Invocation

To programmatically route a request within a custom handler:

import { routeRequest, routingReserveTokens } from './services/router.js';

async function handleChatCompletion(requestBody: any) {
  // Reserve tokens for routing calculation
  const reserve = routingReserveTokens(requestBody.max_tokens);
  
  // Route 1500 tokens + reserve buffer
  const result = routeRequest(
    1500 + reserve,
    undefined,  // skipKeys
    undefined,  // preferredModelDbId
    false,      // requireVision
    false       // requireTools
  );

  // Use the selected provider
  const response = await result.provider.callChat({
    model: result.modelId,
    messages: requestBody.messages,
    apiKey: result.apiKey
  });

  // Release the key lease for reuse
  result.release?.();
  return response;
}

Handling Exhaustion Errors

When all providers are unavailable, inspect diagnostics:

import { summarizeExhaustion, RouteError } from './services/router.js';

try {
  const route = routeRequest(2000);
} catch (error) {
  if (error instanceof RouteError) {
    const diagnosis = summarizeExhaustion(error.diagnostics, error.status);
    console.error('Routing failure:', diagnosis);
    // Output: "All 12 providers exhausted: 8 rate-limited, 4 unhealthy"
  }
}

Summary

• FreeLLMAPI's routing engine implements a bandit-style selection algorithm that continuously learns from provider performance
• Token reservation prevents premature quota exhaustion by reserving output buffers before routing decisions
• Multi-factor scoring weighs intelligence, latency, and available headroom to rank candidates
• Dynamic penalties track 429 errors and failures, decaying every two minutes to reflect recovered providers
• Automatic fallback chains retry failed requests across up to 20 alternative models with wall-clock timeouts
• Exploration mode ensures new models receive 10% of traffic for performance benchmarking
• Custom weights allow users to override default scoring via persisted strategy configurations

Frequently Asked Questions

How does FreeLLMAPI handle rate limit errors from providers?

When a provider returns HTTP 429, the router triggers recordRateLimitHit to apply PENALTY_PER_429 to that model's score. The system simultaneously places the API key on cooldown using isOnCooldown and immediately retries the request with the next highest-scoring candidate in the chain.

What happens when all available models are exhausted?

If the router depletes its fallback chain (up to 20 attempts), it throws a RouteError containing detailed diagnostics. The summarizeExhaustion function generates a human-readable summary indicating how many providers were rate-limited versus unhealthy, allowing clients to implement circuit breaker patterns.

Can users customize which models the router prefers?

Yes. Users can define custom weight presets via BANDIT_PRESETS and per-model overrides through applyModelWeightOverride. These configurations persist in the settings table under CUSTOM_WEIGHTS_KEY and modify the base scoring algorithm without requiring code changes.

How does the exploration mechanism ensure new models get tested?

The router checks EXPLORE_CHANCE (default 0.1) before final selection. If triggered and the candidate model has fewer than EXPLORE_MIN_SAMPLES recorded requests, the system overrides the normal scoring to select that model. This ensures new providers accumulate sufficient latency and success metrics for accurate future scoring.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}