How FreeLLMAPI's Router Selects Models and Handles Automatic Fallback

FreeLLMAPI routes every request through a dynamically scored fallback chain that ranks enabled models by reliability, speed, and intelligence, then performs automatic fallback to the next candidate if API keys are exhausted or rate-limited.

The open-source FreeLLMAPI project (tashfeenahmed/freellmapi) implements a sophisticated routing layer that automatically selects the optimal large language model for each request. Located primarily in server/src/services/router.ts and server/src/services/scoring.ts, the system combines multi-armed bandit algorithms with real-time availability checks to build a prioritized fallback chain. This ensures high availability by gracefully degrading to alternative models when primary options fail.

Building the Active Fallback Chain

The routing process begins by fetching the current configuration from the database. According to the tashfeenahmed/freellmapi source code, the function getActiveChain(db) retrieves either a profile-specific chain or the global fallback configuration, returning an ordered list of enabled models and their associated API keys.

const chain = getActiveChain(db);  // server/src/services/router.ts

This chain represents the complete pool of available models. The router evaluates this list fresh on every request to ensure it reflects the latest state, including newly added keys, updated quotas, or changed routing strategies.

Bandit Scoring and Routing Strategies

Once the chain is loaded, the orderChain(chain, strategy) function applies the configured routing strategy to score and sort every model. FreeLLMAPI supports six built-in strategies: priority, balanced, smartest, fastest, reliable, and custom user-defined weight vectors.

The scoring engine in server/src/services/scoring.ts calculates raw scores using a bandit algorithm. Reliability is computed from a Beta posterior distribution using Thompson sampling for live routing, while dashboard displays use the deterministic expected value. Speed combines throughput metrics with first-byte latency. Intelligence uses a tier-based ranking system. Two guard-rails—a head-room check and a rate-limit penalty—demote models nearing quota or currently returning HTTP 429 errors.

const sortedChain = orderChain(chain, strategy);  // server/src/services/router.ts

The sorted chain places the highest-scoring candidate first, ensuring the "best" available model is attempted according to the chosen strategy.

API Key Selection and Round-Robin Logic

For each candidate model in the sorted chain, the router executes selectKeyForModel(entry, estimatedTokens, skipKeys, diag) to choose a specific API key. This function, also defined in server/src/services/router.ts, implements round-robin selection across all enabled keys for that model.

The selection logic performs several critical checks:

• Cooldown status: Keys in an active cooldown period are skipped
• Rate limits: Per-minute and per-day quotas are validated against server/src/services/ratelimit.ts functions
• Decryption failures: Keys that cannot be decrypted are bypassed
• Token capacity: The estimatedTokens parameter ensures the key’s remaining quota can handle the request

The first usable key encountered becomes the RouteResult. If no key passes these checks, the function returns null, triggering the fallback mechanism.

Automatic Fallback Execution

Automatic fallback occurs when selectKeyForModel returns null or when a downstream error forces a retry. The router implements this as a sequential loop over the sortedChain:

for (const entry of sortedChain) {
  // Vision, tools, and context capability checks occur here
  const route = selectKeyForModel(entry, estimatedTokens, skipKeys, diag);
  if (route) return route;  // Success
}
throw new RouteError('All models exhausted …', 429, diag);  // server/src/services/router.ts

When a model fails due to rate limiting, the router calls recordRateLimitHit() to apply a penalty that pushes the model down the priority list for subsequent requests. These penalties decay every 2 minutes, allowing a model to regain its position once limits clear. If the entire chain is exhausted without finding a viable route, the system throws a RouteError with status code 429.

Sticky Sessions and Pinned Routing

FreeLLMAPI supports session affinity through pinned routing. When a preferredModelDbId is provided (for example, from a previous chat turn), the router moves that specific model to the front of the chain using the routePinnedModel helper.

Unlike the standard fallback chain, pinned routing rotates only across the API keys of the specified model. If all keys for that model are exhausted, routePinnedModel returns null rather than falling back to a different model, allowing the caller to decide whether to trigger a full re-route.

// Pin to model DB ID 42 from previous turn
const pinnedResult = routePinnedModel(42, 1200);
if (pinnedResult) {
  console.log('Pinned model served request');
} else {
  console.log('Pinned model unavailable');
}

Implementation Examples

Automatic Routing with Default Strategy

The routeRequest function encapsulates the full selection logic. It builds the active chain, applies the default balanced strategy, and returns the first viable model/key pair.

import { routeRequest } from '@freellmapi/server';

// Estimate 1500 tokens for prompt + max completion
const result = routeRequest(1500);
console.log(`Using provider ${result.provider.name}, model ${result.modelId}`);

Custom Weight Vectors

Users can override the scoring weights to favor specific characteristics. The setCustomWeights function persists the configuration to CUSTOM_WEIGHTS_KEY and applies it to future requests.

import { setCustomWeights, routeRequest } from '@freellmapi/server';

// Prioritize speed over reliability: 0.2 reliability, 0.7 speed, 0.1 intelligence
setCustomWeights({ reliability: 0.2, speed: 0.7, intelligence: 0.1 });

const res = routeRequest(800);
console.log(`Chosen model: ${res.displayName}`);

Summary

• Dynamic chains: The router fetches fresh configurations via getActiveChain() on every request to ensure current availability.
• Intelligent scoring: Multi-armed bandit algorithms in scoring.ts rank models by reliability (Beta posterior), speed (latency/throughput), and intelligence tiers.
• Automatic fallback: A sequential loop in router.ts tries each model in ranked order, with 429 penalties decaying every 2 minutes.
• Resource-aware selection: selectKeyForModel validates rate limits, cooldowns, and token capacity before committing to a key.
• Session pinning: routePinnedModel isolates routing to a specific model’s keys without cross-model fallback.

Frequently Asked Questions

What routing strategies does FreeLLMAPI support?

FreeLLMAPI supports six strategies: priority (manual ordering), balanced (equal weighting), smartest (intelligence-weighted), fastest (latency-weighted), reliable (uptime-weighted), and custom (user-defined vector). Custom weights are persisted to CUSTOM_WEIGHTS_KEY and allow fine-tuning of the reliability, speed, and intelligence multipliers.

How does the router handle rate limits and cooldowns?

The selectKeyForModel function checks per-minute and per-day limits via server/src/services/ratelimit.ts. Keys exceeding these limits or in an active cooldown are skipped. When a rate limit is hit, recordRateLimitHit() applies a penalty score that decays over 2 minutes, temporarily demoting the affected model in the fallback chain.

What is the difference between automatic fallback and pinned routing?

Automatic fallback iterates through the entire sorted chain of models until any viable key is found. Pinned routing via routePinnedModel restricts selection to keys of a specific model (identified by preferredModelDbId) and returns null if that model is unavailable, preventing automatic fallback to alternative models.

How often does the scoring penalty decay for rate-limited models?

Penalty weights applied after rate-limit hits decay every 2 minutes. This allows the bandit scoring engine to automatically restore a model's priority once its quota resets or cooldown expires, ensuring the fallback chain reflects real-time capacity without manual intervention.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}