How FreeLLMAPI Implements Automatic Failover for Providers: A Technical Deep Dive

FreeLLMAPI guarantees that every request reaches a functional LLM by maintaining a dynamic fallback chain that automatically reorders providers based on real-time health metrics and retries with the next viable entry when a provider fails.

FreeLLMAPI is an open-source routing layer that sits between your application and multiple LLM providers, ensuring high availability through intelligent automatic failover. The system implements a sophisticated fallback mechanism that dynamically prioritizes providers based on reliability, speed, and rate-limit status. This article examines how the tashfeenahmed/freellmapi repository implements automatic failover for providers using a combination of database-backed configuration, real-time scoring, and resilient request routing.

The Fallback Chain Architecture

Database Schema and Chain Construction

The foundation of automatic failover for providers rests on a persistent fallback chain stored in the database. All enabled models are linked to the fallback_config table, which holds the priority and enabled flag for each model. In server/src/services/router.ts, the getActiveChain() function constructs the active chain by joining these tables and filtering for enabled models.

// router.ts – build the active chain
function getActiveChain(db: Database): ChainRow[] {
  …
  return db.prepare(`
    SELECT fc.model_db_id, fc.priority, fc.enabled,
           m.platform, m.model_id, m.display_name, …
    FROM fallback_config fc
    JOIN models m ON m.id = fc.model_db_id AND m.enabled = 1
    ORDER BY fc.priority ASC
  `).all() as ChainRow[];
}

The result is an array of ChainRow objects containing model metadata and priority values, which serves as the ordered list for routing decisions.

Dynamic Routing Strategies

FreeLLMAPI does not rely solely on static priorities. The chain can be ordered according to several routing strategies: priority, balanced, fastest, reliable, smartest, or custom. When a non-priority strategy is selected, the orderChain() function in router.ts scores each model using scoreChainEntry() and sorts by the resulting composite score.

// router.ts – order the chain
function orderChain(chain: ChainRow[], strategy: RoutingStrategy, sampled = true): ChainRow[] {
  const weights = weightsFor(strategy);
  …
  return chain
    .map(e => ({ e, s: scoreChainEntry(e, weights, intelMin, intelMax, sampled).score }))
    .sort((a, b) => b.s - a.s || a.e.priority - b.e.priority)
    .map(x => x.e);
}

The scoring algorithm weighs factors including reliability, speed, intelligence, headroom, and rate-limit penalties, allowing the system to route traffic to the most suitable provider in real time.

Rate Limit Handling and Automatic Demotion

A critical component of automatic failover is the ability to detect and react to provider throttling. When a model returns a 429 response, recordRateLimitHit(modelDbId) is invoked to apply a penalty that temporarily demotes the model in the priority list.

// router.ts – record a 429
export function recordRateLimitHit(modelDbId: number) {
  const existing = rateLimitPenalties.get(modelDbId);
  …
  existing.penalty = Math.min(existing.penalty + PENALTY_PER_429, MAX_PENALTY);
}

These penalties decay over time (every 2 minutes), allowing models to recover automatically. The current penalty is consulted inside scoreChainEntry() via rateLimitFactor(getPenalty(entry.model_db_id)), directly influencing the final routing score and ensuring that rate-limited providers are deprioritized until they stabilize.

Health-Aware Key Selection

Each model may possess multiple API keys stored in the api_keys table. The selectKeyForModel() function iterates through available keys in a round-robin fashion, skipping entries that fail health checks:

• Keys on cooldown (isOnCooldown)
• Keys over quota (canMakeRequest, canUseTokens)
• Keys with unhealthy status (not healthy or unknown)

If decryption of an encrypted key fails, the system immediately marks the key status as error and proceeds to the next candidate.

// router.ts – select a usable key for a model
function selectKeyForModel(entry: ChainRow, estimatedTokens: number, skipKeys?: Set<string>): RouteResult | null {
  const keys = db.prepare(
    "SELECT * FROM api_keys WHERE platform = ? AND enabled = 1 AND status IN ('healthy', 'unknown')"
  ).all(entry.platform) as KeyRow[];
  …
  for (let attempt = 0; attempt < keys.length; attempt++) {
    const key = keys[idx % keys.length];
    …
    if (isOnCooldown(entry.platform, entry.model_id, key.id)) continue;
    if (!canMakeRequest(entry.platform, entry.model_id, key.id, limits)) continue;
    …
    const decryptedKey = decrypt(key.encrypted_key, key.iv, key.auth_tag);
    …
    return { provider: resolvedProvider, … };
  }
  …
}

This granular key-level filtering ensures that failover occurs at the most appropriate level—whether the entire provider is down or only a specific key is exhausted.

The Failover Execution Loop

The public routeRequest() function implements the core automatic failover logic. It walks the ordered fallback chain, applies request-level filters (vision support, tool use, context window limits), and attempts to acquire a valid key via selectKeyForModel().

// router.ts – core routing loop with automatic failover
for (const entry of sortedChain) {
  if (requireVision && !entry.supports_vision) continue;
  if (requireTools && !entry.supports_tools) continue;
  …
  const route = selectKeyForModel(entry, estimatedTokens, skipKeys);
  if (route) return route;   // success → stop looping
}
throw new RouteError('All models exhausted …', 429);

If selectKeyForModel() returns null (no usable key), the router automatically proceeds to the next model in the chain. When a request finally succeeds, recordSuccess(modelDbId) reduces the model's penalty, allowing it to climb back up the priority list. This self-healing mechanism ensures that transient failures do not permanently disable providers.

Practical Implementation Examples

Basic request routing with automatic fallback:

import { routeRequest } from './router';

// Estimate the token count for the request (including max_tokens)
const estimatedTokens = 1200;

// No special requirements – let the router pick the best available model
const result = routeRequest(estimatedTokens);

/*
result = {
  provider: <BaseProvider>,   // concrete provider implementation (e.g. OpenAI)
  modelId: 'gpt-4o',
  apiKey: 'sk-…',
  platform: 'openai',
  // ...
}
*/

Pinning a specific model while preserving failover:

// Force the use of model DB id 42, but allow fallback if unavailable
const pinnedResult = routeRequest(1000, undefined, 42);
if (!pinnedResult) {
  // The pinned model had no usable key → router automatically tried the next one
}

REST API for Chain Management

The fallback configuration is exposed via server/src/routes/fallback.ts, providing REST endpoints to view and modify the chain without restarting the service. The GET /fallback endpoint returns the current chain state, while PUT /fallback accepts a reordered configuration.

PUT /fallback
Content-Type: application/json

[
  { "model_db_id": 23, "priority": 1, "enabled": 1 },
  { "model_db_id": 7,  "priority": 2, "enabled": 1 },
  { "model_db_id": 14, "priority": 3, "enabled": 0 }   // disabled entry
]

The server recomputes the ordering immediately, applying any active bandit strategy. Additionally, server/src/services/fusion.ts provides getOrderedFusionChain() for client-side UI panels that need to reflect the same failover logic.

Summary

FreeLLMAPI implements automatic failover for providers through a multi-layered resilience strategy:

• Persistent fallback chain: Maintains an ordered list of providers in the fallback_config table, joined with live model metadata via getActiveChain().
• Dynamic reordering: Applies routing strategies (balanced, fastest, reliable) using scoreChainEntry() to prioritize the best-performing providers.
• Rate-limit penalties: Automatically demotes throttled providers when recordRateLimitHit() detects 429 responses, with automatic decay over time.
• Health-aware key selection: Uses selectKeyForModel() to perform round-robin selection across multiple keys, skipping cooldowns, quota violations, and decryption failures.
• Self-healing execution: The routeRequest() loop falls back to the next viable provider automatically, while recordSuccess() restores penalized providers to full priority.

Frequently Asked Questions

How does FreeLLMAPI decide which provider to use first?

FreeLLMAPI determines the initial provider using the configured routing strategy. When priority is selected, it uses the static order defined in the database. For other strategies like balanced or fastest, the system calls orderChain() to calculate a composite score for each provider based on reliability, latency, intelligence, and rate-limit penalties, then routes to the highest-scoring entry.

What happens when a provider returns a 429 rate limit error?

When a provider returns a 429 status, the router invokes recordRateLimitHit(modelDbId) to increment a penalty value for that model. This penalty is factored into future routing decisions via rateLimitFactor(), temporarily pushing the provider down the priority list. The penalty decays automatically every two minutes, allowing the provider to recover without manual intervention.

Can I force a specific model while keeping failover active?

Yes. You can pass a specific modelDbId to routeRequest() to pin that model for the conversation. If that model has no usable keys—due to cooldown, quota limits, or health checks—the router automatically proceeds to the next entry in the ordered fallback chain, preserving the failover guarantee while respecting your preference when possible.

How does the system handle multiple API keys for the same provider?

FreeLLMAPI maintains multiple keys per provider in the api_keys table. The selectKeyForModel() function iterates through these keys in a round-robin fashion, skipping any that are on cooldown, over quota, unhealthy, or fail decryption. This ensures that a single exhausted or revoked key does not trigger a full provider failover, maximizing the utilization of available capacity.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}