How the Automatic Fallback Mechanism Works in FreeLLMAPI for 429 and 5xx Errors

FreeLLMAPI routes every request through a sorted fallback chain that automatically retries with the next available model when a provider returns 429 or 5xx errors, applying penalties and cooldowns to prevent immediate reuse of failed endpoints.

FreeLLMAPI implements a robust automatic fallback mechanism that ensures high availability when upstream LLM providers encounter rate limits or server errors. When a provider returns HTTP 429 or 5xx status codes, the system does not abort the request. Instead, it dynamically reorders the fallback chain based on penalized model scores and transparently retries the request with the next eligible provider as implemented in the tashfeenahmed/freellmapi repository.

Understanding the Fallback Chain Architecture

The fallback system operates on a fallback chain—an ordered list of enabled models rebuilt for each request. According to the source code in server/src/services/router.ts, the chain is constructed and sorted according to the active routing strategy (priority, balanced, smartest, etc.).

Penalty-Based Ordering

In priority mode, the orderChain function (lines 72-77 in router.ts) sorts models by calculating priority + getPenalty(modelDbId). Models that have recently triggered rate limits receive higher penalties, causing them to sink lower in the chain for subsequent requests. This ensures that healthy providers are attempted first.

Time-Based Decay

The penalty system caps at MAX_PENALTY = 10 and decays every DECAY_INTERVAL_MS = 2 minutes. The getPenalty function (lines 121-132 in router.ts) reads the current penalty and applies time-based decay before returning the value. This prevents permanently blacklisting a model while ensuring recent failures impact routing decisions.

Step-by-Step Error Handling Flow

When a provider call fails, the system executes a precise sequence to maintain service continuity without manual intervention.

Error Classification and Retry Logic

First, server/src/lib/error-classify.ts determines whether a 429 or 5xx error is retryable. This classification triggers the fallback mechanism rather than immediate failure, signaling the router to initiate the penalty and fallback sequence.

Recording Rate Limit Penalties

Upon detecting a 429 or 5xx response, the router invokes recordRateLimitHit(modelDbId) defined in router.ts (lines 88-99). This function increments the per-model penalty by PENALTY_PER_429 = 3 and stores a timestamp for decay calculations. The penalty is immediately factored into future routing decisions via getPenalty.

Cooldown and Key Selection

The selectKeyForModel function (lines 46-100 in router.ts) evaluates API key availability by checking three conditions in server/src/services/ratelimit.ts:

• Key health status (status = healthy or unknown)
• Cooldown state via isOnCooldown (short-term ≈1 minute for transient spikes, longer for daily quota exhaustion)
• Quota limits via canMakeRequest and canUseTokens

If a key is on cooldown after a rate limit hit, the function returns null, forcing the router to proceed to the next model in the chain.

The Complete Fallback Execution

The routeRequest function (lines 108-191 in router.ts) orchestrates the iteration. It walks the pre-sorted chain, applying vision, tool, and context-window filters, then attempts selectKeyForModel for each entry. If a model returns null (indicating all keys are on cooldown or exhausted), the loop continues automatically to the next model. When the chain is exhausted without success, the system throws a RouteError with status 429, signaling that all providers are temporarily unavailable.

Code Examples for Implementing Fallback Logic

// Recording a rate limit hit to penalize a model
import { recordRateLimitHit } from './services/router';

// When a provider returns 429/5xx, update the penalty
recordRateLimitHit(modelDbId);
// This bumps the penalty by 3, causing the model to sink in priority ordering

// Inspecting current penalties for monitoring
import { getAllPenalties } from './services/router';

// Returns all current penalty scores with decay calculations
console.table(getAllPenalties());

// Handling the complete routing flow with error handling
import { routeRequest, RouteError } from './services/router';

try {
  const result = routeRequest(estimatedTokens);
  // result.provider contains the successful provider
  // result.apiKey contains the decrypted key used
} catch (err) {
  if (err instanceof RouteError && err.status === 429) {
    console.warn('All models exhausted - waiting for cooldowns');
    // Implement client-side retry logic here
  }
}

Summary

• FreeLLMAPI uses a sorted fallback chain that rebuilds per request to route around failed providers, with the entry point at server/src/routes/proxy.ts.
• Penalties (PENALTY_PER_429 = 3) are applied to models returning 429/5xx errors via recordRateLimitHit, capped at MAX_PENALTY = 10 and decaying every 2 minutes.
• Cooldown logic in ratelimit.ts prevents immediate reuse of rate-limited keys through isOnCooldown and quota checks like canMakeRequest.
• Automatic iteration occurs in routeRequest, which transparently falls back to the next model until success or chain exhaustion.
• Exhaustion handling throws a RouteError with status 429 when no models remain available, propagated through server/src/routes/proxy.ts.

Frequently Asked Questions

How long does a model stay penalized after a 429 error?

Penalties decay every DECAY_INTERVAL_MS (2 minutes) and are capped at MAX_PENALTY = 10. The penalty affects sorting immediately via orderChain but gradually reduces over time, allowing the model to return to its normal priority position as time passes without additional errors.

What happens if all models in the fallback chain are rate limited?

If routeRequest iterates through the entire chain without finding a usable key, it throws a RouteError with HTTP status 429. This signals that all providers are currently exhausted, and the client should implement exponential backoff before retrying the request.

Does the fallback mechanism work for 5xx errors as well as 429?

Yes. The server/src/lib/error-classify.ts module identifies both 429 (Too Many Requests) and 5xx (Server Error) status codes as retryable errors. Both trigger recordRateLimitHit, apply the same penalty increment of 3, and initiate the cooldown and fallback sequence handled by selectKeyForModel.

Can specific API keys be excluded from the fallback chain?

Yes. The selectKeyForModel function automatically filters keys based on health status and cooldown state. Keys marked as unhealthy or currently on cooldown via isOnCooldown are excluded from selection, ensuring only viable keys are considered for each routing attempt until penalties decay.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}