# How the Automatic Fallback Mechanism Works in FreeLLMAPI for 429 and 5xx Errors

> Discover how FreeLLMAPI automatically retries requests with fallback providers for 429 and 5xx errors, ensuring uninterrupted service by penalizing failed endpoints.

- Repository: [Tashfeen/freellmapi](https://github.com/tashfeenahmed/freellmapi)
- Tags: internals
- Published: 2026-06-22

---

**FreeLLMAPI routes every request through a sorted fallback chain that automatically retries with the next available model when a provider returns 429 or 5xx errors, applying penalties and cooldowns to prevent immediate reuse of failed endpoints.**

FreeLLMAPI implements a robust **automatic fallback mechanism** that ensures high availability when upstream LLM providers encounter rate limits or server errors. When a provider returns HTTP 429 or 5xx status codes, the system does not abort the request. Instead, it dynamically reorders the fallback chain based on penalized model scores and transparently retries the request with the next eligible provider as implemented in the tashfeenahmed/freellmapi repository.

## Understanding the Fallback Chain Architecture

The fallback system operates on a **fallback chain**—an ordered list of enabled models rebuilt for each request. According to the source code in [`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts), the chain is constructed and sorted according to the active routing strategy (priority, balanced, smartest, etc.).

### Penalty-Based Ordering

In priority mode, the `orderChain` function (lines 72-77 in [`router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/router.ts)) sorts models by calculating `priority + getPenalty(modelDbId)`. Models that have recently triggered rate limits receive higher penalties, causing them to sink lower in the chain for subsequent requests. This ensures that healthy providers are attempted first.

### Time-Based Decay

The penalty system caps at `MAX_PENALTY = 10` and decays every `DECAY_INTERVAL_MS = 2 minutes`. The `getPenalty` function (lines 121-132 in [`router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/router.ts)) reads the current penalty and applies time-based decay before returning the value. This prevents permanently blacklisting a model while ensuring recent failures impact routing decisions.

## Step-by-Step Error Handling Flow

When a provider call fails, the system executes a precise sequence to maintain service continuity without manual intervention.

### Error Classification and Retry Logic

First, [`server/src/lib/error-classify.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/lib/error-classify.ts) determines whether a 429 or 5xx error is retryable. This classification triggers the fallback mechanism rather than immediate failure, signaling the router to initiate the penalty and fallback sequence.

### Recording Rate Limit Penalties

Upon detecting a 429 or 5xx response, the router invokes `recordRateLimitHit(modelDbId)` defined in [`router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/router.ts) (lines 88-99). This function increments the per-model penalty by `PENALTY_PER_429 = 3` and stores a timestamp for decay calculations. The penalty is immediately factored into future routing decisions via `getPenalty`.

### Cooldown and Key Selection

The `selectKeyForModel` function (lines 46-100 in [`router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/router.ts)) evaluates API key availability by checking three conditions in [`server/src/services/ratelimit.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/ratelimit.ts):
- **Key health status** (`status` = healthy or unknown)
- **Cooldown state** via `isOnCooldown` (short-term ≈1 minute for transient spikes, longer for daily quota exhaustion)
- **Quota limits** via `canMakeRequest` and `canUseTokens`

If a key is on cooldown after a rate limit hit, the function returns `null`, forcing the router to proceed to the next model in the chain.

## The Complete Fallback Execution

The `routeRequest` function (lines 108-191 in [`router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/router.ts)) orchestrates the iteration. It walks the pre-sorted chain, applying vision, tool, and context-window filters, then attempts `selectKeyForModel` for each entry. If a model returns `null` (indicating all keys are on cooldown or exhausted), the loop continues automatically to the next model. When the chain is exhausted without success, the system throws a `RouteError` with status 429, signaling that all providers are temporarily unavailable.

## Code Examples for Implementing Fallback Logic

```typescript
// Recording a rate limit hit to penalize a model
import { recordRateLimitHit } from './services/router';

// When a provider returns 429/5xx, update the penalty
recordRateLimitHit(modelDbId);
// This bumps the penalty by 3, causing the model to sink in priority ordering

```

```typescript
// Inspecting current penalties for monitoring
import { getAllPenalties } from './services/router';

// Returns all current penalty scores with decay calculations
console.table(getAllPenalties());

```

```typescript
// Handling the complete routing flow with error handling
import { routeRequest, RouteError } from './services/router';

try {
  const result = routeRequest(estimatedTokens);
  // result.provider contains the successful provider
  // result.apiKey contains the decrypted key used
} catch (err) {
  if (err instanceof RouteError && err.status === 429) {
    console.warn('All models exhausted - waiting for cooldowns');
    // Implement client-side retry logic here
  }
}

```

## Summary

- **FreeLLMAPI** uses a sorted fallback chain that rebuilds per request to route around failed providers, with the entry point at [`server/src/routes/proxy.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/routes/proxy.ts).
- **Penalties** (`PENALTY_PER_429 = 3`) are applied to models returning 429/5xx errors via `recordRateLimitHit`, capped at `MAX_PENALTY = 10` and decaying every 2 minutes.
- **Cooldown logic** in [`ratelimit.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/ratelimit.ts) prevents immediate reuse of rate-limited keys through `isOnCooldown` and quota checks like `canMakeRequest`.
- **Automatic iteration** occurs in `routeRequest`, which transparently falls back to the next model until success or chain exhaustion.
- **Exhaustion handling** throws a `RouteError` with status 429 when no models remain available, propagated through [`server/src/routes/proxy.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/routes/proxy.ts).

## Frequently Asked Questions

### How long does a model stay penalized after a 429 error?

Penalties decay every `DECAY_INTERVAL_MS` (2 minutes) and are capped at `MAX_PENALTY = 10`. The penalty affects sorting immediately via `orderChain` but gradually reduces over time, allowing the model to return to its normal priority position as time passes without additional errors.

### What happens if all models in the fallback chain are rate limited?

If `routeRequest` iterates through the entire chain without finding a usable key, it throws a `RouteError` with HTTP status 429. This signals that all providers are currently exhausted, and the client should implement exponential backoff before retrying the request.

### Does the fallback mechanism work for 5xx errors as well as 429?

Yes. The [`server/src/lib/error-classify.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/lib/error-classify.ts) module identifies both 429 (Too Many Requests) and 5xx (Server Error) status codes as retryable errors. Both trigger `recordRateLimitHit`, apply the same penalty increment of 3, and initiate the cooldown and fallback sequence handled by `selectKeyForModel`.

### Can specific API keys be excluded from the fallback chain?

Yes. The `selectKeyForModel` function automatically filters keys based on health status and cooldown state. Keys marked as unhealthy or currently on cooldown via `isOnCooldown` are excluded from selection, ensuring only viable keys are considered for each routing attempt until penalties decay.