# How the FreeLLMAPI Router Handles 429 Rate Limits and 5xx Provider Errors

> Learn how the FreeLLMAPI router efficiently manages 429 rate limits and 5xx provider errors by penalizing exhausted models and dynamically reordering its priority queue.

- Repository: [Tashfeen/freellmapi](https://github.com/tashfeenahmed/freellmapi)
- Tags: how-to-guide
- Published: 2026-08-26

---

**The FreeLLMAPI router distinguishes between 429 rate-limit errors and 5xx failures by applying a heavy penalty weight of 3 to rate-limited models versus a weight of 1 to transient errors, dynamically reordering the priority queue to ensure requests never hit exhausted providers.**

The FreeLLMAPI routing layer implements intelligent error classification to maintain high availability across multiple LLM providers. Located primarily in [`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts), the system treats upstream errors not as terminal failures but as signals that adjust real-time model rankings, enabling automatic failover without client-side intervention.

## Error Classification: Hard Limits vs. Transient Failures

The router categorizes provider responses into two distinct severity tiers that trigger different penalty mechanisms.

### 429 and 402 Rate-Limit Responses

When a provider returns **429 (Too Many Requests)** or **402 (Payment Required)**, the router invokes `recordRateLimitHit(modelDbId)` with `PENALTY_PER_429` set to **3**. This represents the strongest short-term health signal in the system. The model is immediately demoted in the priority ladder, causing subsequent requests to prefer alternative providers. The router also marks the specific model key as **on cooldown** via `isOnCooldown()` checks, ensuring it is skipped for the remainder of the current request cycle.

### 5xx, Timeout, and Empty Stream Failures

For **transient failures** including 5xx server errors, network timeouts, or empty response streams, the router calls `recordModelFailure(modelDbId)` with the default `PENALTY_PER_FAIL` weight of **1**. These errors receive lighter penalties because they typically indicate temporary instability rather than quota exhaustion. Models penalized for transient failures remain higher in the routing queue compared to rate-limited alternatives.

## The Penalty Scoring and Decay System

The router maintains a **penalty map** (`rateLimitPenalties`) that tracks each model's current reputation score. This system provides self-healing capabilities through automatic decay.

### Dynamic Score Adjustment

- **Success**: `recordSuccess(modelDbId)` reduces the penalty by 1, clearing the entry when reaching 0
- **Transient failure**: Increases penalty by 1 via `recordModelFailure()`
- **Rate limit**: Increases penalty by 3 via `recordRateLimitHit()`

### Time-Based Decay

Penalties automatically decrease over time to prevent permanent blacklisting. The system uses `DECAY_INTERVAL_MS` (2 minutes) and `DECAY_AMOUNT` (1) to periodically reduce scores. A model that hit a 429 will therefore climb back up the priority ladder after approximately 6 minutes of stability, assuming no additional errors occur.

## Request Routing and Fallback Logic

When routing a request, the FreeLLMAPI router executes a structured candidate selection process that incorporates real-time penalty data via `getPenalty()`.

### Candidate Selection and Sorting

1. The router builds a candidate list of available models and their API keys
2. It calculates composite scores combining reliability, speed, headroom, and current penalty weights
3. It sorts candidates so that models with lower penalties rank higher

### Cooldown and Retry Behavior

During request execution:

- If a provider returns **429**, the router records the hit with the heavier penalty, marks the key as cooled down, and immediately proceeds to the next candidate
- If a model exhausts all its keys or all keys enter cooldown, the router advances to the next model in the priority list
- This process continues until either a successful response returns or the candidate pool empties

### Exhaustion Handling

When **no model can serve the request**, the router throws a `RouteError` with **status 429** containing a diagnostics array generated by `summarizeExhaustion()`. The proxy route in [`server/src/routes/proxy.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/routes/proxy.ts) transforms this internal error into a public JSON payload:

```json
{
  "error": {
    "type": "rate_limit_error",
    "message": "All models exhausted. Add more API keys..."
  }
}

```

## Implementation Examples

### Handling Route Errors in Client Code

```typescript
import { routeRequest, RouteError } from '@/services/router';

try {
  const result = await routeRequest({
    model: 'gpt-4',
    prompt: userPrompt,
    maxTokens: 512,
  });
  // Access result.provider, result.apiKey, result.response
} catch (err) {
  if (err instanceof RouteError && err.status === 429) {
    // All providers exhausted
    console.warn(err.message);
  } else {
    console.error('Unexpected routing failure', err);
  }
}

```

### Monitoring Model Penalties

```typescript
import { getAllPenalties } from '@/services/router';

const penalties = getAllPenalties();
penalties.forEach(p => {
  console.log(`Model ${p.modelDbId}: penalty ${p.penalty} (hits ${p.count})`);
});

```

### Simulating Rate Limits for Testing

```typescript
import { recordRateLimitHit } from '@/services/router';

// Manually apply 429 penalty to model ID 42
recordRateLimitHit(42);

```

## Key Source Files

The error handling logic spans several coordinated modules:

- **[`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts)** – Core routing logic, penalty tracking, `recordRateLimitHit()`, `recordModelFailure()`, and `summarizeExhaustion()`
- **[`server/src/services/ratelimit.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/ratelimit.ts)** – Cooldown checking and lease handling utilities
- **[`server/src/services/provider-quota.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/provider-quota.ts)** – Response inspection and quota exhaustion detection
- **[`server/src/routes/proxy.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/routes/proxy.ts)** – Error formatting and public API response mapping
- **[`server/src/lib/fallback-loop.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/lib/fallback-loop.ts)** – Retry loop implementation that coordinates with the router's penalty system

## Summary

- **Immediate demotion**: 429 errors trigger `recordRateLimitHit()` with a 3-point penalty, instantly lowering a model's priority compared to 5xx failures which receive only 1 point
- **Automatic recovery**: The penalty decay system (`DECAY_INTERVAL_MS` of 2 minutes) ensures models eventually return to full availability without manual intervention
- **Graceful degradation**: The router iterates through candidates using `isOnCooldown()` checks to skip exhausted providers before throwing a `RouteError` with status 429
- **Client transparency**: When all models fail, `summarizeExhaustion()` provides diagnostic details through the proxy layer's standardized error format

## Frequently Asked Questions

### How does FreeLLMAPI distinguish between a rate limit and a server error?

The router inspects HTTP status codes and response headers through [`provider-quota.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/provider-quota.ts). **429 and 402** codes trigger `recordRateLimitHit()` with a heavy penalty weight of 3, while **5xx codes, timeouts, or empty streams** invoke `recordModelFailure()` with a lighter weight of 1. This distinction ensures quota-exhausted providers are bypassed more aggressively than temporarily unstable ones.

### What happens when all providers return 429 errors?

When every model in the candidate pool is exhausted, the router throws a `RouteError` with status 429. The `summarizeExhaustion()` function generates a diagnostics array explaining why each model was skipped. The proxy route in [`server/src/routes/proxy.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/routes/proxy.ts) catches this and returns a JSON error with type `"rate_limit_error"` advising clients to add more API keys.

### How long does a model stay penalized after a 429 error?

Penalties decay automatically using `DECAY_INTERVAL_MS` (2 minutes) and `DECAY_AMOUNT` (1). Since a 429 adds 3 penalty points, a model typically recovers full priority after approximately 6 minutes of stability. Successful requests during this period via `recordSuccess()` can accelerate recovery by manually reducing the penalty count.

### Can I programmatically check which models are currently penalized?

Yes. Import `getAllPenalties()` from [`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts) to retrieve the current state of the `rateLimitPenalties` map. This returns an array of objects containing `modelDbId`, current `penalty` score, and hit count, useful for building monitoring dashboards or debugging routing decisions.