How the FreeLLMAPI Router Handles 429 Rate Limits and 5xx Provider Errors

The FreeLLMAPI router distinguishes between 429 rate-limit errors and 5xx failures by applying a heavy penalty weight of 3 to rate-limited models versus a weight of 1 to transient errors, dynamically reordering the priority queue to ensure requests never hit exhausted providers.

The FreeLLMAPI routing layer implements intelligent error classification to maintain high availability across multiple LLM providers. Located primarily in server/src/services/router.ts, the system treats upstream errors not as terminal failures but as signals that adjust real-time model rankings, enabling automatic failover without client-side intervention.

Error Classification: Hard Limits vs. Transient Failures

The router categorizes provider responses into two distinct severity tiers that trigger different penalty mechanisms.

429 and 402 Rate-Limit Responses

When a provider returns 429 (Too Many Requests) or 402 (Payment Required), the router invokes recordRateLimitHit(modelDbId) with PENALTY_PER_429 set to 3. This represents the strongest short-term health signal in the system. The model is immediately demoted in the priority ladder, causing subsequent requests to prefer alternative providers. The router also marks the specific model key as on cooldown via isOnCooldown() checks, ensuring it is skipped for the remainder of the current request cycle.

5xx, Timeout, and Empty Stream Failures

For transient failures including 5xx server errors, network timeouts, or empty response streams, the router calls recordModelFailure(modelDbId) with the default PENALTY_PER_FAIL weight of 1. These errors receive lighter penalties because they typically indicate temporary instability rather than quota exhaustion. Models penalized for transient failures remain higher in the routing queue compared to rate-limited alternatives.

The Penalty Scoring and Decay System

The router maintains a penalty map (rateLimitPenalties) that tracks each model's current reputation score. This system provides self-healing capabilities through automatic decay.

Dynamic Score Adjustment

• Success: recordSuccess(modelDbId) reduces the penalty by 1, clearing the entry when reaching 0
• Transient failure: Increases penalty by 1 via recordModelFailure()
• Rate limit: Increases penalty by 3 via recordRateLimitHit()

Time-Based Decay

Penalties automatically decrease over time to prevent permanent blacklisting. The system uses DECAY_INTERVAL_MS (2 minutes) and DECAY_AMOUNT (1) to periodically reduce scores. A model that hit a 429 will therefore climb back up the priority ladder after approximately 6 minutes of stability, assuming no additional errors occur.

Request Routing and Fallback Logic

When routing a request, the FreeLLMAPI router executes a structured candidate selection process that incorporates real-time penalty data via getPenalty().

Candidate Selection and Sorting

1. The router builds a candidate list of available models and their API keys
2. It calculates composite scores combining reliability, speed, headroom, and current penalty weights
3. It sorts candidates so that models with lower penalties rank higher

Cooldown and Retry Behavior

During request execution:

• If a provider returns 429, the router records the hit with the heavier penalty, marks the key as cooled down, and immediately proceeds to the next candidate
• If a model exhausts all its keys or all keys enter cooldown, the router advances to the next model in the priority list
• This process continues until either a successful response returns or the candidate pool empties

Exhaustion Handling

When no model can serve the request, the router throws a RouteError with status 429 containing a diagnostics array generated by summarizeExhaustion(). The proxy route in server/src/routes/proxy.ts transforms this internal error into a public JSON payload:

{
  "error": {
    "type": "rate_limit_error",
    "message": "All models exhausted. Add more API keys..."
  }
}

Implementation Examples

Handling Route Errors in Client Code

import { routeRequest, RouteError } from '@/services/router';

try {
  const result = await routeRequest({
    model: 'gpt-4',
    prompt: userPrompt,
    maxTokens: 512,
  });
  // Access result.provider, result.apiKey, result.response
} catch (err) {
  if (err instanceof RouteError && err.status === 429) {
    // All providers exhausted
    console.warn(err.message);
  } else {
    console.error('Unexpected routing failure', err);
  }
}

Monitoring Model Penalties

import { getAllPenalties } from '@/services/router';

const penalties = getAllPenalties();
penalties.forEach(p => {
  console.log(`Model ${p.modelDbId}: penalty ${p.penalty} (hits ${p.count})`);
});

Simulating Rate Limits for Testing

import { recordRateLimitHit } from '@/services/router';

// Manually apply 429 penalty to model ID 42
recordRateLimitHit(42);

Key Source Files

The error handling logic spans several coordinated modules:

• server/src/services/router.ts – Core routing logic, penalty tracking, recordRateLimitHit(), recordModelFailure(), and summarizeExhaustion()
• server/src/services/ratelimit.ts – Cooldown checking and lease handling utilities
• server/src/services/provider-quota.ts – Response inspection and quota exhaustion detection
• server/src/routes/proxy.ts – Error formatting and public API response mapping
• server/src/lib/fallback-loop.ts – Retry loop implementation that coordinates with the router's penalty system

Summary

• Immediate demotion: 429 errors trigger recordRateLimitHit() with a 3-point penalty, instantly lowering a model's priority compared to 5xx failures which receive only 1 point
• Automatic recovery: The penalty decay system (DECAY_INTERVAL_MS of 2 minutes) ensures models eventually return to full availability without manual intervention
• Graceful degradation: The router iterates through candidates using isOnCooldown() checks to skip exhausted providers before throwing a RouteError with status 429
• Client transparency: When all models fail, summarizeExhaustion() provides diagnostic details through the proxy layer's standardized error format

Frequently Asked Questions

How does FreeLLMAPI distinguish between a rate limit and a server error?

The router inspects HTTP status codes and response headers through provider-quota.ts. 429 and 402 codes trigger recordRateLimitHit() with a heavy penalty weight of 3, while 5xx codes, timeouts, or empty streams invoke recordModelFailure() with a lighter weight of 1. This distinction ensures quota-exhausted providers are bypassed more aggressively than temporarily unstable ones.

What happens when all providers return 429 errors?

When every model in the candidate pool is exhausted, the router throws a RouteError with status 429. The summarizeExhaustion() function generates a diagnostics array explaining why each model was skipped. The proxy route in server/src/routes/proxy.ts catches this and returns a JSON error with type "rate_limit_error" advising clients to add more API keys.

How long does a model stay penalized after a 429 error?

Penalties decay automatically using DECAY_INTERVAL_MS (2 minutes) and DECAY_AMOUNT (1). Since a 429 adds 3 penalty points, a model typically recovers full priority after approximately 6 minutes of stability. Successful requests during this period via recordSuccess() can accelerate recovery by manually reducing the penalty count.

Can I programmatically check which models are currently penalized?

Yes. Import getAllPenalties() from server/src/services/router.ts to retrieve the current state of the rateLimitPenalties map. This returns an array of objects containing modelDbId, current penalty score, and hit count, useful for building monitoring dashboards or debugging routing decisions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}