# How FreeLLMAPI Implements Automatic Failover for LLM Requests When Providers Hit Rate Limits

> Discover how FreeLLMAPI automatically redirects LLM requests when providers hit rate limits. Seamlessly handles up to 20 provider switches per request for uninterrupted service.

- Repository: [Tashfeen/freellmapi](https://github.com/tashfeenahmed/freellmapi)
- Tags: how-to-guide
- Published: 2026-06-29

---

**FreeLLMAPI routes every OpenAI-compatible request through a bounded retry loop that automatically skips any model or API key returning a rate-limit error and attempts the next candidate in the configured fallback chain, transparently handling up to 20 provider switches per request.**

FreeLLMAPI is an open-source LLM proxy that unifies multiple providers under a single OpenAI-compatible API. When a backend provider returns a 429 or quota error, the system automatically fails over to the next available model without client intervention. This automatic failover for LLM requests is implemented through a sophisticated routing and retry mechanism that tracks temporary failures, applies cooldown periods, and maintains an ordered list of fallback candidates.

## The Routing and Retry Architecture

Every incoming chat completion request triggers a multi-stage pipeline that prepares for potential failures before the first provider call is made.

### Estimating Tokens and Building the Fallback Chain

Before executing the request, FreeLLMAPI estimates the total token count and calls `resolveRoutingChain` in [`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts) (lines 364-374) to generate an ordered list of **model + key** candidates. This routing chain represents the priority order for failover attempts.

If the request specifies `"model": "auto"`, the system resolves a model-group to obtain the optimal fallback sequence based on current availability, pricing, and configured priorities.

### The Bounded Retry Loop

The core failover logic resides in `proxyRouter.post('/chat/completions')` within [`server/src/routes/proxy.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/routes/proxy.ts) (lines 87-108). The implementation uses a hard limit of **20 retry attempts** (`MAX_RETRIES = 20`) to prevent infinite loops while providing ample opportunity to find a healthy provider.

```typescript
const MAX_RETRIES = 20;
for (let attempt = 0; attempt < MAX_RETRIES; attempt++) {
  const route = routeRequest(estimatedTotal, skipKeys, preferredModel,
                             false, false, skipModels, chain);
  try {
    const result = await route.provider.chatCompletion(
      route.apiKey, messages, route.modelId,
      { temperature, max_tokens, top_p, stop },
      quotaContextForRoute(route, 'chat/completions')
    );
    // Success → return standardized response to client
    return;
  } catch (err) {
    if (isRetryableError(err)) {
      // Rate-limit detected: mark key/model for skipping
      skipKeys.add(`${route.platform}:${route.modelId}:${route.keyId}`);
      if (isModelNotFoundError(err) || isModelAccessForbiddenError(err))
        skipModels.add(route.modelDbId);
      setCooldown(...);          // Apply per-provider back-off
      recordRateLimitHit(route.modelDbId);
      learnLimitFromError(route.modelDbId, err);
      lastError = err;
      continue;                 // Attempt next candidate in chain
    }
    // Non-retryable errors propagate immediately
    res.status(502).json({ error: { message: `Provider error (${route.displayName})` } });
    return;
  }
}

```

## Detecting and Classifying Rate Limit Errors

Accurate failover depends on distinguishing transient rate limits from permanent errors.

### Error Classification Logic

The `isRetryableError` function in [`server/src/lib/error-classify.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/lib/error-classify.ts) (lines 17-22) identifies retryable conditions using HTTP status codes and message content analysis. A 429 status code or any error message containing "rate limit", "too many", or "quota" triggers the automatic failover mechanism.

```typescript
// From server/src/lib/error-classify.ts
export function isRetryableError(err: any): boolean {
  // Checks for 429 status, "rate limit", "too many", "quota" substrings
  // Returns true if the error is transient and should trigger failover
}

```

### Skip Lists and Back-off Strategies

When a rate limit is detected, FreeLLMAPI immediately adds the failing key to a `skipKeys` Set using the format `${route.platform}:${route.modelId}:${route.keyId}`, ensuring this specific credential is not reused during the current request cycle. If the error indicates model-level unavailability (via `isModelNotFoundError` or `isModelAccessForbiddenError`), the entire model is added to `skipModels`.

The system then invokes `setCooldown` from [`server/src/services/ratelimit.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/ratelimit.ts) (lines 7-14) to enforce per-provider RPD (Requests Per Day) and TPD (Tokens Per Day) limits, applying exponential back-off to prevent immediate reconnection to the saturated endpoint.

## Analytics and Adaptive Learning

FreeLLMAPI persists rate-limit events via `recordRateLimitHit` in [`server/src/services/ratelimit.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/ratelimit.ts) (lines 225-226) for operational analytics. Additionally, the `learnLimitFromError` function extracts rate-limit metadata from provider error responses to dynamically adjust internal throttling thresholds, improving routing decisions for subsequent requests.

## Exhaustion Handling

If all candidates in the routing chain are exhausted or `MAX_RETRIES` is reached, the proxy returns a unified 429 error to the client. This final response, generated in [`server/src/routes/proxy.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/routes/proxy.ts) (lines 1098-1115), aggregates the last encountered provider error, giving the client clear visibility that all configured backends are temporarily unavailable.

## Summary

- **Automatic failover** in FreeLLMAPI operates through a bounded retry loop with up to 20 attempts per request.
- **Route selection** uses `resolveRoutingChain` to pre-calculate an ordered list of model/key candidates before execution begins.
- **Error classification** in [`error-classify.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/error-classify.ts) detects 429 status codes and rate-limit keywords to trigger failover.
- **Skip tracking** prevents reuse of rate-limited keys and unavailable models within the same request lifecycle.
- **Cooldown enforcement** via `setCooldown` respects per-provider quota limits and applies back-off periods.
- **Final exhaustion** returns a clean 429 error when no providers remain available in the chain.

## Frequently Asked Questions

### How does FreeLLMAPI know which provider to try next during a failover?

The system calls `routeRequest` in the retry loop, passing `skipKeys` and `skipModels` Sets that exclude recently failed credentials. This function consults the pre-built routing chain from `resolveRoutingChain` and selects the next highest-priority candidate that is not in the exclusion lists, ensuring each retry attempts a different provider or model.

### What happens if all 20 retry attempts fail?

Once `MAX_RETRIES` is reached or the routing chain is exhausted, FreeLLMAPI returns a 429 status code with a descriptive error message indicating all providers are rate-limited. This final response is constructed in [`server/src/routes/proxy.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/routes/proxy.ts) (lines 1098-1115) and includes the last error encountered to aid debugging.

### Does FreeLLMAPI support automatic model selection for failover?

Yes. When the request specifies `"model": "auto"`, the system resolves a model-group through `resolveRoutingChain` to determine the optimal fallback sequence automatically. This allows the proxy to route requests to the best available provider without hardcoding specific model names in client applications.

### How does the system prevent immediate reuse of rate-limited API keys?

Failed keys are added to a `skipKeys` Set using a composite identifier `${route.platform}:${route.modelId}:${route.keyId}`. This Set is passed to subsequent `routeRequest` calls within the same retry loop, ensuring the specific credential is bypassed. Additionally, `setCooldown` enforces time-based back-off at the provider level to prevent rapid reconnection attempts.