How FreeLLMAPI Implements Automatic Failover for LLM Requests When Providers Hit Rate Limits

FreeLLMAPI routes every OpenAI-compatible request through a bounded retry loop that automatically skips any model or API key returning a rate-limit error and attempts the next candidate in the configured fallback chain, transparently handling up to 20 provider switches per request.

FreeLLMAPI is an open-source LLM proxy that unifies multiple providers under a single OpenAI-compatible API. When a backend provider returns a 429 or quota error, the system automatically fails over to the next available model without client intervention. This automatic failover for LLM requests is implemented through a sophisticated routing and retry mechanism that tracks temporary failures, applies cooldown periods, and maintains an ordered list of fallback candidates.

The Routing and Retry Architecture

Every incoming chat completion request triggers a multi-stage pipeline that prepares for potential failures before the first provider call is made.

Estimating Tokens and Building the Fallback Chain

Before executing the request, FreeLLMAPI estimates the total token count and calls resolveRoutingChain in server/src/services/router.ts (lines 364-374) to generate an ordered list of model + key candidates. This routing chain represents the priority order for failover attempts.

If the request specifies "model": "auto", the system resolves a model-group to obtain the optimal fallback sequence based on current availability, pricing, and configured priorities.

The Bounded Retry Loop

The core failover logic resides in proxyRouter.post('/chat/completions') within server/src/routes/proxy.ts (lines 87-108). The implementation uses a hard limit of 20 retry attempts (MAX_RETRIES = 20) to prevent infinite loops while providing ample opportunity to find a healthy provider.

const MAX_RETRIES = 20;
for (let attempt = 0; attempt < MAX_RETRIES; attempt++) {
  const route = routeRequest(estimatedTotal, skipKeys, preferredModel,
                             false, false, skipModels, chain);
  try {
    const result = await route.provider.chatCompletion(
      route.apiKey, messages, route.modelId,
      { temperature, max_tokens, top_p, stop },
      quotaContextForRoute(route, 'chat/completions')
    );
    // Success → return standardized response to client
    return;
  } catch (err) {
    if (isRetryableError(err)) {
      // Rate-limit detected: mark key/model for skipping
      skipKeys.add(`${route.platform}:${route.modelId}:${route.keyId}`);
      if (isModelNotFoundError(err) || isModelAccessForbiddenError(err))
        skipModels.add(route.modelDbId);
      setCooldown(...);          // Apply per-provider back-off
      recordRateLimitHit(route.modelDbId);
      learnLimitFromError(route.modelDbId, err);
      lastError = err;
      continue;                 // Attempt next candidate in chain
    }
    // Non-retryable errors propagate immediately
    res.status(502).json({ error: { message: `Provider error (${route.displayName})` } });
    return;
  }
}

Detecting and Classifying Rate Limit Errors

Accurate failover depends on distinguishing transient rate limits from permanent errors.

Error Classification Logic

The isRetryableError function in server/src/lib/error-classify.ts (lines 17-22) identifies retryable conditions using HTTP status codes and message content analysis. A 429 status code or any error message containing "rate limit", "too many", or "quota" triggers the automatic failover mechanism.

// From server/src/lib/error-classify.ts
export function isRetryableError(err: any): boolean {
  // Checks for 429 status, "rate limit", "too many", "quota" substrings
  // Returns true if the error is transient and should trigger failover
}

Skip Lists and Back-off Strategies

When a rate limit is detected, FreeLLMAPI immediately adds the failing key to a skipKeys Set using the format ${route.platform}:${route.modelId}:${route.keyId}, ensuring this specific credential is not reused during the current request cycle. If the error indicates model-level unavailability (via isModelNotFoundError or isModelAccessForbiddenError), the entire model is added to skipModels.

The system then invokes setCooldown from server/src/services/ratelimit.ts (lines 7-14) to enforce per-provider RPD (Requests Per Day) and TPD (Tokens Per Day) limits, applying exponential back-off to prevent immediate reconnection to the saturated endpoint.

Analytics and Adaptive Learning

FreeLLMAPI persists rate-limit events via recordRateLimitHit in server/src/services/ratelimit.ts (lines 225-226) for operational analytics. Additionally, the learnLimitFromError function extracts rate-limit metadata from provider error responses to dynamically adjust internal throttling thresholds, improving routing decisions for subsequent requests.

Exhaustion Handling

If all candidates in the routing chain are exhausted or MAX_RETRIES is reached, the proxy returns a unified 429 error to the client. This final response, generated in server/src/routes/proxy.ts (lines 1098-1115), aggregates the last encountered provider error, giving the client clear visibility that all configured backends are temporarily unavailable.

Summary

• Automatic failover in FreeLLMAPI operates through a bounded retry loop with up to 20 attempts per request.
• Route selection uses resolveRoutingChain to pre-calculate an ordered list of model/key candidates before execution begins.
• Error classification in error-classify.ts detects 429 status codes and rate-limit keywords to trigger failover.
• Skip tracking prevents reuse of rate-limited keys and unavailable models within the same request lifecycle.
• Cooldown enforcement via setCooldown respects per-provider quota limits and applies back-off periods.
• Final exhaustion returns a clean 429 error when no providers remain available in the chain.

Frequently Asked Questions

How does FreeLLMAPI know which provider to try next during a failover?

The system calls routeRequest in the retry loop, passing skipKeys and skipModels Sets that exclude recently failed credentials. This function consults the pre-built routing chain from resolveRoutingChain and selects the next highest-priority candidate that is not in the exclusion lists, ensuring each retry attempts a different provider or model.

What happens if all 20 retry attempts fail?

Once MAX_RETRIES is reached or the routing chain is exhausted, FreeLLMAPI returns a 429 status code with a descriptive error message indicating all providers are rate-limited. This final response is constructed in server/src/routes/proxy.ts (lines 1098-1115) and includes the last error encountered to aid debugging.

Does FreeLLMAPI support automatic model selection for failover?

Yes. When the request specifies "model": "auto", the system resolves a model-group through resolveRoutingChain to determine the optimal fallback sequence automatically. This allows the proxy to route requests to the best available provider without hardcoding specific model names in client applications.

How does the system prevent immediate reuse of rate-limited API keys?

Failed keys are added to a skipKeys Set using a composite identifier ${route.platform}:${route.modelId}:${route.keyId}. This Set is passed to subsequent routeRequest calls within the same retry loop, ensuring the specific credential is bypassed. Additionally, setCooldown enforces time-based back-off at the provider level to prevent rapid reconnection attempts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}