Retry Mechanism in FreeLLMAPI: Automatic Failover Across LLM Providers

FreeLLMAPI implements an automatic per-request retry mechanism that caps attempts at 20 retries with a 45-second time budget, classifies errors to distinguish retryable from fatal failures, and applies intelligent cooldowns to failed provider keys and models.

The retry mechanism in FreeLLMAPI ensures resilient LLM inference by automatically failing over across multiple providers when requests encounter transient errors. Implemented in the tashfeenahmed/freellmapi repository, this system powers all OpenAI-, Anthropic-, and Responses-compatible endpoints (/v1/chat/completions, /v1/completions, /v1/messages) without requiring client-side changes.

Core Retry Architecture

The central retry logic resides in server/src/lib/fallback-loop.ts, which implements a stateful loop that manages request attempts across provider keys and models. The runFallbackLoop() function orchestrates the entire process, consuming a FallbackHooks interface that surfaces define to provide provider-specific behavior.

Each request enters the loop with fresh state created via newFallbackState(), carrying metadata about attempted keys, accumulated failures, and timing information. The loop continues until either a successful response is obtained, the retry budget is exhausted, or a non-retryable error forces immediate termination.

Retry Limits and Time Budgets

FreeLLMAPI enforces two complementary constraints to prevent runaway retries: a hard attempt limit and a wall-clock time budget.

Maximum Retry Attempts

The system defines FALLBACK_MAX_RETRIES = 20 in fallback-loop.ts (lines 58-60). The loop guarantees that the first attempt always executes, and critically, the first retry is also guaranteed to run regardless of remaining budget. Subsequent retries consume from this pool until exhausted.

Wall-Clock Budget

A configurable time budget defaults to 45,000 milliseconds (45 seconds) via FALLBACK_TIME_BUDGET_MS. The loop checks this budget before starting each new attempt (lines 29-34); once elapsed, the loop aborts without launching additional attempts. Operators can override this via the FALLBACK_TIME_BUDGET_MS environment variable, which the getFallbackTimeBudgetMs() function reads at runtime.

Error Classification and Retry Eligibility

Not all errors warrant retries. The server/src/lib/error-classify.ts module provides discriminant functions including isRetryableError(), isRateLimitSignal(), and isProviderLevelError() that categorize failures:

• Retryable errors: HTTP 429 (rate limit), 5xx server errors, network timeouts, and empty completions trigger the retry logic
• Non-retryable errors: Authentication failures (except 401 key depletion), model-not-found errors, and validation errors terminate the request immediately without failover

This classification ensures FreeLLMAPI does not waste budget on permanently failing configurations while aggressively retrying transient provider issues.

Cooldown and Circuit Breaking Logic

When isRetryableError() returns true, the cooldownDecisionForError() function (lines 74-88) determines how long to bench the offending resource.

Key-Level Cooldowns

For rate-limit responses containing a Retry-After header, FreeLLMAPI respects the provider's authoritative guidance. Otherwise, it applies heuristic ladders (e.g., 90 seconds or daily-midnight resets). The setCooldown() function in server/src/services/ratelimit.ts records these periods per provider key, preventing the router from selecting exhausted keys in subsequent attempts.

Model-Level Protection

The system tracks failures across keys using a sliding window (MODEL_FAILURE_WINDOW_MS) via noteModelFailure(). When failure counts exceed thresholds, the entire model enters a cooldown state, protecting downstream providers from cascading overload when a specific model variant experiences widespread issues.

Authentication Failure Handling

HTTP 401 responses receive special treatment distinct from other retryable errors. When a key returns 401, FreeLLMAPI:

1. Skips the key for the current request
2. Benches it for 5 minutes (AUTH_FAILURE_COOLDOWN_MS) via setCooldown()
3. Triggers immediate re-validation via triggerKeyRevalidation() (lines 42-50)

This isolation prevents repeatedly attempting requests with revoked or invalid credentials while allowing automatic recovery if the key becomes valid again.

Exhaustion Handling and Client Diagnostics

When retries are depleted, the exhaustedRetryError() function (lines 76-84) constructs a unified error response conforming to OpenAI's error schema. The response includes appropriate HTTP status codes (429, 502, 503, 404, 413) and, when available, a calculated retryAtMs timestamp in the error payload.

For operational visibility, optional response headers expose internal state:

• X-Fallback-Attempts: Count of provider hops attempted
• X-Fallback-Trail: Sequence of providers tried
• X-Fallback-Detail: Granular failure reasons

Enable these by setting FALLBACK_DETAIL_HEADER=true, which isFallbackDetailHeaderEnabled() checks (lines 83-94).

Usage and Configuration Examples

Basic Request with Automatic Retries

Clients require no special handling; retries occur transparently:

curl -X POST https://api.freellmapi.com/v1/chat/completions \
  -H "Authorization: Bearer $FREELLM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
        "model": "gpt-4o",
        "messages": [{"role":"user","content":"Hello"}]
      }'

If upstream providers return 429 errors with Retry-After headers, FreeLLMAPI automatically retries until success or budget exhaustion, ultimately returning a 429 with retryAtMs if applicable.

Extending the Retry Budget

Increase the time budget for latency-tolerant workloads:

export FALLBACK_TIME_BUDGET_MS=120000

# Now allows up to 2 minutes of retry time

Enabling Debug Headers

Troubleshoot routing decisions with detailed headers:

export FALLBACK_DETAIL_HEADER=true
curl -i https://api.freellmapi.com/v1/chat/completions ...

# Response includes X-Fallback-Attempts, X-Fallback-Trail, X-Fallback-Detail

Programmatic Loop Invocation

Custom surfaces can invoke the loop directly:

import { runFallbackLoop, newFallbackState } from './fallback-loop';
import { myHooks } from './my-hooks';

await runFallbackLoop({
  ...myHooks,
  maxRetries: 10,
  timeBudgetMs: 60000,
  state: newFallbackState(),
});

The FallbackHooks interface requires implementations for key selection, request execution, and error handling, allowing the same retry logic to power diverse API surfaces.

Summary

• Hard limits: Maximum 20 retries with guaranteed first retry, constrained by a default 45-second wall-clock budget
• Smart filtering: error-classify.ts distinguishes retryable transient errors from permanent configuration failures
• Hierarchical cooldowns: setCooldown() manages per-key quarantines while noteModelFailure() protects against model-level outages
• Auth isolation: 401 errors trigger 5-minute key quarantine and immediate re-validation via triggerKeyRevalidation()
• Operational transparency: Optional X-Fallback-* headers reveal attempt counts and failure trails when FALLBACK_DETAIL_HEADER is enabled

Frequently Asked Questions

How many retries does FreeLLMAPI attempt by default?

By default, FreeLLMAPI allows up to 20 retries as defined by FALLBACK_MAX_RETRIES in server/src/lib/fallback-loop.ts. However, the first attempt and the first retry are always guaranteed to execute regardless of remaining budget, meaning the absolute minimum attempts for any request is two.

What happens when the retry budget is exhausted?

When the loop exhausts the FALLBACK_TIME_BUDGET_MS (default 45,000ms) or exceeds FALLBACK_MAX_RETRIES, exhaustedRetryError() generates a standardized error response. This includes the appropriate HTTP status code (typically 429, 502, or 503), a descriptive message, and a retryAtMs field indicating when the client should retry the request.

How does FreeLLMAPI handle rate limit errors from providers?

For HTTP 429 responses, the cooldownDecisionForError() function extracts the Retry-After header if present and benches the specific provider key via setCooldown() for that duration. If no header exists, it applies heuristic cooldown periods. The router then excludes this key from subsequent attempts in the current request and future requests until the cooldown expires.

Can I customize retry behavior without modifying code?

Yes. Set the FALLBACK_TIME_BUDGET_MS environment variable to adjust the wall-clock timeout (in milliseconds). Enable detailed debugging headers by setting FALLBACK_DETAIL_HEADER=true. These variables are read at runtime by getFallbackTimeBudgetMs() and isFallbackDetailHeaderEnabled() respectively, allowing operational tuning without redeployment.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}