# Retry Mechanism in FreeLLMAPI: Automatic Failover Across LLM Providers

> Discover FreeLLMAPI's retry mechanism for automatic failover. It features 20 retries, a 45-second budget, error classification, and intelligent cooldowns for reliable LLM provider access.

- Repository: [Tashfeen/freellmapi](https://github.com/tashfeenahmed/freellmapi)
- Tags: how-to-guide
- Published: 2026-08-26

---

**FreeLLMAPI implements an automatic per-request retry mechanism that caps attempts at 20 retries with a 45-second time budget, classifies errors to distinguish retryable from fatal failures, and applies intelligent cooldowns to failed provider keys and models.**

The retry mechanism in FreeLLMAPI ensures resilient LLM inference by automatically failing over across multiple providers when requests encounter transient errors. Implemented in the `tashfeenahmed/freellmapi` repository, this system powers all OpenAI-, Anthropic-, and Responses-compatible endpoints (`/v1/chat/completions`, `/v1/completions`, `/v1/messages`) without requiring client-side changes.

## Core Retry Architecture

The central retry logic resides in **[`server/src/lib/fallback-loop.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/lib/fallback-loop.ts)**, which implements a stateful loop that manages request attempts across provider keys and models. The `runFallbackLoop()` function orchestrates the entire process, consuming a `FallbackHooks` interface that surfaces define to provide provider-specific behavior.

Each request enters the loop with fresh state created via `newFallbackState()`, carrying metadata about attempted keys, accumulated failures, and timing information. The loop continues until either a successful response is obtained, the retry budget is exhausted, or a non-retryable error forces immediate termination.

## Retry Limits and Time Budgets

FreeLLMAPI enforces two complementary constraints to prevent runaway retries: a hard attempt limit and a wall-clock time budget.

### Maximum Retry Attempts

The system defines **`FALLBACK_MAX_RETRIES = 20`** in [`fallback-loop.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/fallback-loop.ts) (lines 58-60). The loop guarantees that the first attempt always executes, and critically, the **first retry is also guaranteed to run** regardless of remaining budget. Subsequent retries consume from this pool until exhausted.

### Wall-Clock Budget

A configurable time budget defaults to **45,000 milliseconds** (45 seconds) via `FALLBACK_TIME_BUDGET_MS`. The loop checks this budget *before* starting each new attempt (lines 29-34); once elapsed, the loop aborts without launching additional attempts. Operators can override this via the `FALLBACK_TIME_BUDGET_MS` environment variable, which the `getFallbackTimeBudgetMs()` function reads at runtime.

## Error Classification and Retry Eligibility

Not all errors warrant retries. The **[`server/src/lib/error-classify.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/lib/error-classify.ts)** module provides discriminant functions including `isRetryableError()`, `isRateLimitSignal()`, and `isProviderLevelError()` that categorize failures:

- **Retryable errors**: HTTP 429 (rate limit), 5xx server errors, network timeouts, and empty completions trigger the retry logic
- **Non-retryable errors**: Authentication failures (except 401 key depletion), model-not-found errors, and validation errors terminate the request immediately without failover

This classification ensures FreeLLMAPI does not waste budget on permanently failing configurations while aggressively retrying transient provider issues.

## Cooldown and Circuit Breaking Logic

When `isRetryableError()` returns true, the **`cooldownDecisionForError()`** function (lines 74-88) determines how long to bench the offending resource.

### Key-Level Cooldowns

For rate-limit responses containing a `Retry-After` header, FreeLLMAPI respects the provider's authoritative guidance. Otherwise, it applies heuristic ladders (e.g., 90 seconds or daily-midnight resets). The `setCooldown()` function in [`server/src/services/ratelimit.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/ratelimit.ts) records these periods per provider key, preventing the router from selecting exhausted keys in subsequent attempts.

### Model-Level Protection

The system tracks failures across keys using a sliding window (`MODEL_FAILURE_WINDOW_MS`) via `noteModelFailure()`. When failure counts exceed thresholds, the entire model enters a cooldown state, protecting downstream providers from cascading overload when a specific model variant experiences widespread issues.

## Authentication Failure Handling

HTTP 401 responses receive special treatment distinct from other retryable errors. When a key returns 401, FreeLLMAPI:

1. Skips the key for the current request
2. Benches it for **5 minutes** (`AUTH_FAILURE_COOLDOWN_MS`) via `setCooldown()`
3. Triggers immediate re-validation via `triggerKeyRevalidation()` (lines 42-50)

This isolation prevents repeatedly attempting requests with revoked or invalid credentials while allowing automatic recovery if the key becomes valid again.

## Exhaustion Handling and Client Diagnostics

When retries are depleted, the **`exhaustedRetryError()`** function (lines 76-84) constructs a unified error response conforming to OpenAI's error schema. The response includes appropriate HTTP status codes (429, 502, 503, 404, 413) and, when available, a calculated `retryAtMs` timestamp in the error payload.

For operational visibility, optional response headers expose internal state:
- **`X-Fallback-Attempts`**: Count of provider hops attempted
- **`X-Fallback-Trail`**: Sequence of providers tried
- **`X-Fallback-Detail`**: Granular failure reasons

Enable these by setting `FALLBACK_DETAIL_HEADER=true`, which `isFallbackDetailHeaderEnabled()` checks (lines 83-94).

## Usage and Configuration Examples

### Basic Request with Automatic Retries

Clients require no special handling; retries occur transparently:

```bash
curl -X POST https://api.freellmapi.com/v1/chat/completions \
  -H "Authorization: Bearer $FREELLM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
        "model": "gpt-4o",
        "messages": [{"role":"user","content":"Hello"}]
      }'

```

If upstream providers return 429 errors with `Retry-After` headers, FreeLLMAPI automatically retries until success or budget exhaustion, ultimately returning a 429 with `retryAtMs` if applicable.

### Extending the Retry Budget

Increase the time budget for latency-tolerant workloads:

```bash
export FALLBACK_TIME_BUDGET_MS=120000

# Now allows up to 2 minutes of retry time

```

### Enabling Debug Headers

Troubleshoot routing decisions with detailed headers:

```bash
export FALLBACK_DETAIL_HEADER=true
curl -i https://api.freellmapi.com/v1/chat/completions ...

# Response includes X-Fallback-Attempts, X-Fallback-Trail, X-Fallback-Detail

```

### Programmatic Loop Invocation

Custom surfaces can invoke the loop directly:

```typescript
import { runFallbackLoop, newFallbackState } from './fallback-loop';
import { myHooks } from './my-hooks';

await runFallbackLoop({
  ...myHooks,
  maxRetries: 10,
  timeBudgetMs: 60000,
  state: newFallbackState(),
});

```

The `FallbackHooks` interface requires implementations for key selection, request execution, and error handling, allowing the same retry logic to power diverse API surfaces.

## Summary

- **Hard limits**: Maximum 20 retries with guaranteed first retry, constrained by a default 45-second wall-clock budget
- **Smart filtering**: [`error-classify.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/error-classify.ts) distinguishes retryable transient errors from permanent configuration failures
- **Hierarchical cooldowns**: `setCooldown()` manages per-key quarantines while `noteModelFailure()` protects against model-level outages
- **Auth isolation**: 401 errors trigger 5-minute key quarantine and immediate re-validation via `triggerKeyRevalidation()`
- **Operational transparency**: Optional `X-Fallback-*` headers reveal attempt counts and failure trails when `FALLBACK_DETAIL_HEADER` is enabled

## Frequently Asked Questions

### How many retries does FreeLLMAPI attempt by default?

By default, FreeLLMAPI allows up to 20 retries as defined by `FALLBACK_MAX_RETRIES` in [`server/src/lib/fallback-loop.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/lib/fallback-loop.ts). However, the first attempt and the first retry are always guaranteed to execute regardless of remaining budget, meaning the absolute minimum attempts for any request is two.

### What happens when the retry budget is exhausted?

When the loop exhausts the `FALLBACK_TIME_BUDGET_MS` (default 45,000ms) or exceeds `FALLBACK_MAX_RETRIES`, `exhaustedRetryError()` generates a standardized error response. This includes the appropriate HTTP status code (typically 429, 502, or 503), a descriptive message, and a `retryAtMs` field indicating when the client should retry the request.

### How does FreeLLMAPI handle rate limit errors from providers?

For HTTP 429 responses, the `cooldownDecisionForError()` function extracts the `Retry-After` header if present and benches the specific provider key via `setCooldown()` for that duration. If no header exists, it applies heuristic cooldown periods. The router then excludes this key from subsequent attempts in the current request and future requests until the cooldown expires.

### Can I customize retry behavior without modifying code?

Yes. Set the `FALLBACK_TIME_BUDGET_MS` environment variable to adjust the wall-clock timeout (in milliseconds). Enable detailed debugging headers by setting `FALLBACK_DETAIL_HEADER=true`. These variables are read at runtime by `getFallbackTimeBudgetMs()` and `isFallbackDetailHeaderEnabled()` respectively, allowing operational tuning without redeployment.