What Happens When All Providers in the FreeLLMAPI Fallback Chain Fail?

When every provider in the FreeLLMAPI fallback chain fails, the router returns an HTTP 502 Bad Gateway with an X-Fallback-Attempts header and a JSON error body containing the code "fallback_exhausted".

FreeLLMAPI is an open-source unified API gateway that aggregates multiple LLM providers into a single endpoint. When all configured providers in the fallback chain are exhausted due to rate limits, timeouts, or server errors, the system implements a structured failure protocol defined in server/src/services/router.ts and server/src/lib/error-classify.ts.

How the Fallback Chain Exhaustion Occurs

The FreeLLMAPI router attempts each enabled provider sequentially until it either succeeds or exhausts the chain. For every provider attempt, the system performs three validation steps:

1. Health Check: Verifies the API key is healthy and under rate limits.
2. SDK Execution: Calls the provider's native SDK.
3. Error Classification: If the provider returns 429 (rate-limited), any 5xx error, or a timeout, the router cool-downs that key, records the failure, and advances to the next provider.

This process continues for up to 20 attempts (configurable) before the chain is considered exhausted.

The Exhaustion Response Protocol

When the router determines that every available provider has failed, it stops retrying and returns a standardized error response to the client.

HTTP Status and Headers

The response includes:

• Status Code: 502 Bad Gateway (or 422 for malformed requests)
• Header: X-Fallback-Attempts: N where N represents the number of providers tried before giving up

Error Body Structure

The JSON payload follows this exact schema:

{
  "error": {
    "message": "All providers in the fallback chain failed",
    "code": "fallback_exhausted"
  }
}

This error structure is generated in server/src/services/router.ts and finalized through server/src/lib/error-classify.ts, which converts provider-level failures into router-level HTTP codes.

Handling Fallback Exhaustion in Client Code

You can detect and handle this condition using any OpenAI-compatible client by checking for the 502 status code.

Python Client Example

from openai import OpenAI, APIError

client = OpenAI(
    base_url="http://localhost:3001/v1",
    api_key="freellmapi-your-unified-key"
)

try:
    resp = client.chat.completions.create(
        model="auto",
        messages=[{"role": "user", "content": "Explain quantum tunnelling"}],
    )
    print(resp.choices[0].message.content)
except APIError as exc:
    if exc.status_code == 502:
        attempts = exc.headers.get("x-fallback-attempts", "unknown")
        print(f"All providers exhausted after {attempts} attempts")
    else:
        raise

cURL Request and Response

curl -i http://localhost:3001/v1/chat/completions \
  -H "Authorization: Bearer freellmapi-your-unified-key" \
  -H "Content-Type: application/json" \
  -d '{
        "model": "auto",
        "messages": [{"role": "user", "content": "Give me a poem about clouds"}]
      }'

When the fallback chain is fully exhausted, the response appears as:

HTTP/1.1 502 Bad Gateway
X-Fallback-Attempts: 20
Content-Type: application/json

{
  "error": {
    "message": "All providers in the fallback chain failed",
    "code": "fallback_exhausted"
  }
}

Node.js Detection

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "http://localhost:3001/v1",
  apiKey: "freellmapi-your-unified-key",
});

try {
  const result = await client.chat.completions.create({
    model: "auto",
    messages: [{ role: "user", content: "What is the capital of Estonia?" }],
  });
  console.log(result.choices[0].message.content);
} catch (err) {
  if (err.response?.status === 502) {
    const attempts = err.response.headers["x-fallback-attempts"];
    console.warn(`Fallback chain exhausted after ${attempts} provider attempts`);
  } else {
    throw err;
  }
}

Implementation Details in the Source Code

The fallback exhaustion logic is distributed across three key files in the tashfeenahmed/freellmapi repository:

server/src/services/router.ts Contains the core routing loop that iterates through providers and generates the 502 response when the fallback list is exhausted.

server/src/lib/error-classify.ts Transforms provider-specific errors (timeouts, rate limits, 5xx responses) into standardized router-level HTTP codes, including the 502 status for exhausted fallback chains.

server/src/services/fusion.ts Implements "wave" style fallback for tool-heavy requests, which ultimately delegates to the same 502 exhaustion path when every provider in the wave configuration fails.

Summary

• FreeLLMAPI attempts up to 20 provider calls sequentially before declaring the fallback chain exhausted.
• HTTP 502 Bad Gateway is returned when all providers fail, accompanied by the X-Fallback-Attempts header showing the attempt count.
• Error classification occurs in server/src/lib/error-classify.ts, while the routing logic resides in server/src/services/router.ts.
• Client applications should inspect for status code 502 and the fallback_exhausted error code to handle complete provider exhaustion.
• Tool-heavy requests use the same exhaustion protocol through server/src/services/fusion.ts even when processed in "waves."

Frequently Asked Questions

What is the maximum number of fallback attempts in FreeLLMAPI?

FreeLLMAPI attempts up to 20 providers by default before exhausting the fallback chain, though this value is configurable in the router settings. The attempt count is communicated to clients via the X-Fallback-Attempts header in the final 502 response.

How does FreeLLMAPI classify provider errors that trigger a fallback?

According to server/src/lib/error-classify.ts, the router triggers a fallback and cool-down when a provider returns HTTP 429 (rate-limited), any 5xx server error, or experiences a timeout. These errors cause the router to mark the key as unhealthy and advance to the next provider in the chain.

Can I distinguish between a single provider failure and a complete fallback chain failure?

Yes. Single provider failures are handled internally by the router and trigger automatic retries. Only when the entire chain is exhausted does FreeLLMAPI return HTTP 502 Bad Gateway with the error code "fallback_exhausted". Successful requests return standard 200 OK responses, while malformed requests return 422.

What happens to tool-heavy requests when all providers fail?

As implemented in server/src/services/fusion.ts, tool-heavy requests use a "wave" style fallback that attempts multiple providers in parallel groups. If every provider in every wave fails, the system falls back to the same 502 Bad Gateway response with the fallback_exhausted error code, identical to standard sequential fallback exhaustion.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}