How the Reliable Strategy Works in FreeLLMAPI: Automatic Provider Fallback Explained

FreeLLMAPI’s Reliable Strategy guarantees high-availability LLM calls by automatically failing over to alternative providers when the primary model is unavailable, utilizing health monitoring, real-time scoring, and configurable retry logic.

The tashfeenahmed/freellmapi repository implements a self-healing LLM proxy that maximizes uptime through its Reliable Strategy—a server-side mechanism that transparently retries failed requests across multiple providers. Built into the TypeScript services layer, this strategy eliminates single points of failure without requiring client-side changes.

Health Monitoring and Provider State

Tracking Provider Health (model-state.ts)

Every LLM provider (OpenAI, Anthropic, Gemini, Cohere) is wrapped in a health monitor defined in server/src/services/model-state.ts. This service periodically polls each provider’s /v1/health endpoint and stores the binary health state—healthy or unhealthy—in memory. By maintaining this real-time inventory, FreeLLMAPI ensures that only responsive providers are considered for routing.

Scoring and Routing Logic

Dynamic Provider Scoring (scoring.ts)

When a request arrives, server/src/services/scoring.ts assigns a priority score to every candidate model. The algorithm weighs current quota utilization, recent latency metrics, and health status to rank providers. Higher scores indicate lower-risk candidates for primary routing.

Fallback Chain Construction (router.ts)

The server/src/services/router.ts assembles an ordered fallback chain for each request. It begins with the user-specified model, then appends alternatives sorted by their real-time scores from the scoring service. This ordered list represents the retry sequence used if the primary provider fails.

Fallback Execution and Recovery

Request Retention and Retry Logic (request-retention.ts)

The server/src/services/request-retention.ts executes the LLM call against the first provider in the chain. If the request encounters a network error, HTTP 429 (rate limit), or any non-2xx response, the service captures the failure and immediately retries with the next provider in the sequence. This process repeats until a successful response is received or the chain is exhausted.

Penalty Inspection and Recovery

Failed providers are temporarily demoted by server/src/services/penalty-inspector.ts, which reduces their routing scores to prevent thundering herds. This penalty decays over time as the provider recovers, ensuring proportional traffic distribution.

Lifecycle Management

Model Retirement and Weight Overrides

Providers that consistently error are marked for retirement in server/src/services/model-retirement.ts, temporarily excluding them from all fallback chains. Meanwhile, server/src/services/model-weight-overrides.ts allows administrators to manually adjust provider preferences or force specific routing orders without restarting the service, offering fine-grained control over the Reliable Strategy’s behavior.

Implementing Reliable Strategy in Client Applications

While the Reliable Strategy operates transparently on the server, the FreeLLMAPI client SDK—exposed via client/src/index.ts—provides options for debugging and manual chain configuration.

Automatic Fallback (Default Behavior)

By default, all requests automatically utilize the Reliable Strategy without additional configuration:

import { createClient } from '@freellmapi/client';

const client = createClient({ apiKey: process.env.FREELLMAPI_KEY });

const response = await client.chat.completions.create({
  model: 'gpt-4o',
  messages: [{ role: 'user', content: 'Explain quantum tunnelling.' }],
});

console.log(response.choices[0].message.content);

Customizing the Fallback Chain

Developers can override the automatic ordering for testing or compliance requirements:

import { createClient } from '@freellmapi/client';

const client = createClient({
  apiKey: process.env.FREELLMAPI_KEY,
  fallbackChain: ['anthropic/claude-3-opus', 'gemini/gemini-pro']
});

const resp = await client.chat.completions.create({
  model: 'gpt-4o-mini',
  messages: [{ role: 'user', content: 'Summarize the latest news.' }],
});

Inspecting the Active Provider

To debug which provider ultimately served the request, inspect the provider field in the response:

const result = await client.chat.completions.create({
  model: 'gpt-4o',
  messages: [{ role: 'user', content: 'What is the capital of Mongolia?' }],
});

console.log('Provider used:', result.provider); // e.g., "anthropic"

Summary

• Health checks in server/src/services/model-state.ts track real-time provider availability.
• Real-time scoring in server/src/services/scoring.ts ranks candidates by quota, latency, and health.
• Intelligent routing in server/src/services/router.ts constructs ordered fallback chains.
• Automatic retries via server/src/services/request-retention.ts handle failures transparently.
• Penalty management in server/src/services/penalty-inspector.ts prevents routing to unstable providers.
• Model lifecycle controls in server/src/services/model-retirement.ts and server/src/services/model-weight-overrides.ts allow administrative overrides.

Frequently Asked Questions

What triggers a fallback in FreeLLMAPI’s Reliable Strategy?

A fallback triggers when the primary provider returns a network timeout, HTTP 429 rate-limit response, 5xx server error, or any non-2xx status code. The request-retention.ts service catches these exceptions and initiates the retry sequence with the next provider in the chain.

How does FreeLLMAPI decide which provider to use next?

The system consults server/src/services/scoring.ts to rank healthy alternatives by latency, quota availability, and recent error rates. The router.ts service then selects the highest-scoring candidate from the pre-constructed fallback chain.

Can developers customize the provider fallback order?

Yes. While the server automatically optimizes chains based on health and performance, the client SDK accepts a fallbackChain array in the configuration options, allowing developers to force a specific sequence for testing or compliance requirements.

What happens when a provider is retired?

When server/src/services/model-retirement.ts retires a provider due to persistent failures, that provider is temporarily excluded from all fallback chains. It remains unavailable for routing until health checks indicate recovery or an administrator manually reinstates it via server/src/services/model-weight-overrides.ts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}