How FreeLLMAPI Handles Provider API Key Rotation and Health Checks

FreeLLMAPI maintains a pool of API keys for each LLM provider and rotates them using Thompson-sampled bandit scoring for reliability, round-robin ordering for unproven keys, and automated health checks every 5 minutes that disable credentials after 3 consecutive validation failures.

Managing multiple API keys across different LLM providers requires intelligent load distribution and fault tolerance. In the tashfeenahmed/freellmapi codebase, the routing and health-check subsystems work together to ensure requests always land on viable credentials while automatically removing compromised or expired keys from rotation. This article examines the specific mechanisms—from probabilistic scoring algorithms to jittered scheduling—that enable robust provider API key rotation and health checks.

Key Selection and Rotation Strategy

The selectKeyForModel function in server/src/services/router.ts (lines ~1430–1460) serves as the entry point for choosing which key handles an incoming request. The system implements a tiered selection strategy that balances exploitation of proven keys with exploration of newer additions.

Thompson-Sampled Bandit Scoring

When historical reliability data exists for keys, FreeLLMAPI applies a Thompson-sampled bandit score via the orderKeysByScore helper (lines ~828–847 in router.ts). This algorithm assigns approximately 78% weight to reliability metrics, ranking keys by their statistical likelihood of success. The highest-scoring key is attempted first, ensuring traffic gravitates toward the most dependable credentials while maintaining probabilistic diversity to avoid overloading single keys.

Round-Robin Fallback and Quota Weighting

For keys without observed performance data, the system falls back to a round-robin cursor tracked via roundRobinIndex. This guarantees fair distribution across new or recently added credentials. Additionally, optional quota weighting through orderKeysByRemainingQuota can reorder the round-robin list based on remaining provider quota, preventing premature exhaustion of high-traffic keys.

Automated Health Check System

The server/src/services/health.ts module implements a self-healing validation layer that runs independently of the request path. This subsystem continuously monitors key validity without disrupting live traffic.

Scheduling and Jitter

Health checks execute on a jittered schedule to prevent thundering herd problems against provider endpoints. The nextHealthCheckDelayMs function (lines ~886–890) calculates intervals using CHECK_INTERVAL_MS (5 minutes) plus a random variance of CHECK_INTERVAL_JITTER (±20%). This randomization ensures that distributed instances or restarted nodes do not synchronize their probe waves.

Provider-Wise Interleaving and Concurrency Controls

To avoid hammering a single provider, the runHealthPass function (lines ~998–1060) uses interleaveByProvider to distribute checks evenly across different services. Concurrency is strictly limited to DEFAULT_HEALTH_CHECK_CONCURRENCY (8 parallel probes), with a minimum spacing of DEFAULT_MIN_SPACING_MS (1000 milliseconds) between requests to the same provider. These values are configurable via the HEALTH_CHECK_CONCURRENCY and HEALTH_CHECK_MIN_SPACING_MS environment variables (see getHealthCheckConcurrency and getMinSpacingMs, lines ~42–50).

Validation Logic and Auto-Disable Thresholds

The checkKeyHealth function (lines ~83–130) validates each key by calling the provider’s validateKey method through the key’s dedicated proxy tunnel (withKeyProxy). The system distinguishes between credential failures and network blips:

• Authentication errors (401/403) increment the consecutive‑failure counter via recordInvalidFailure (lines ~64–80)
• Transport errors (DNS timeouts, connection resets) are logged but do not count toward the failure threshold

After 3 consecutive validation failures, the key is automatically disabled (enabled = 0), immediately removing it from the rotation pool without manual intervention.

Recovery and Self-Healing Mechanisms

FreeLLMAPI implements bidirectional health state management, allowing keys to recover from transient error states without administrative action.

Automatic Re-enabling on Success

When a request successfully completes using a key previously marked with an error status, the routing layer invokes markKeyHealthyFromRequest (lines ~204–209 in health.ts). This function clears the error flag and resets the consecutive failure counter, effectively promoting the key back to healthy status. This design ensures that temporary network partitions or brief provider outages do not permanently blacklist otherwise valid credentials.

Implementation Details and Code Examples

Starting the Health Checker

Initialize the periodic validation system at server startup:

import { startHealthChecker } from '@/services/health';
import { Scheduler } from '@/lib/scheduler';

// Scheduler wraps setTimeout/clearTimeout for jittered execution
const scheduler = new Scheduler();
startHealthChecker(scheduler);

Manual Health Check Trigger

Force an immediate, non-jittered validation pass for all enabled keys:

import { checkAllKeys } from '@/services/health';

// Useful for administrative dashboards or debugging
await checkAllKeys({ force: true });

Request-Time Key Selection

The following pattern demonstrates how the router selects credentials during request handling:

import { selectKeyForModel } from '@/services/router';
import type { RouteResult } from '@/services/router';

const route: RouteResult | null = selectKeyForModel(
  modelEntry,        // Chain/model configuration
  1024,              // Estimated token count
  new Set(),         // Keys already attempted in this request
  [],                // Diagnostic log array
);

if (!route) {
  // Return 503 or trigger fallback logic
}

Marking Keys Healthy After Success

Restore a key to active rotation following a successful API call:

import { markKeyHealthyFromRequest } from '@/services/health';

// Call after confirming the request succeeded
markKeyHealthyFromRequest(keyId);

Summary

• Statistical rotation: FreeLLMAPI uses Thompson-sampled bandit scoring (orderKeysByScore) to prioritize reliable keys, falling back to round-robin for untested credentials.
• Jittered health checks: The system validates keys every 5 minutes with 20% random jitter, interleaving providers to distribute load.
• Intelligent failure handling: Only authentication errors (401/403) count toward the 3-strike auto-disable threshold; transport errors are ignored to prevent churn during network blips.
• Self-healing recovery: Keys automatically return to healthy status after successfully serving a live request (markKeyHealthyFromRequest).
• Resource protection: Concurrency is capped at 8 parallel checks with 1000ms minimum spacing per provider to respect rate limits.

Frequently Asked Questions

How does FreeLLMAPI decide which API key to use for a request?

The selectKeyForModel function in server/src/services/router.ts first checks for historical reliability data. If available, it orders keys using Thompson-sampled bandit scoring (78% reliability weight) and selects the highest scorer. For new keys without data, it uses a simple round-robin cursor, optionally weighted by remaining quota.

What triggers an API key to be automatically disabled?

A key is disabled after 3 consecutive validation failures recorded by recordInvalidFailure in server/src/services/health.ts. Only authentication errors (401/403) increment this counter; transport-level failures like DNS timeouts or connection resets do not count toward the threshold.

How often does FreeLLMAPI check the health of provider keys?

The health checker runs every 5 minutes with a ±20% random jitter applied via nextHealthCheckDelayMs to prevent synchronized probe waves. This interval is controlled by the CHECK_INTERVAL_MS and CHECK_INTERVAL_JITTER constants.

Can a disabled or error-marked key return to active rotation?

Yes. The markKeyHealthyFromRequest function automatically promotes a key back to healthy status when it successfully completes a live request. This self-healing mechanism ensures that keys disabled due to transient network issues can recover without manual administrator intervention.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}