How the Health Service in FreeLLMAPI Monitors Provider Status: Architecture and Implementation

The FreeLLMAPI Health Service continuously validates registered LLM provider keys through scheduled health passes that probe API credentials every 5 minutes with randomized jitter, automatically disabling keys after three consecutive invalid credential failures while distinguishing transient network errors from authentication issues.

FreeLLMAPI is an open-source proxy server that aggregates multiple LLM provider APIs behind a unified interface. Its server-side Health Service ensures high availability by actively monitoring the validity of stored API keys against live provider endpoints. This article examines the implementation details found in the tashfeenahmed/freellmapi repository, tracing the flow from scheduled checks to auto-disable logic and public health endpoints.

Scheduled Health Passes with Jitter Control

When the FreeLLMAPI server initializes, it invokes startHealthChecker() located in server/src/index.ts to register a recurring background job. The system uses a base probe interval of 5 minutes defined by CHECK_INTERVAL_MS, but adds randomized CHECK_INTERVAL_JITTER to prevent thundering herd problems where multiple server instances might simultaneously bombard the same provider.

This jitter mechanism ensures that distributed deployments stagger their health checks, reducing load spikes on upstream LLM APIs. The scheduler persists until server shutdown, continuously enqueueing new health passes without blocking the main event loop.

Concurrency and Rate Limiting

Each health pass respects strict resource constraints to avoid overwhelming providers or the local event loop. In server/src/services/health.ts, the checkAllKeys() function uses getHealthCheckConcurrency()—defaulting to 8 parallel probes—to limit simultaneous network requests.

Additionally, the service enforces a minimum gap between probes targeting the same provider using getMinSpacingMs(), which defaults to 1000 milliseconds. This per-provider throttling prevents aggressive polling of a single endpoint while allowing concurrent checks across different providers.

Key-Level Health Validation Logic

The core validation logic resides in checkKeyHealth(keyId) within server/src/services/health.ts. For each key, the function executes the following steps:

1. Database Retrieval: Fetches the key record from the SQLite database.
2. Provider Resolution: Calls resolveProvider to instantiate the correct provider implementation.
3. Proxy Validation: Decrypts the stored API key and validates it through withKeyProxy, which mimics real traffic patterns against the provider’s actual API endpoint.

The function interprets the validation result into three distinct statuses:

• 'healthy': The provider accepted the credentials and returned a valid response.
• 'invalid': The provider rejected the credentials (HTTP 401/403). This triggers recordInvalidFailure(), which increments a per-key failure counter. After three consecutive invalid failures, the system automatically disables the key to prevent further wasted requests.
• 'error': A transport-level failure occurred (DNS resolution, timeout, or TLS error). The service records this status but does not increment the invalid failure counter, recognizing that network instability does not indicate bad credentials.

Bulk Health Checks and Provider Buckets

The checkAllKeys(opts?) function orchestrates mass validation by gathering every enabled key and organizing them into providerBucket groups. This bucketing ensures that the minimum spacing rules apply per-provider rather than globally.

After executing parallel checks within the configured concurrency limits, the function returns a HealthPassResult object containing:

• healthyProviders: An array of providers where all associated keys passed validation.
• totalProviders: The count of distinct providers under test.
• ratio: The proportion of healthy providers to total providers.

This aggregation allows downstream services to quickly assess overall system health without scanning individual key records.

Degradation Snapshots and System Integration

In server/src/services/degradation.ts, the Degradation Service consumes the HealthPassResult to build a HealthSnapshot. This snapshot stores the same three metrics—healthy providers, total providers, and ratio—in a module-level variable lastSnapshot.

Dashboard UIs and fallback mechanisms read this snapshot to display provider availability or trigger degraded mode when ratios fall below acceptable thresholds. The snapshot pattern decouples health monitoring from client-facing status pages, ensuring that expensive health checks do not block HTTP requests.

Public Health Endpoint

The server exposes the health state via the /health route defined in server/src/routes/health.ts. This endpoint accepts optional query parameters to check a specific key ID via checkKeyHealth() or return the global checkAllKeys() results.

The route returns JSON reflecting the current KeyStatus enum values ('healthy', 'invalid', 'error', or 'unknown') defined in shared/types.ts, allowing external monitoring systems to poll for status without database access.

Usage Examples

Trigger a manual health check for a single key:

import { checkKeyHealth } from '@/server/src/services/health.js';

const status = await checkKeyHealth(42);
console.log(`Key 42 status: ${status}`);   // → 'healthy' | 'invalid' | 'error'

Run a full health pass programmatically:

import { checkAllKeys } from '@/server/src/services/health.js';

const result = await checkAllKeys();
console.log(`Healthy providers: ${result.healthyProviders.length}/${result.totalProviders}`);

Query the public HTTP health endpoint:

fetch('https://api.example.com/health')
  .then(r => r.json())
  .then(data => console.log('Health snapshot:', data));

Summary

• The Health Service initiates at server start via startHealthChecker(), probing every 5 minutes with randomized jitter to prevent synchronized load spikes.
• Concurrency is capped at 8 parallel probes with 1000ms minimum spacing between checks targeting the same provider, protecting upstream APIs from aggressive polling.
• Credential validation distinguishes authentication failures from network errors, applying a three-strike auto-disable rule only to invalid credentials (401/403) while ignoring transient transport issues.
• Bulk checks aggregate results into HealthPassResult objects that the Degradation Service converts to snapshots for dashboard visualization and fallback logic.
• The /health endpoint exposes real-time status via HTTP, returning standardized KeyStatus values without exposing raw database connections.

Frequently Asked Questions

How often does FreeLLMAPI check provider health?

The Health Service runs scheduled health passes every 5 minutes (CHECK_INTERVAL_MS), with added random jitter (CHECK_INTERVAL_JITTER) to prevent multiple instances from probing providers simultaneously. This interval balances freshness of status data against API rate limit concerns.

What happens when a provider key fails validation?

If validation returns an authentication error (HTTP 401/403), the system calls recordInvalidFailure() to increment a per-key failure counter. After three consecutive invalid failures, the key is automatically disabled. Transport errors like timeouts or DNS failures do not count toward this limit and do not disable keys.

How does the system handle network timeouts versus authentication errors?

The checkKeyHealth() function explicitly categorizes transport-level failures (DNS, timeout, TLS) as 'error' status without incrementing the invalid failure counter. Only explicit credential rejection by the provider triggers the failure counter and potential auto-disable logic.

Can I manually trigger a health check for a specific provider key?

Yes. Import checkKeyHealth from server/src/services/health.ts and pass the key ID as an argument. You can also use the /health HTTP endpoint with a query parameter to check individual keys, or call checkAllKeys() to validate the entire keyring programmatically.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}