How the Health Check Service Monitors Provider Keys in FreeLLMAPI

The FreeLLMAPI health check service runs every five minutes to validate stored provider API keys, automatically disabling any key that returns three consecutive invalid responses while preserving keys that encounter transient network errors.

FreeLLMAPI manages provider API keys in a SQLite database table named api_keys and relies on a background health monitoring system to ensure request reliability. The health check service operates as a scheduled process within the server, periodically decrypting and testing each key against its respective provider to maintain accurate status records and prevent routing traffic to broken credentials.

Health Check Architecture Overview

The health monitoring system consists of a central service that runs inside the server process alongside HTTP endpoints for manual intervention. The service maintains an in-memory failure counter map to track consecutive invalid responses without querying the database on every attempt.

The Scheduling Mechanism

When the server boots, src/index.ts invokes startHealthChecker() from src/services/health.ts to initialize the background process. This creates a setInterval timer that fires every CHECK_INTERVAL_MS (5 minutes, or 5 * 60 * 1000 milliseconds), ensuring continuous monitoring without manual intervention.

Step-by-Step Validation Workflow

The validation process follows a strict pipeline that handles each key individually, from database retrieval to provider-specific authentication testing.

Fetching Enabled Keys

The checkAllKeys() function selects all rows from the api_keys table where enabled = 1. This query ensures the system only validates active keys, ignoring those already manually disabled. According to the source code in src/services/health.ts (lines 60–63), this fetch operation initiates each monitoring cycle.

Provider-Specific Validation

For each enabled key, the service calls checkKeyHealth(keyId), which executes the following steps (lines 13–29 in health.ts):

1. Database Retrieval: Reads the complete key record from SQLite.
2. Provider Resolution: Calls resolveProvider to determine the correct provider implementation (e.g., OpenAI, Anthropic).
3. Decryption: Decrypts the stored API key using the internal decrypt utility.
4. Validation: Invokes the provider's validateKey method with an options object containing origin: 'health' to distinguish health checks from production traffic.

Status Determination and Database Updates

The service updates the status and last_checked_at columns based on the provider response:

• Valid Response: When validateKey returns true, the status is set to 'healthy' and the in-memory failure counter for that key is cleared (lines 31–38).
• Invalid Response: If the provider returns false (indicating authentication failure or invalid credentials), the status becomes 'invalid' and the service increments the consecutive failure counter (lines 39–46).
• Transport Errors: Network failures, DNS errors, timeouts, and TLS issues are caught in the catch block. The status is set to 'error', but the failure counter is not incremented because these errors do not prove the key itself is invalid (lines 50–56).

Automatic Key Disabling Logic

FreeLLMAPI implements automatic circuit-breaking to prevent using compromised or expired credentials. The system defines a constant CONSECUTIVE_FAILURES_TO_DISABLE = 3 that determines when a key should be automatically disabled.

When a key's failure counter reaches this threshold, the service executes an update query setting enabled = 0 in the database. This ensures that three consecutive invalid responses permanently remove the key from the routing pool until an administrator manually re-enables it.

Distinguishing Transport Errors from Invalid Keys

The health service differentiates between semantic validation failures and infrastructure problems. Transport-level errors—including connection timeouts, certificate validation failures, and DNS resolution errors—result in a status of 'error' but do not count toward the auto-disable threshold. This design prevents temporary network blips from disabling valid keys while still surfacing connectivity issues for observability.

Integration with Request Routing

The health status directly influences the request routing logic in src/services/router.ts. The router filters available keys using status IN ('healthy', 'unknown') (lines 553–722), ensuring that keys marked as 'invalid', 'error', or 'rate_limited' are never selected for production traffic. This integration guarantees that only verified, working credentials handle user requests.

Manual Health Check Endpoints

The src/routes/health.ts file exposes HTTP endpoints for administrative oversight:

• GET /api/health: Returns a summary of health status across all providers.
• POST /api/health/check/:keyId: Forces immediate validation of a specific key ID, calling checkKeyHealth directly.
• POST /api/health/check-all: Triggers a bulk validation sweep equivalent to the scheduled check.

Observability and Context Propagation

When the health service calls provider validation methods, it passes an origin field set to 'health' in the context object. This allows the quota observation system in src/services/provider-quota.ts (lines 20–21) to differentiate health check traffic from actual user requests, preventing monitoring operations from consuming rate limits or skewing usage analytics.

Code Examples

Triggering a Single Key Health Check via HTTP

import fetch from 'node-fetch';

async function checkKey(keyId: number) {
  const resp = await fetch(`http://localhost:3001/api/health/check/${keyId}`, {
    method: 'POST',
  });
  const body = await resp.json();
  console.log(`Key ${keyId} status:`, body.status);
}

checkKey(42);

This endpoint invokes checkKeyHealth(keyId) internally, updating the database and returning the new status in the response.

Forcing a Full Health Sweep

curl -X POST http://localhost:3001/api/health/check-all

This executes checkAllKeys(), which loops through every enabled key in the database.

Using the Health Service Directly

import { checkKeyHealth, startHealthChecker } from './services/health.js';

// Run a one-off validation for key 7
await checkKeyHealth(7);

// Start the periodic background checker (normally called from index.ts)
startHealthChecker();

Example Provider Validation Implementation

// src/providers/openai-compat.ts (excerpt)
export async function validateKey(apiKey: string, opts: ValidateOpts) {
  // Performs a lightweight request to the models endpoint
  const resp = await fetch(`${opts.baseUrl}/v1/models`, {
    headers: { Authorization: `Bearer ${apiKey}` },
  });
  return resp.ok; // Returns true for 2xx responses, false for 401/403
}

The health service supplies {origin: 'health'} in the options, allowing providers to implement special handling for health check requests if needed.

Summary

• Periodic Execution: The service runs every 5 minutes via setInterval initialized in src/index.ts.
• Selective Validation: Only keys with enabled = 1 are checked, pulling from the api_keys SQLite table.
• Status States: Keys cycle through 'healthy', 'invalid', 'error', and 'unknown' states based on provider responses.
• Auto-Disable Protection: Three consecutive 'invalid' results automatically set enabled = 0, while transport errors do not count toward this limit.
• Routing Integration: The request router in src/services/router.ts filters out unhealthy keys, ensuring traffic only hits verified credentials.
• Administrative Access: REST endpoints allow manual triggering of individual or bulk health checks.

Frequently Asked Questions

How often does FreeLLMAPI check provider key health?

The health check service runs automatically every five minutes. This interval is controlled by the CHECK_INTERVAL_MS constant (set to 300000 milliseconds) in src/services/health.ts, initialized when the server starts via startHealthChecker() in src/index.ts.

What happens when a provider key fails validation?

When a key returns an invalid response, FreeLLMAPI sets its status to 'invalid' and increments an in-memory failure counter. If the key fails three consecutive validations (CONSECUTIVE_FAILURES_TO_DISABLE), the system automatically sets enabled = 0 in the database, removing it from the routing pool. Administrators must manually re-enable the key after investigating the failure.

How does FreeLLMAPI handle network errors during health checks?

Network errors—including DNS failures, TLS certificate errors, and connection timeouts—are caught separately from authentication failures. The system sets the key status to 'error' but does not increment the failure counter, preventing temporary connectivity issues from disabling valid keys. This distinction is handled in the catch block of checkKeyHealth() in src/services/health.ts (lines 50–56).

Can I manually trigger a health check for a specific API key?

Yes. FreeLLMAPI exposes a POST /api/health/check/:keyId endpoint that forces immediate validation of a specific key. This endpoint internally calls checkKeyHealth() and updates the database with the result. You can also trigger bulk validation for all enabled keys using POST /api/health/check-all, or query the current status summary via GET /api/health.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}