How the Health Checker Determines API Key Status in FreeLLMAPI

The health checker validates API keys by decrypting stored credentials and calling provider-specific validation, marking keys as 'healthy', 'invalid', or 'error' based on the response, and automatically disabling keys after three consecutive failures.

In the tashfeenahmed/freellmapi repository, the health checker is a critical background service that ensures only valid API keys remain active for LLM provider requests. This TypeScript-based system periodically verifies credential integrity by executing a multi-step validation pipeline defined in server/src/services/health.ts. Understanding how the health checker determines API key status helps operators maintain reliable service uptime and troubleshoot authentication failures proactively.

The Health Checking Architecture

The health monitoring system centers on server/src/services/health.ts, which exposes the core checkKeyHealth(keyId) function. A scheduler instantiated from src/lib/scheduler.ts drives automation by repeatedly invoking checkAllKeys(), which iterates through all enabled API keys and triggers individual health checks. Operators can programmatically control this lifecycle using startHealthChecker(scheduler) to initiate monitoring or stopHealthChecker() to halt background checks, typically invoking these during server startup and shutdown sequences.

Step-by-Step Validation Process

When checkKeyHealth(keyId) executes, it follows a rigorous seven-step workflow to determine the current status of a stored credential.

1. Load the Key Record

The process begins by querying the api_keys table using the provided keyId (line 16). This retrieves the encrypted key material, associated platform identifier, optional custom base_url, and current metadata required for validation.

2. Resolve the Provider Implementation

Using the stored platform and base_url values, the function calls resolveProvider (line 19) from server/src/providers/index.ts to instantiate a concrete provider class. This abstraction allows the health checker to support multiple LLM backends through a unified interface.

3. Decrypt the Stored Secret

Before validation, the encrypted key is decrypted using AES-256-GCM via the decrypt utility from server/src/lib/crypto.ts (line 23). The operation requires the saved initialization vector (IV) and authentication tag to securely reconstruct the plaintext credential without exposing it in logs or memory longer than necessary.

4. Execute Provider Validation

The provider's validateKey method receives the decrypted key along with contextual metadata including the platform type, quota pool identifier, endpoint set to models, and origin set to health (lines 24-30). This method performs a live network request to the provider's authentication endpoint to verify the key is active and authorized.

5. Classify the Status Outcome

Based on the provider's response, the checker assigns one of three distinct statuses (lines 50-66):

• healthy – The provider accepted the credentials and returned a successful authentication response.
• invalid – The provider explicitly rejected the key (HTTP 401/403), indicating revoked or expired credentials.
• error – A network-level failure occurred (DNS timeout, TLS handshake error, or connection refused) that prevented validation completion.

6. Persist Results to Database

The api_keys table is immediately updated with the new status value and a last_checked_at timestamp (lines 34-35, 63-64). This persistence ensures that subsequent API requests can reference current health states without triggering redundant validation calls.

7. Track Consecutive Failures

The system maintains a per-key failure counter to prevent repeated use of compromised credentials. When a key returns invalid, the counter increments (lines 40-42). Upon reaching CONSECUTIVE_FAILURES_TO_DISABLE (set to 3), the key is automatically disabled by setting enabled = 0 (lines 43-46). Successfully validated keys reset this counter to zero (line 38).

Automated Monitoring Configuration

By default, the scheduler runs checkAllKeys() at approximately five-minute intervals, ensuring continuous surveillance of the credential pool. Each cycle fetches all enabled keys from the database and dispatches them individually through checkKeyHealth.

import { Scheduler } from './lib/scheduler';
import { startHealthChecker } from './services/health';

const scheduler = new Scheduler();
startHealthChecker(scheduler);

To cease monitoring gracefully, call stopHealthChecker(), which terminates the recurring task without affecting in-flight validations.

Debugging and Manual Validation

For on-demand diagnostics or troubleshooting specific keys, developers can bypass the scheduler and invoke checkKeyHealth directly:

import { checkKeyHealth } from './services/health';

async function debugKey(keyId: number) {
  const status = await checkKeyHealth(keyId);
  console.log(`Key ${keyId} health status: ${status}`);
}

Interpreting Health Status Values

Status	Meaning	Action Taken
healthy	Valid credentials confirmed by provider	Failure counter reset
invalid	Authentication rejected by provider	Failure counter incremented; key disabled after 3 failures
error	Network/DNS/TLS failure prevented validation	No failure counter change; retry on next cycle

Summary

• The health checker resides in server/src/services/health.ts and operates via checkKeyHealth(keyId).
• Validation requires decrypting AES-encrypted keys and calling provider-specific validateKey methods.
• Three statuses exist: healthy, invalid, and error, with only invalid incrementing the failure counter.
• Keys automatically disable after three consecutive invalid results via CONSECUTIVE_FAILURES_TO_DISABLE.
• The scheduler runs checks every five minutes by default, controllable via startHealthChecker() and stopHealthChecker().

Frequently Asked Questions

What are the three possible API key health statuses?

The system classifies keys as healthy when credentials are valid, invalid when the provider rejects authentication, and error when network or TLS issues prevent reaching the provider. Only invalid statuses count toward the automatic disabling threshold.

How does the health checker handle network timeouts?

Network, DNS, and TLS errors are caught as exceptions and mapped to the error status (lines 50-66). Unlike invalid responses, these transient failures do not increment the consecutive failure counter, allowing the system to retry validation on the next scheduled cycle without penalizing the key.

When does the health checker automatically disable an API key?

A key disables automatically when the internal failure counter reaches CONSECUTIVE_FAILURES_TO_DISABLE, which is hardcoded to 3 (lines 43-46). This occurs only after three separate validation cycles return invalid, not error, ensuring temporary network blips do not deactivate valid credentials.

Can I manually check a single API key without waiting for the scheduler?

Yes. Import checkKeyHealth from server/src/services/health.ts and await the result with a specific keyId. This bypasses the checkAllKeys() batch process and immediately executes the seven-step validation pipeline, returning the current status string for debugging or administrative purposes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}