What Is the Health Checking Service in FreeLLMAPI?

The health checking service in FreeLLMAPI periodically validates every stored API key every 5 minutes, updates the SQLite database with healthy, invalid, or error statuses, and automatically disables keys after three consecutive failures to ensure only valid credentials are used for LLM requests.

FreeLLMAPI is an open-source API gateway that aggregates multiple LLM provider keys. The health checking service runs as a background watchdog that maintains the integrity of the credential store by continuously verifying which keys remain active and isolating those that fail validation.

Core Responsibilities of the Health Checking Service

The service operates as a scheduled background job that performs three critical functions: validating credentials against provider endpoints, persisting status metadata, and preventing degraded keys from entering the request pipeline.

Periodic Validation Cycle

Every 5 minutes (CHECK_INTERVAL_MS), the service queries the SQLite database for all active keys using SELECT id, platform FROM api_keys WHERE enabled = 1. For each record returned, it resolves the appropriate provider via resolveProvider and invokes the provider-specific validateKey routine, typically against the provider's models endpoint. This confirms that the API key is still authorized and functional for that platform.

Status Tracking and Persistence

Validation results are categorized into a KeyStatus enum with three states: healthy (validation succeeded), invalid (explicit rejection by provider), or error (network/transport failure). The service updates the status column and the last_checked_at timestamp in server/src/services/health.ts for every key processed, creating a real-time audit trail visible to administrators.

Automatic Failure Handling and Key Disabling

Beyond monitoring, the service implements automated remediation by tracking failure patterns and disabling compromised keys before they impact user requests.

The Three-Strike Failure Threshold

When a key fails validation with isValid === false, the service increments a per-key failure counter stored in the database. If the counter reaches three consecutive failures (CONSECUTIVE_FAILURES_TO_DISABLE), the service automatically sets enabled = 0 for that key in the api_keys table, effectively removing it from the rotation pool.

Distinguishing Transport Errors from Invalid Keys

The service differentiates between credential issues and transient infrastructure problems. Transport-level errors—including DNS failures, connection timeouts, and TLS handshake issues—are recorded as error status but do not increment the failure counter. This design prevents temporary provider outages from unnecessarily disabling valid API keys.

Scheduler Integration and Service Lifecycle

The health checker integrates with the application's job scheduling abstraction to manage its execution lifecycle cleanly without blocking the main server thread.

NodeScheduler Abstraction

The service relies on the Scheduler interface defined in server/src/lib/scheduler.ts, with the concrete implementation NodeScheduler providing the execution context. This abstraction allows the health checks to run as recurring background tasks using scheduler.every to register the interval job.

Starting and Stopping the Service

The startHealthChecker(scheduler) function accepts a Scheduler instance and registers the recurring validation job that invokes checkAllKeys(). The service can be gracefully shut down via stopHealthChecker(), which cleans up scheduled jobs. The entry point in server/src/index.ts bootstraps this by creating a NodeScheduler instance and passing it to the health checker on server startup.

Implementation Details and Code Examples

The following patterns demonstrate how to interact with the health checking system programmatically.

Bootstrapping the Health Checker

To initialize the service when the server starts:

// Typical pattern from server/src/index.ts
import { NodeScheduler } from './lib/scheduler.js';
import { startHealthChecker } from './services/health.js';

const scheduler = new NodeScheduler();
startHealthChecker(scheduler);

Manual Key Validation

For administrative scripts or CLI tools, validate a specific key on demand using checkKeyHealth:

import { checkKeyHealth } from './services/health.js';

async function probeKey(keyId: number) {
  const status = await checkKeyHealth(keyId);
  console.log(`Key ${keyId} status → ${status}`);
}

Querying Health Status from the Database

Access the latest validation states directly from the SQLite database:

import { getDb } from '../db/index.js';

function listKeyHealth() {
  const db = getDb();
  const rows = db
    .prepare(`SELECT id, platform, status, enabled FROM api_keys`)
    .all();
  console.table(rows);
}

Key Files in the Repository

• server/src/services/health.ts: Contains checkKeyHealth, checkAllKeys, startHealthChecker, and stopHealthChecker implementations.
• server/src/lib/scheduler.ts: Defines the Scheduler interface and NodeScheduler class used for job scheduling.
• server/src/services/provider-quota.ts: Provides inferQuotaPoolKey, utilized during validation to attach quota metadata.
• server/src/index.ts: Server entry point that instantiates the scheduler and launches the health checking service.

Summary

• The health checking service runs every 5 minutes to validate all enabled API keys against their respective LLM providers.
• It updates the status and last_checked_at columns in the SQLite database, categorizing results as healthy, invalid, or error.
• Keys failing validation three consecutive times are automatically disabled (enabled = 0), while transport errors do not count toward this threshold.
• The service integrates with NodeScheduler in server/src/lib/scheduler.ts for background execution and can be controlled via startHealthChecker and stopHealthChecker.

Frequently Asked Questions

How often does FreeLLMAPI check the health of API keys?

The service validates all enabled keys every 5 minutes using the CHECK_INTERVAL_MS constant. This interval balances the freshness of status data against API rate limit considerations and provider load.

What happens when an API key fails validation?

A failed validation (where isValid === false) increments a per-key failure counter. Upon reaching three consecutive failures (CONSECUTIVE_FAILURES_TO_DISABLE), the service automatically sets the key's enabled flag to 0 in the database, preventing it from being selected for future requests.

Are network timeouts counted as validation failures?

No. Transport-level errors such as DNS failures, connection timeouts, and TLS errors are recorded with an error status but explicitly do not increment the failure counter. This prevents temporary network issues from disabling otherwise valid API keys.

Can I manually trigger a health check for a specific key?

Yes. Import checkKeyHealth from server/src/services/health.ts and pass the key ID to validate a single credential on demand. This is useful for debugging or administrative verification outside the normal 5-minute cycle.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}