# How the Health Check Service Monitors Provider Keys in FreeLLMAPI

> Learn how FreeLLMAPI's health check service monitors provider keys, automatically disabling invalid ones and preserving those with transient network issues.

- Repository: [Tashfeen/freellmapi](https://github.com/tashfeenahmed/freellmapi)
- Tags: internals
- Published: 2026-06-24

---

**The FreeLLMAPI health check service runs every five minutes to validate stored provider API keys, automatically disabling any key that returns three consecutive invalid responses while preserving keys that encounter transient network errors.**

FreeLLMAPI manages provider API keys in a SQLite database table named `api_keys` and relies on a background health monitoring system to ensure request reliability. The health check service operates as a scheduled process within the server, periodically decrypting and testing each key against its respective provider to maintain accurate status records and prevent routing traffic to broken credentials.

## Health Check Architecture Overview

The health monitoring system consists of a central service that runs inside the server process alongside HTTP endpoints for manual intervention. The service maintains an in-memory failure counter map to track consecutive invalid responses without querying the database on every attempt.

### The Scheduling Mechanism

When the server boots, [`src/index.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/src/index.ts) invokes `startHealthChecker()` from [`src/services/health.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/src/services/health.ts) to initialize the background process. This creates a `setInterval` timer that fires every `CHECK_INTERVAL_MS` (5 minutes, or `5 * 60 * 1000` milliseconds), ensuring continuous monitoring without manual intervention.

## Step-by-Step Validation Workflow

The validation process follows a strict pipeline that handles each key individually, from database retrieval to provider-specific authentication testing.

### Fetching Enabled Keys

The `checkAllKeys()` function selects all rows from the `api_keys` table where `enabled = 1`. This query ensures the system only validates active keys, ignoring those already manually disabled. According to the source code in [`src/services/health.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/src/services/health.ts) (lines 60–63), this fetch operation initiates each monitoring cycle.

### Provider-Specific Validation

For each enabled key, the service calls `checkKeyHealth(keyId)`, which executes the following steps (lines 13–29 in [`health.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/health.ts)):

1. **Database Retrieval**: Reads the complete key record from SQLite.
2. **Provider Resolution**: Calls `resolveProvider` to determine the correct provider implementation (e.g., OpenAI, Anthropic).
3. **Decryption**: Decrypts the stored API key using the internal `decrypt` utility.
4. **Validation**: Invokes the provider's `validateKey` method with an options object containing `origin: 'health'` to distinguish health checks from production traffic.

### Status Determination and Database Updates

The service updates the `status` and `last_checked_at` columns based on the provider response:

- **Valid Response**: When `validateKey` returns `true`, the status is set to `'healthy'` and the in-memory failure counter for that key is cleared (lines 31–38).
- **Invalid Response**: If the provider returns `false` (indicating authentication failure or invalid credentials), the status becomes `'invalid'` and the service increments the consecutive failure counter (lines 39–46).
- **Transport Errors**: Network failures, DNS errors, timeouts, and TLS issues are caught in the `catch` block. The status is set to `'error'`, but **the failure counter is not incremented** because these errors do not prove the key itself is invalid (lines 50–56).

## Automatic Key Disabling Logic

FreeLLMAPI implements automatic circuit-breaking to prevent using compromised or expired credentials. The system defines a constant `CONSECUTIVE_FAILURES_TO_DISABLE = 3` that determines when a key should be automatically disabled.

When a key's failure counter reaches this threshold, the service executes an update query setting `enabled = 0` in the database. This ensures that three consecutive invalid responses permanently remove the key from the routing pool until an administrator manually re-enables it.

## Distinguishing Transport Errors from Invalid Keys

The health service differentiates between semantic validation failures and infrastructure problems. Transport-level errors—including connection timeouts, certificate validation failures, and DNS resolution errors—result in a status of `'error'` but do not count toward the auto-disable threshold. This design prevents temporary network blips from disabling valid keys while still surfacing connectivity issues for observability.

## Integration with Request Routing

The health status directly influences the request routing logic in [`src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/src/services/router.ts). The router filters available keys using `status IN ('healthy', 'unknown')` (lines 553–722), ensuring that keys marked as `'invalid'`, `'error'`, or `'rate_limited'` are never selected for production traffic. This integration guarantees that only verified, working credentials handle user requests.

## Manual Health Check Endpoints

The [`src/routes/health.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/src/routes/health.ts) file exposes HTTP endpoints for administrative oversight:

- **GET `/api/health`**: Returns a summary of health status across all providers.
- **POST `/api/health/check/:keyId`**: Forces immediate validation of a specific key ID, calling `checkKeyHealth` directly.
- **POST `/api/health/check-all`**: Triggers a bulk validation sweep equivalent to the scheduled check.

## Observability and Context Propagation

When the health service calls provider validation methods, it passes an `origin` field set to `'health'` in the context object. This allows the quota observation system in [`src/services/provider-quota.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/src/services/provider-quota.ts) (lines 20–21) to differentiate health check traffic from actual user requests, preventing monitoring operations from consuming rate limits or skewing usage analytics.

## Code Examples

### Triggering a Single Key Health Check via HTTP

```typescript
import fetch from 'node-fetch';

async function checkKey(keyId: number) {
  const resp = await fetch(`http://localhost:3001/api/health/check/${keyId}`, {
    method: 'POST',
  });
  const body = await resp.json();
  console.log(`Key ${keyId} status:`, body.status);
}

checkKey(42);

```

This endpoint invokes `checkKeyHealth(keyId)` internally, updating the database and returning the new status in the response.

### Forcing a Full Health Sweep

```bash
curl -X POST http://localhost:3001/api/health/check-all

```

This executes `checkAllKeys()`, which loops through every enabled key in the database.

### Using the Health Service Directly

```typescript
import { checkKeyHealth, startHealthChecker } from './services/health.js';

// Run a one-off validation for key 7
await checkKeyHealth(7);

// Start the periodic background checker (normally called from index.ts)
startHealthChecker();

```

### Example Provider Validation Implementation

```typescript
// src/providers/openai-compat.ts (excerpt)
export async function validateKey(apiKey: string, opts: ValidateOpts) {
  // Performs a lightweight request to the models endpoint
  const resp = await fetch(`${opts.baseUrl}/v1/models`, {
    headers: { Authorization: `Bearer ${apiKey}` },
  });
  return resp.ok; // Returns true for 2xx responses, false for 401/403
}

```

The health service supplies `{origin: 'health'}` in the options, allowing providers to implement special handling for health check requests if needed.

## Summary

- **Periodic Execution**: The service runs every 5 minutes via `setInterval` initialized in [`src/index.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/src/index.ts).
- **Selective Validation**: Only keys with `enabled = 1` are checked, pulling from the `api_keys` SQLite table.
- **Status States**: Keys cycle through `'healthy'`, `'invalid'`, `'error'`, and `'unknown'` states based on provider responses.
- **Auto-Disable Protection**: Three consecutive `'invalid'` results automatically set `enabled = 0`, while transport errors do not count toward this limit.
- **Routing Integration**: The request router in [`src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/src/services/router.ts) filters out unhealthy keys, ensuring traffic only hits verified credentials.
- **Administrative Access**: REST endpoints allow manual triggering of individual or bulk health checks.

## Frequently Asked Questions

### How often does FreeLLMAPI check provider key health?

The health check service runs automatically every five minutes. This interval is controlled by the `CHECK_INTERVAL_MS` constant (set to `300000` milliseconds) in [`src/services/health.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/src/services/health.ts), initialized when the server starts via `startHealthChecker()` in [`src/index.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/src/index.ts).

### What happens when a provider key fails validation?

When a key returns an invalid response, FreeLLMAPI sets its status to `'invalid'` and increments an in-memory failure counter. If the key fails three consecutive validations (`CONSECUTIVE_FAILURES_TO_DISABLE`), the system automatically sets `enabled = 0` in the database, removing it from the routing pool. Administrators must manually re-enable the key after investigating the failure.

### How does FreeLLMAPI handle network errors during health checks?

Network errors—including DNS failures, TLS certificate errors, and connection timeouts—are caught separately from authentication failures. The system sets the key status to `'error'` but **does not increment the failure counter**, preventing temporary connectivity issues from disabling valid keys. This distinction is handled in the `catch` block of `checkKeyHealth()` in [`src/services/health.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/src/services/health.ts) (lines 50–56).

### Can I manually trigger a health check for a specific API key?

Yes. FreeLLMAPI exposes a `POST /api/health/check/:keyId` endpoint that forces immediate validation of a specific key. This endpoint internally calls `checkKeyHealth()` and updates the database with the result. You can also trigger bulk validation for all enabled keys using `POST /api/health/check-all`, or query the current status summary via `GET /api/health`.