How FreeLLMAPI Manages Rate Limit Counter Persistence: A Hybrid SQLite Approach

FreeLLMAPI implements a dual-layer persistence strategy that writes every request and token usage record to a SQLite database while maintaining hot in-memory sliding windows, ensuring rate limit counters survive process restarts without degrading real-time request throughput.

FreeLLMAPI is an open-source LLM routing proxy that enforces provider rate limits across multiple AI platforms. According to the source code in the tashfeenahmed/freellmapi repository, the system manages rate limit counter persistence through a hybrid architecture that combines durable SQLite storage with high-performance in-memory tracking.

Database Schema for Rate Limit Storage

The persistence layer relies on two tables created by the migration file server/src/db/migrations/20260101_000000_legacy_baseline.ts.

The rate_limit_usage Table

This table stores granular records for every request and token consumption event. The schema (lines 104-112) includes fields for platform, model, key identification, usage kind, and precise timestamps:

CREATE TABLE IF NOT EXISTS rate_limit_usage (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    platform TEXT NOT NULL,
    model_id TEXT NOT NULL,
    key_id INTEGER NOT NULL,
    kind TEXT NOT NULL CHECK (kind IN ('request','tokens')),
    tokens INTEGER NOT NULL DEFAULT 0,
    created_at_ms INTEGER NOT NULL,
    created_at TEXT NOT NULL DEFAULT (datetime('now'))
);

The rate_limit_cooldowns Table

The same migration creates rate_limit_cooldowns to store per-key expiration timestamps when models enter a cooldown period after hitting rate limits.

Recording Usage to the Database

In server/src/services/ratelimit.ts, the recordRequest function (lines 44-55) persists usage data immediately after updating in-memory windows. This ensures the SQLite database remains the authoritative source of truth:

export function recordRequest(platform, modelId, keyId) {
    const now = Date.now();
    // …push timestamps into in‑memory windows…
    recordUsage(platform, modelId, keyId, 'request', 0, now);
    clearNullLimitHits(platform, modelId, keyId);
}

The recordTokens function follows an identical pattern, writing token counts to the same table with kind set to 'tokens'.

Querying Persisted Counters

When performing limit checks, the system queries the database to calculate usage within sliding windows. The countPersistedRequests function (lines 58-76) demonstrates this aggregation:

function countPersistedRequests(..., windowMs, now) {
    return withDb(db => {
        const row = db.prepare(`
            SELECT COUNT(*) AS used
            FROM rate_limit_usage
            WHERE platform = ?
              AND model_id = ?
              AND key_id = ?
              AND kind = 'request'
              AND created_at_ms > ?
        `).get(platform, modelId, keyId, now - windowMs);
        return row.used;
    });
}

Token consumption uses sumPersistedTokens, which applies SUM(tokens) instead of COUNT(*). Both functions utilize the withDb helper to ensure connection management.

In-Memory Fallback and Sliding Windows

If the database is unavailable (when withDb returns undefined), FreeLLMAPI falls back to pure in-memory tracking. The module maintains a Map<string, Window> keyed by "platform:modelId:keyId:type" where type can be rpm, rpd, tpm, or tpd (lines 100-115).

The memoryRequestCount and memoryTokenCount functions provide the same sliding-window logic without SQLite overhead, ensuring rate limiting continues even during database outages.

Provider-Wide Daily Caps

Some providers (such as OpenRouter's free tier) impose shared daily request caps across all models for a single API key. FreeLLMAPI handles this through providerDailyRequestCount (lines 221-234), which aggregates persisted rows across all models for a given key. When the database is unreachable, it falls back to summing the per-model in-memory windows.

Persisting Cooldowns After Errors

After receiving 429, 402, or 403 responses, the router may bench a specific model and key combination. The expiry timestamp is stored in rate_limit_cooldowns via persistCooldown. On each request, isOnCooldown (lines 311-340) first queries the database and updates the in-memory cooldowns map if a persisted record exists, ensuring cooldowns survive restarts.

Practical Implementation Example

The following example demonstrates how to check limits before making requests and persist usage afterward:

import { canMakeRequest, canUseTokens, recordRequest, recordTokens } from './services/ratelimit.js';

const limits = {
  rpm: 20,   // requests per minute (may be null)
  rpd: 200,  // requests per day
  tpm: 250_000,
  tpd: null, // tokens per day unknown
};

if (canMakeRequest('openrouter', 'deepseek/deepseek-v3.1:free', 1, limits)) {
  // proceed with the call …
  // after a successful call:
  recordRequest('openrouter', 'deepseek/deepseek-v3.1:free', 1);
} else {
  console.warn('Rate limit exceeded – skip this model/key');
}

// Token‑budget check before sending a large prompt
const estimatedTokens = 1500;
if (canUseTokens('groq', 'llama-3.3-70b-versatile', 2, estimatedTokens, {
  tpm: 8000,
  tpd: null,
})) {
  // send the request
  recordTokens('groq', 'llama-3.3-70b-versatile', 2, estimatedTokens);
} else {
  console.warn('Token budget exceeded');
}

Summary

• Hybrid architecture: FreeLLMAPI combines SQLite persistence with in-memory sliding windows for both durability and performance.
• Granular storage: Every request and token event writes to rate_limit_usage, while cooldown states live in rate_limit_cooldowns.
• Automatic fallback: When withDb returns undefined, the system seamlessly switches to memoryRequestCount and memoryTokenCount without dropping limit enforcement.
• Provider-aware: Special handling for provider-wide daily caps aggregates usage across models when querying the database.
• Source location: All core logic resides in server/src/services/ratelimit.ts, with schema definitions in server/src/db/migrations/20260101_000000_legacy_baseline.ts.

Frequently Asked Questions

What database does FreeLLMAPI use for rate limit persistence?

FreeLLMAPI uses SQLite for all rate limit persistence. The database stores individual usage records in the rate_limit_usage table and cooldown timestamps in rate_limit_cooldowns, accessed through the withDb helper defined in the database module.

How does FreeLLMAPI handle rate limits when the database is offline?

The system implements a graceful degradation strategy. If the withDb helper returns undefined (indicating the SQLite connection is unavailable), FreeLLMAPI falls back to memoryRequestCount and memoryTokenCount functions that maintain sliding windows in a JavaScript Map. This ensures rate limiting continues to function even during database outages.

Does FreeLLMAPI support provider-wide daily request limits?

Yes. For providers like OpenRouter that enforce daily caps across all models for a given API key, FreeLLMAPI uses providerDailyRequestCount (lines 221-234). This function aggregates persisted records across all models when the database is available, or sums in-memory windows as a fallback.

How are cooldowns persisted after hitting a rate limit?

When the router encounters a 429, 402, or 403 response, it calls persistCooldown to write the expiration timestamp to the rate_limit_cooldowns table. On subsequent requests, isOnCooldown first checks this database table and synchronizes the result to an in-memory cache, ensuring cooldowns survive process restarts while remaining fast to query.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}