How FreeLLMAPI Implements Per-Key Rate Limit Tracking (RPM, RPD, TPM, TPD) and Cooldown Management

FreeLLMAPI uses a hybrid in-memory sliding-window and SQLite persistence layer to track per-key usage across four axes (RPM, RPD, TPM, TPD), with escalating cooldown periods triggered by HTTP 429, 402, or 403 responses to prevent provider throttling.

FreeLLMAPI manages rate limits for LLM providers through a sophisticated tracking system implemented in server/src/services/ratelimit.ts. The system monitors both request volume and token consumption across minute and day windows, ensuring compliance with provider quotas while maintaining crash resilience through database persistence.

Sliding-Window Rate Tracking Architecture

The core mechanism relies on a composite key strategy that uniquely identifies each usage window.

Composite Key Structure

Each rate limit window is indexed by a string encoding the platform, model, key ID, and the specific axis (rpm, rpd, tpm, or tpd). As defined in lines 11–13 of server/src/services/ratelimit.ts, this key format ensures granular tracking per API credential.

In-Memory Window Maps

The system maintains a Map<string, Window> (lines 12–14) to store active sliding windows in memory. This provides O(1) lookup performance for hot-path rate limit checks during request routing.

SQLite Persistence Layer

To survive process restarts, every request and token usage is written to the rate_limit_usage table. The recordUsage function (lines 49–55) inserts usage records, while the system reads persisted counts via SQL aggregation queries (lines 65–76 for requests, lines 86–96 for tokens). This hybrid approach ensures that requestCount and tokenCount functions (lines 19–23 and 31–35) report accurate cumulative usage even after service redeployment.

Rate Limit Axes and Validation

FreeLLMAPI enforces four distinct consumption limits per API key.

Request-Based Limits (RPM/RPD)

The canMakeRequest function (lines 38–55) validates against Requests Per Minute (RPM) and Requests Per Day (RPD). It returns false when either sliding window exceeds the configured threshold, preventing quota exhaustion.

Token-Based Limits (TPM/TPD)

Similarly, canUseTokens (lines 57–76) monitors Tokens Per Minute (TPM) and Tokens Per Day (TPD). This is critical for models with large context windows where a single request might consume thousands of tokens.

Provider-Wide Daily Caps

Some providers like OpenRouter enforce a single daily quota across all models. The system checks provider-specific caps (lines 93–100) by aggregating daily request counts across models (lines 22–33), acting as a gate before per-model checks (lines 38–42).

Cooldown Management and Escalation

When providers signal rate limit violations, FreeLLMAPI implements a tiered cooldown strategy to prevent repeated failed attempts.

Cooldown Triggers and Duration Tiers

The system distinguishes between transient and quota-based rate limiting:

• Transient 429 (RPM/TPM): Fixed 90-second cooldown defined by TRANSIENT_COOLDOWN_MS (lines 1–3)
• Daily quota exhaustion (RPD/TPD): Escalating durations from 2 minutes to 24 hours calculated by getNextCooldownDuration (lines 74–99) based on cooldownHits and COOLDOWN_DURATIONS tiers
• Payment Required (402): 24-hour cooldown via PAYMENT_REQUIRED_COOLDOWN_MS (lines 4–9)
• Model Forbidden (403): 24-hour cooldown via MODEL_FORBIDDEN_COOLDOWN_MS (lines 11–17)

Persistent Cooldown Storage

Cooldowns are stored both in-memory (Map<string, CooldownEntry> at lines 73–75) and in the rate_limit_cooldowns table. The setCooldown function (lines 93–99) writes to both layers, while isOnCooldown (lines 100–121) checks the database first, cleaning expired entries before falling back to the in-memory map. Functions like persistCooldown, persistedCooldownExpiry, and clearPersistedCooldown (lines 59–71) manage the SQL lifecycle.

Router Integration

The request router in server/src/services/router.ts (line 578) calls isOnCooldown before attempting any provider request. If a key is benched, the router skips it automatically, implementing a circuit-breaker pattern that respects provider rate limits.

Implementation Examples

Checking rate limits before routing:

import { canMakeRequest, canUseTokens } from './services/ratelimit.js';

const limits = { rpm: 60, rpd: 1000, tpm: 20000, tpd: 500000 };
if (!canMakeRequest('openai', 'gpt-4', keyId, limits)) {
  console.warn('RPM/RPD limit reached – skip this key');
}
if (!canUseTokens('openai', 'gpt-4', keyId, estimatedTokens, limits)) {
  console.warn('TPM/TPD limit would be exceeded');
}

Recording usage after successful completion:

import { recordRequest, recordTokens } from './services/ratelimit.js';

recordRequest('openai', 'gpt-4', keyId);
recordTokens('openai', 'gpt-4', keyId, tokensUsed);

Applying escalated cooldowns after rate limit errors:

import { getCooldownDurationForLimit, setCooldown } from './services/ratelimit.js';

const cdMs = getCooldownDurationForLimit('openai', 'gpt-4', keyId, {
  rpd: limits.rpd,
  tpd: limits.tpd,
});
setCooldown('openai', 'gpt-4', keyId, cdMs);

Skipping benched keys in the router:

import { isOnCooldown } from './services/ratelimit.js';

if (isOnCooldown(entry.platform, entry.model_id, key.id)) {
  continue; // Skip this key, it's currently cooling down
}

Summary

• Composite Key Strategy: Rate limits are tracked per platform, model, key ID, and axis (RPM/RPD/TPM/TPD) using string-encoded keys in server/src/services/ratelimit.ts.
• Hybrid Persistence: In-memory sliding windows provide speed, while SQLite tables (rate_limit_usage and rate_limit_cooldowns) ensure durability across restarts.
• Four-Axis Tracking: The system independently monitors requests and tokens across both 60-second and 24-hour windows.
• Tiered Cooldowns: Transient 429 errors trigger 90-second pauses, while daily quota violations escalate from 2 minutes to 24 hours, with special 24-hour bans for 402/403 errors.
• Router Integration: The router consults isOnCooldown before each request, automatically skipping benched credentials to prevent provider throttling.

Frequently Asked Questions

How does FreeLLMAPI ensure rate limit data survives a server restart?

The system writes every request and token usage to the rate_limit_usage SQLite table via recordUsage (lines 49–55). When calculating current usage, the requestCount and tokenCount functions (lines 19–23 and 31–35) query these persisted records, ensuring that rate limit windows remain accurate even after the process restarts or crashes.

What is the difference between transient and escalating cooldowns in FreeLLMAPI?

Transient cooldowns apply to minute-level rate limits (RPM/TPM) and last a fixed 90 seconds (TRANSIENT_COOLDOWN_MS). Escalating cooldowns apply when daily quotas (RPD/TPD) are exhausted, using getNextCooldownDuration (lines 74–99) to progressively increase ban durations from 2 minutes up to 24 hours based on how many times the key has hit its limit.

Can FreeLLMAPI handle provider-wide daily caps across multiple models?

Yes. The system checks provider-wide daily limits in server/src/services/ratelimit.ts (lines 93–100) by aggregating requests across all models for a given provider (lines 22–33). This prevents the router from exhausting shared quotas like OpenRouter's free tier daily allowance, even when individual model limits haven't been reached.

Where does the router check for active cooldowns before sending requests?

The request router in server/src/services/router.ts (line 578) calls isOnCooldown before attempting any provider request. This function checks both the in-memory cooldown map and the persisted rate_limit_cooldowns table, ensuring that recently rate-limited keys are temporarily excluded from the rotation until their cooldown expires.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}