How FreeLLMAPI Manages Escalating Cooldowns: Technical Deep Dive

FreeLLMAPI implements an escalating cooldown system that tracks rate-limit violations across a rolling 24-hour window, dynamically increasing ban durations from seconds up to 24 hours based on violation frequency while allowing early recovery via heuristic probes.

FreeLLMAPI protects downstream LLM providers from abusive traffic patterns through a sophisticated cooldown mechanism implemented in TypeScript. The system, located primarily in server/src/services/ratelimit.ts, uses an in-memory cache backed by SQLite to track key health and automatically escalates penalties for repeatedly failing keys.

Core Architecture of the Cooldown System

Dual-Layer Storage with In-Memory Maps

The system maintains two primary data structures in memory to balance performance with persistence. The cooldowns Map stores the active expiry timestamp for each key-model-platform triple using a composite key format (platform:modelId:keyId:cooldown). This allows O(1) lookup times when the router checks if a key is banned before sending a request.

// In-memory storage structure (ratelimit.ts)
const cooldowns = new Map<string, number>();  // key -> expiry timestamp

Source: ratelimit.ts #L839‑L842

The cooldownHits Escalation Tracker

Parallel to the expiry map, the system tracks violation history in a cooldownHits Map. Every time setCooldown() is invoked—whether from an explicit rate-limit response or a heuristic failure—the current timestamp is appended to an array associated with that key. This history powers the escalation logic by revealing how frequently a specific key has entered cooldown within the monitoring window.

// Tracking violation timestamps for escalation calculation
cooldownHits.set(compositeKey, [
  ...(cooldownHits.get(compositeKey) || []),
  Date.now()
]);

Source: ratelimit.ts #L866‑L868

Rolling Window Calculation and Duration Escalation

24-Hour Violation Window

When determining the severity of a new cooldown, the system filters the cooldownHits array to consider only events within the last 24 hours (defined as the DAY constant). This rolling window prevents ancient violations from permanently marking a key as high-risk while ensuring recent abuse patterns factor into the penalty calculation.

// Filtering hits to rolling 24-hour window
const recentHits = hits.filter(timestamp => 
  Date.now() - timestamp < DAY
);

Source: ratelimit.ts #L880‑L882

Dynamic Duration Calculation

The heart of the escalation mechanism resides in the getNextCooldownDuration logic (spanning approximately lines 856–910). This function calculates the appropriate penalty duration based on the count of recent hits:

• Low violation count: Applies a minimal penalty (approximately 90 seconds) for occasional rate-limit hits
• Escalating thresholds: Progressively increases duration as hit counts rise through defined thresholds
• Maximum ceiling: Hard-caps penalties at 24 hours to prevent indefinite banning

The implementation uses a tiered approach where each additional violation within the 24-hour window pushes the key into a higher severity bracket, ensuring that persistent offenders face exponentially longer exclusion periods while transient errors receive minimal punishment.

Source: ratelimit.ts #L856‑L910

Persistence and Lifecycle Management

SQLite Storage Schema

Beyond in-memory tracking, every cooldown persists to the rate_limit_cooldowns table in SQLite. This dual-write strategy ensures that cooldown states survive process restarts and horizontal scaling scenarios. The setCooldown function handles both the Map insertion and the database UPSERT operation atomically.

// Persistence logic example
await db.run(`
  INSERT INTO rate_limit_cooldowns 
  (platform, model_id, key_id, expires_at, source) 
  VALUES (?, ?, ?, ?, ?)
  ON CONFLICT(platform, model_id, key_id) 
  DO UPDATE SET expires_at = excluded.expires_at
`, [platform, modelId, keyId, expiryTimestamp, source]);

Source: ratelimit.ts #L1185‑L1198

Automated Cleanup of Expired Cooldowns

A background maintenance routine periodically prunes both the in-memory cooldowns Map and the SQLite database of entries where expires_at < Date.now(). This prevents memory bloat and database growth from accumulating stale penalty records.

// Cleanup routine removes expired entries
for (const [key, expiry] of cooldowns.entries()) {
  if (expiry < Date.now()) {
    cooldowns.delete(key);
    await db.run('DELETE FROM rate_limit_cooldowns WHERE key = ?', [key]);
  }
}

Source: ratelimit.ts #L1156‑L1173

Early Recovery via Heuristic Probing

The cooldown-probe.ts service implements a forgiveness mechanism for heuristic cooldowns (those triggered by error patterns rather than explicit 429 responses). A background job periodically attempts small test requests against keys currently in cooldown. If the probe succeeds, the system immediately clears the cooldown through maybeClearCooldown, restoring the key to the active pool ahead of schedule.

This prevents transient network blips or temporary provider maintenance from sidelining keys for the full escalated duration, while still maintaining protection against genuinely broken or rate-limited credentials.

Source: cooldown-probe.ts

Integration with Request Routing

The router (router.ts) consults the cooldown system before dispatching requests to providers. By checking isOnCooldown against the in-memory Map, the system avoids routing traffic to penalized keys, effectively load-balancing around broken credentials without wasting upstream requests.

Summary

• Dual storage: FreeLLMAPI uses an in-memory Map for fast lookups and SQLite for durability across restarts
• Escalation tracking: The cooldownHits Map maintains a 24-hour rolling window of violations per key
• Dynamic penalties: getNextCooldownDuration calculates ban lengths from ~90 seconds up to 24 hours based on recent violation frequency
• Automatic cleanup: Expired entries are purged from both memory and database to prevent resource leaks
• Probe forgiveness: The probe service allows early cooldown clearance when keys demonstrate restored health

Frequently Asked Questions

How does FreeLLMAPI determine the length of a cooldown penalty?

FreeLLMAPI calculates cooldown duration by counting violations in the last 24 hours from the cooldownHits Map. The getNextCooldownDuration function applies tiered logic where each additional violation increases the penalty, starting at approximately 90 seconds and escalating to a maximum of 24 hours according to the implementation in lines 856–910 of ratelimit.ts.

What happens to cooldowns when the server restarts?

Because the system writes all cooldowns to the rate_limit_cooldowns table in SQLite, persisted penalties survive process restarts. On startup, the service hydrates the in-memory cooldowns Map from the database, ensuring no loss of rate-limit protection state.

Can a key be released from cooldown before the expiry time?

Yes. The cooldown-probe.ts service periodically tests keys under heuristic cooldown. If a probe request succeeds, maybeClearCooldown removes the entry from both the in-memory Map and the database, allowing immediate reuse of the key ahead of the scheduled expiry.

Where does the 24-hour escalation window get defined?

The rolling window constant (DAY) is defined within ratelimit.ts and used in the filter logic around line 880. When evaluating escalation, the system filters the cooldownHits array to include only timestamps where Date.now() - timestamp < DAY, ensuring only recent violations influence the penalty calculation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}