# How FreeLLMAPI Implements Per-Key Rate Limit Tracking (RPM, RPD, TPM, TPD) and Cooldown Management

> Discover how FreeLLMAPI tracks per-key rate limits (RPM, RPD, TPM, TPD) and manages cooldowns using a hybrid in-memory and SQLite approach. Prevent provider throttling effectively.

- Repository: [Tashfeen/freellmapi](https://github.com/tashfeenahmed/freellmapi)
- Tags: internals
- Published: 2026-06-25

---

**FreeLLMAPI uses a hybrid in-memory sliding-window and SQLite persistence layer to track per-key usage across four axes (RPM, RPD, TPM, TPD), with escalating cooldown periods triggered by HTTP 429, 402, or 403 responses to prevent provider throttling.**

FreeLLMAPI manages rate limits for LLM providers through a sophisticated tracking system implemented in [`server/src/services/ratelimit.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/ratelimit.ts). The system monitors both request volume and token consumption across minute and day windows, ensuring compliance with provider quotas while maintaining crash resilience through database persistence.

## Sliding-Window Rate Tracking Architecture

The core mechanism relies on a composite key strategy that uniquely identifies each usage window.

### Composite Key Structure

Each rate limit window is indexed by a string encoding the **platform**, **model**, **key ID**, and the specific **axis** (`rpm`, `rpd`, `tpm`, or `tpd`). As defined in lines 11–13 of [`server/src/services/ratelimit.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/ratelimit.ts), this key format ensures granular tracking per API credential.

### In-Memory Window Maps

The system maintains a `Map<string, Window>` (lines 12–14) to store active sliding windows in memory. This provides O(1) lookup performance for hot-path rate limit checks during request routing.

### SQLite Persistence Layer

To survive process restarts, every request and token usage is written to the `rate_limit_usage` table. The `recordUsage` function (lines 49–55) inserts usage records, while the system reads persisted counts via SQL aggregation queries (lines 65–76 for requests, lines 86–96 for tokens). This hybrid approach ensures that `requestCount` and `tokenCount` functions (lines 19–23 and 31–35) report accurate cumulative usage even after service redeployment.

## Rate Limit Axes and Validation

FreeLLMAPI enforces four distinct consumption limits per API key.

### Request-Based Limits (RPM/RPD)

The `canMakeRequest` function (lines 38–55) validates against **Requests Per Minute (RPM)** and **Requests Per Day (RPD)**. It returns `false` when either sliding window exceeds the configured threshold, preventing quota exhaustion.

### Token-Based Limits (TPM/TPD)

Similarly, `canUseTokens` (lines 57–76) monitors **Tokens Per Minute (TPM)** and **Tokens Per Day (TPD)**. This is critical for models with large context windows where a single request might consume thousands of tokens.

### Provider-Wide Daily Caps

Some providers like OpenRouter enforce a single daily quota across all models. The system checks provider-specific caps (lines 93–100) by aggregating daily request counts across models (lines 22–33), acting as a gate before per-model checks (lines 38–42).

## Cooldown Management and Escalation

When providers signal rate limit violations, FreeLLMAPI implements a tiered cooldown strategy to prevent repeated failed attempts.

### Cooldown Triggers and Duration Tiers

The system distinguishes between transient and quota-based rate limiting:

- **Transient 429 (RPM/TPM)**: Fixed 90-second cooldown defined by `TRANSIENT_COOLDOWN_MS` (lines 1–3)
- **Daily quota exhaustion (RPD/TPD)**: Escalating durations from 2 minutes to 24 hours calculated by `getNextCooldownDuration` (lines 74–99) based on `cooldownHits` and `COOLDOWN_DURATIONS` tiers
- **Payment Required (402)**: 24-hour cooldown via `PAYMENT_REQUIRED_COOLDOWN_MS` (lines 4–9)
- **Model Forbidden (403)**: 24-hour cooldown via `MODEL_FORBIDDEN_COOLDOWN_MS` (lines 11–17)

### Persistent Cooldown Storage

Cooldowns are stored both in-memory (`Map<string, CooldownEntry>` at lines 73–75) and in the `rate_limit_cooldowns` table. The `setCooldown` function (lines 93–99) writes to both layers, while `isOnCooldown` (lines 100–121) checks the database first, cleaning expired entries before falling back to the in-memory map. Functions like `persistCooldown`, `persistedCooldownExpiry`, and `clearPersistedCooldown` (lines 59–71) manage the SQL lifecycle.

### Router Integration

The request router in [`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts) (line 578) calls `isOnCooldown` before attempting any provider request. If a key is benched, the router skips it automatically, implementing a circuit-breaker pattern that respects provider rate limits.

## Implementation Examples

Checking rate limits before routing:

```typescript
import { canMakeRequest, canUseTokens } from './services/ratelimit.js';

const limits = { rpm: 60, rpd: 1000, tpm: 20000, tpd: 500000 };
if (!canMakeRequest('openai', 'gpt-4', keyId, limits)) {
  console.warn('RPM/RPD limit reached – skip this key');
}
if (!canUseTokens('openai', 'gpt-4', keyId, estimatedTokens, limits)) {
  console.warn('TPM/TPD limit would be exceeded');
}

```

Recording usage after successful completion:

```typescript
import { recordRequest, recordTokens } from './services/ratelimit.js';

recordRequest('openai', 'gpt-4', keyId);
recordTokens('openai', 'gpt-4', keyId, tokensUsed);

```

Applying escalated cooldowns after rate limit errors:

```typescript
import { getCooldownDurationForLimit, setCooldown } from './services/ratelimit.js';

const cdMs = getCooldownDurationForLimit('openai', 'gpt-4', keyId, {
  rpd: limits.rpd,
  tpd: limits.tpd,
});
setCooldown('openai', 'gpt-4', keyId, cdMs);

```

Skipping benched keys in the router:

```typescript
import { isOnCooldown } from './services/ratelimit.js';

if (isOnCooldown(entry.platform, entry.model_id, key.id)) {
  continue; // Skip this key, it's currently cooling down
}

```

## Summary

- **Composite Key Strategy**: Rate limits are tracked per platform, model, key ID, and axis (RPM/RPD/TPM/TPD) using string-encoded keys in [`server/src/services/ratelimit.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/ratelimit.ts).
- **Hybrid Persistence**: In-memory sliding windows provide speed, while SQLite tables (`rate_limit_usage` and `rate_limit_cooldowns`) ensure durability across restarts.
- **Four-Axis Tracking**: The system independently monitors requests and tokens across both 60-second and 24-hour windows.
- **Tiered Cooldowns**: Transient 429 errors trigger 90-second pauses, while daily quota violations escalate from 2 minutes to 24 hours, with special 24-hour bans for 402/403 errors.
- **Router Integration**: The router consults `isOnCooldown` before each request, automatically skipping benched credentials to prevent provider throttling.

## Frequently Asked Questions

### How does FreeLLMAPI ensure rate limit data survives a server restart?

The system writes every request and token usage to the `rate_limit_usage` SQLite table via `recordUsage` (lines 49–55). When calculating current usage, the `requestCount` and `tokenCount` functions (lines 19–23 and 31–35) query these persisted records, ensuring that rate limit windows remain accurate even after the process restarts or crashes.

### What is the difference between transient and escalating cooldowns in FreeLLMAPI?

Transient cooldowns apply to minute-level rate limits (RPM/TPM) and last a fixed 90 seconds (`TRANSIENT_COOLDOWN_MS`). Escalating cooldowns apply when daily quotas (RPD/TPD) are exhausted, using `getNextCooldownDuration` (lines 74–99) to progressively increase ban durations from 2 minutes up to 24 hours based on how many times the key has hit its limit.

### Can FreeLLMAPI handle provider-wide daily caps across multiple models?

Yes. The system checks provider-wide daily limits in [`server/src/services/ratelimit.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/ratelimit.ts) (lines 93–100) by aggregating requests across all models for a given provider (lines 22–33). This prevents the router from exhausting shared quotas like OpenRouter's free tier daily allowance, even when individual model limits haven't been reached.

### Where does the router check for active cooldowns before sending requests?

The request router in [`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts) (line 578) calls `isOnCooldown` before attempting any provider request. This function checks both the in-memory cooldown map and the persisted `rate_limit_cooldowns` table, ensuring that recently rate-limited keys are temporarily excluded from the rotation until their cooldown expires.