# How FreeLLMAPI Handles Rate Limiting Across Different Providers

> FreeLLMAPI manages rate limiting for various LLM providers using sliding-window counters and dynamic quota observation. Prevent quota exhaustion with escalating cooldowns.

- Repository: [Tashfeen/freellmapi](https://github.com/tashfeenahmed/freellmapi)
- Tags: deep-dive
- Published: 2026-06-24

---

**FreeLLMAPI implements a multi-layered rate limiting system that tracks requests and tokens per provider, model, and API key using sliding-window counters, while dynamically observing provider quotas and applying escalating cooldowns to prevent quota exhaustion.**

FreeLLMAPI (tashfeenahmed/freellmapi) manages rate limiting across different providers through a sophisticated routing layer that respects contractual quotas while maximizing availability. The system tracks usage at the provider-key-model level, maintains persistent counters in SQLite, and automatically falls back to alternative models when limits are reached.

## Multi-Layered Rate Limiting Architecture

The rate limiting system operates across five distinct layers to protect both the service and upstream providers from overconsumption. Each layer targets specific quota types, from per-minute request bursts to daily provider-wide aggregates. The architecture lives primarily in [`server/src/services/ratelimit.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/ratelimit.ts) and [`server/src/services/provider-quota.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/provider-quota.ts).

## Per-Minute and Per-Day Request and Token Caps

The core enforcement mechanism tracks **Requests Per Minute (RPM)**, **Requests Per Day (RPD)**, **Tokens Per Minute (TPM)**, and **Tokens Per Day (TPD)** for every provider-key-model combination.

Sliding-window counters stored in memory (`windows` map) provide real-time tracking, while the SQLite `rate_limit_usage` table ensures persistence across restarts. When the router evaluates a key, it calls `canMakeRequest()` for request quotas and `canUseTokens()` for token quotas, both defined in [`server/src/services/ratelimit.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/ratelimit.ts).

```typescript
import {
  canMakeRequest,
  canUseTokens,
} from './ratelimit';

const limits = {
  rpm: entry.rpm_limit,
  rpd: entry.rpd_limit,
  tpm: entry.tpm_limit,
  tpd: entry.tpd_limit,
};

if (!canMakeRequest(entry.platform, entry.model_id, key.id, limits)) {
  continue; // Skip to next key
}
if (!canUseTokens(entry.platform, entry.model_id, key.id, estimatedTokens, limits)) {
  continue;
}

```

## Provider-Wide Daily Caps

Some providers, such as OpenRouter's free tier, enforce a single daily request quota shared across all models on an account. FreeLLMAPI handles this through configurable environment variables and the `canUseProvider()` function.

The system reads caps from `PROVIDER_DAILY_REQUEST_CAP_<PLATFORM>` variables, falling back to `DEFAULT_PROVIDER_DAILY_REQUEST_CAPS`. The `providerDailyRequestCount` aggregates usage across every model for the same key, stored in the rate limit tables.

```bash

# Override OpenRouter daily cap

PROVIDER_DAILY_REQUEST_CAP_OPENROUTER=50

# Disable the cap entirely

PROVIDER_DAILY_REQUEST_CAP_OPENROUTER=0

```

## Dynamic Quota Observation from Response Headers

When providers return quota information in HTTP headers (e.g., `x-ratelimit-limit-requests`), FreeLLMAPI learns and adapts to actual upstream limits. The `parseQuotaObservationsFromResponse` function in [`server/src/services/provider-quota.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/provider-quota.ts) extracts limits, timestamps, and retry-after values.

These observations populate `provider_quota_state` and `provider_quota_observations` tables, feeding back into the router's decision-making to ensure the system respects provider-side constraints even when they differ from configured defaults.

## Cooldown Mechanisms and Error Handling

The system implements **exponential backoff** through escalating cooldown periods when providers return specific error codes. When a key receives a 429 (rate limit), 402 (payment required), or 403 (model forbidden) response, it enters a cooldown state.

The `isOnCooldown()` function checks against an in-memory `cooldowns` map backed by the `rate_limit_cooldowns` SQLite table. Cooldown durations escalate based on 24-hour throttle frequency: 2 minutes → 10 minutes → 1 hour → 24 hours.

```typescript
import { getNextCooldownDuration } from './ratelimit';

const nextMs = getNextCooldownDuration(platform, modelId, keyId);
// The router automatically applies cooldowns on 429 errors

```

## Router Integration and Fallback Logic

The request router ([`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts)) orchestrates rate limiting checks during key selection. The `selectKeyForModel` function validates keys sequentially using the rate limiting helpers.

```typescript
if (isOnCooldown(entry.platform, entry.model_id, kid)) continue;
if (!canUseProvider(entry.platform, kid)) continue;
if (!canMakeRequest(entry.platform, entry.model_id, kid, limits)) continue;
if (!canUseTokens(entry.platform, entry.model_id, kid, estimatedTokens, limits)) continue;

```

If all keys for a model are exhausted, the router proceeds to the next model in the fallback chain. When the entire chain is depleted, it throws a `RouteError` with status 429, signaling clients to wait for limit resets.

## Recording Usage

After successful requests, the system updates counters to maintain accurate sliding windows. The `recordRequest()` function increments request counts, while `recordTokens()` updates token consumption.

```typescript
import { recordRequest, recordTokens } from './ratelimit';

// After successful provider call
recordRequest(platform, modelId, keyId);

// After receiving response with token count
recordTokens(platform, modelId, keyId, tokens);

```

## Summary

- FreeLLMAPI enforces rate limiting at the provider-key-model granularity using sliding-window counters persisted in SQLite.
- The system tracks both request volume (RPM/RPD) and token consumption (TPM/TPD) through helpers in [`server/src/services/ratelimit.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/ratelimit.ts).
- Provider-wide daily caps accommodate shared quotas across models via environment variables and the `canUseProvider()` function.
- Dynamic quota observation parses provider response headers to learn actual limits in real-time via [`server/src/services/provider-quota.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/provider-quota.ts).
- Escalating cooldowns (2 minutes to 24 hours) protect against repeated rate limit errors while the router automatically falls back to alternative models.

## Frequently Asked Questions

### How does FreeLLMAPI handle different rate limit tiers between providers?

The system normalizes limits through the `limits` object passed to checking functions, with each provider-key-model combination maintaining independent counters. Provider-specific daily caps are configured via `PROVIDER_DAILY_REQUEST_CAP_<PLATFORM>` environment variables, allowing custom thresholds for each platform's unique constraints.

### What happens when a provider returns a 429 error?

When a provider returns HTTP 429 (rate limit), 402 (payment required), or 403 (model forbidden), FreeLLMAPI immediately places the key on cooldown using an escalating duration pattern. The cooldown starts at 2 minutes and increases to 10 minutes, 1 hour, or 24 hours based on how many times the key has been throttled in the previous 24-hour window.

### Can rate limit data survive server restarts?

Yes. While sliding-window counters operate in memory for performance, FreeLLMAPI persists rate limit usage and cooldown states to SQLite tables (`rate_limit_usage` and `rate_limit_cooldowns`). This ensures that quota tracking remains accurate across application restarts or deployments.

### How does the router decide which model to use when primary keys are rate limited?

The router evaluates keys sequentially using `isOnCooldown()`, `canUseProvider()`, `canMakeRequest()`, and `canUseTokens()`. If all keys for a model fail these checks, the router skips to the next model in the fallback chain. Only when the entire chain is exhausted does it return a 429 error to the client.