# How FreeLLMAPI Handles Rate Limiting for LLM Providers: A Technical Deep Dive

> Discover how FreeLLMAPI manages rate limiting across LLM providers with per key, per model, and provider-wide caps for reliable request routing. Learn about sliding windows and exponential backoff.

- Repository: [Tashfeen/freellmapi](https://github.com/tashfeenahmed/freellmapi)
- Tags: deep-dive
- Published: 2026-06-26

---

**FreeLLMAPI enforces rate limits at multiple layers—per key, per model, and per provider—using sliding-window counters, provider-wide daily caps, and exponential backoff cooldowns to ensure reliable request routing across diverse LLM platforms.**

FreeLLMAPI is an open-source router that aggregates multiple LLM providers behind a single API. To prevent quota exhaustion and ensure fair usage across different upstream services, the system implements a sophisticated rate limiting strategy. This article examines how the codebase in `tashfeenahmed/freellmapi` manages request throttling, token consumption, and automatic failover when limits are reached.

## The Multi-Layer Rate Limiting Architecture

The rate limiting system operates across five distinct layers, each targeting different quota dimensions from individual requests to provider-wide daily aggregates.

### Per-Minute and Per-Day Request Caps (RPM/RPD)

At the most granular level, FreeLLMAPI tracks **requests per minute (RPM)** and **requests per day (RPD)** for every unique *provider → key → model* combination. 

The system maintains **sliding-window counters** in an in-memory `windows` map, backed by persistent storage in the SQLite `rate_limit_usage` table. This dual approach ensures that quota state survives server restarts while maintaining high-performance lookups during request routing.

According to the source code in [`server/src/services/ratelimit.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/ratelimit.ts), the router checks these limits via the `canMakeRequest()` function before selecting a key for any model.

### Token-Based Limits (TPM/TPD)

Beyond request counts, FreeLLMAPI enforces **tokens per minute (TPM)** and **tokens per day (TPD)** limits. These constraints account for the total token consumption (prompt + completion) rather than just the number of API calls.

The tracking mechanism mirrors the request-based approach—using the same sliding-window infrastructure—but increments counters based on token counts instead of timestamps. The `canUseTokens()` function in [`server/src/services/ratelimit.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/ratelimit.ts) validates these limits before routing, accepting an `estimatedTokens` parameter to preemptively check capacity.

### Provider-Wide Daily Caps

Some providers, notably OpenRouter's free tier, enforce a single daily quota shared across **all** models on an account. FreeLLMAPI handles this through the `canUseProvider()` check.

These caps are configured via environment variables following the pattern `PROVIDER_DAILY_REQUEST_CAP_<PLATFORM>`, with fallback defaults defined in `DEFAULT_PROVIDER_DAILY_REQUEST_CAPS`. The system aggregates usage across every model for the same key using the `providerDailyRequestCount` tracker, ensuring that switching models within the same provider does not circumvent daily limits.

```bash

# Override OpenRouter's daily cap in your .env file

PROVIDER_DAILY_REQUEST_CAP_OPENROUTER=50

# Set to 0 to disable the cap entirely

PROVIDER_DAILY_REQUEST_CAP_OPENROUTER=0

```

### Cooldown and Error Handling

When a provider returns specific HTTP status codes—**429 (Rate Limit)**, **402 (Payment Required)**, or **403 (Model Forbidden)**—FreeLLMAPI places the key/model combination into a cooldown state.

The `cooldowns` map stores expiry timestamps in memory, with persistent backup in the `rate_limit_cooldowns` table. The duration follows an **escalating backoff strategy**: 2 minutes → 10 minutes → 1 hour → 24 hours, based on how many times the key has been throttled in the last 24 hours.

The `isOnCooldown()` function checks this state during key selection, and `getNextCooldownDuration()` calculates the appropriate penalty period.

### Dynamic Quota Observation

FreeLLMAPI adapts to provider-specific limit changes by parsing response headers. The `parseQuotaObservationsFromResponse()` function in [`server/src/services/provider-quota.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/provider-quota.ts) extracts official limits, timestamps, and retry-after values from headers like `x-ratelimit-limit-requests`.

These observations populate the `provider_quota_state` and `provider_quota_observations` tables, creating a feedback loop where the router learns and respects the provider's current quota configuration rather than relying solely on static configuration.

## How the Router Enforces Limits

The request routing logic in [`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts) implements a sequential validation pipeline when selecting keys for a model. The router iterates through available keys and applies the following checks in order:

```typescript
if (isOnCooldown(entry.platform, entry.model_id, kid)) continue;
if (!canUseProvider(entry.platform, kid)) continue;                 // Provider-wide daily cap
if (!canMakeRequest(entry.platform, entry.model_id, kid, limits)) continue; // RPM/RPD
if (!canUseTokens(entry.platform, entry.model_id, kid, estimatedTokens, limits)) continue; // TPM/TPD

```

If any check fails, the router skips that key and attempts the next available one. When **all** keys for a specific model are exhausted, the system falls back to the next model in the chain. Only when the entire fallback chain is depleted does FreeLLMAPI throw a `RouteError` (HTTP 429), signaling that the client must wait for quota reset.

## Implementation Examples

### Checking Rate Limit Status Before Routing

When implementing custom routing logic or extending the selector, use the public helper functions from [`server/src/services/ratelimit.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/ratelimit.ts):

```typescript
import {
  canMakeRequest,
  canUseTokens,
  canUseProvider,
  isOnCooldown,
} from './ratelimit';

// Limits are derived from the model configuration
const limits = {
  rpm: entry.rpm_limit,
  rpd: entry.rpd_limit,
  tpm: entry.tpm_limit,
  tpd: entry.tpd_limit,
};

// Comprehensive check before using a key
if (
  isOnCooldown(entry.platform, entry.model_id, key.id) ||
  !canUseProvider(entry.platform, key.id) ||
  !canMakeRequest(entry.platform, entry.model_id, key.id, limits) ||
  !canUseTokens(entry.platform, entry.model_id, key.id, estimatedTokens, limits)
) {
  continue; // Skip to next key
}

```

### Recording Usage After Successful Requests

After a successful API call, update the counters to maintain accurate sliding-window state:

```typescript
import { recordRequest, recordTokens } from './ratelimit';

// Increment request counter
recordRequest(platform, modelId, keyId);

// Update token consumption (prompt + completion)
recordTokens(platform, modelId, keyId, tokens);

```

### Calculating Cooldown Duration

To determine how long a key should be benched after hitting a rate limit:

```typescript
import { getNextCooldownDuration } from './ratelimit';

const nextCooldownMs = getNextCooldownDuration(platform, modelId, keyId);
// Returns: 120000 (2min), 600000 (10min), 3600000 (1hr), or 86400000 (24hr)

```

## Summary

- **Multi-layer protection**: FreeLLMAPI enforces limits at the request, token, and provider-wide levels using [`server/src/services/ratelimit.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/ratelimit.ts).
- **Sliding-window tracking**: RPM/RPD and TPM/TPD counters use in-memory `windows` maps with SQLite persistence via `rate_limit_usage`.
- **Provider-wide daily caps**: Configurable via `PROVIDER_DAILY_REQUEST_CAP_<PLATFORM>` environment variables to handle shared quotas across models.
- **Intelligent cooldown**: HTTP 429/402/403 responses trigger escalating cooldowns (2min → 24hr) stored in `rate_limit_cooldowns`.
- **Dynamic learning**: [`server/src/services/provider-quota.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/provider-quota.ts) parses response headers to adapt to provider-specific limits in real-time.
- **Router integration**: [`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts) sequences limit checks before key selection, enabling automatic fallback when quotas are exhausted.

## Frequently Asked Questions

### How does FreeLLMAPI handle rate limit errors from providers?

When a provider returns HTTP 429, 402, or 403, FreeLLMAPI immediately places the specific key/model combination into a cooldown state using the `setCooldown` mechanism. The duration escalates based on recent throttle history: starting at 2 minutes and increasing to 10 minutes, 1 hour, or 24 hours for repeated violations. This protects both the upstream service and your account status while allowing the router to seamlessly fall back to alternative keys or models.

### Can I configure different daily limits for different providers?

Yes. FreeLLMAPI supports provider-specific daily request caps through environment variables. Set `PROVIDER_DAILY_REQUEST_CAP_<PLATFORM>` (e.g., `PROVIDER_DAILY_REQUEST_CAP_OPENROUTER`) to override defaults. Setting the value to `0` disables the cap entirely for that platform. These settings are read during the `canUseProvider()` check in [`server/src/services/ratelimit.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/ratelimit.ts).

### Where is rate limit data stored and does it persist across restarts?

Rate limit state uses a hybrid storage approach. Active counters live in the in-memory `windows` map for performance, while the SQLite tables `rate_limit_usage` and `rate_limit_cooldowns` provide persistence. This ensures that if the server restarts, the system retains accurate quota consumption data and active cooldown states, preventing accidental quota violations on startup.

### How does the router decide which key to use when multiple are available?

The router in [`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts) evaluates keys sequentially using four checks: `isOnCooldown()`, `canUseProvider()`, `canMakeRequest()`, and `canUseTokens()`. The first key passing all validations is selected. If no keys pass for a given model, the router proceeds to the next model in the fallback chain. Only when all models are exhausted does it return a 429 error to the client.