How FreeLLMAPI Handles Rate Limiting for LLM Providers: A Technical Deep Dive

FreeLLMAPI enforces rate limits at multiple layers—per key, per model, and per provider—using sliding-window counters, provider-wide daily caps, and exponential backoff cooldowns to ensure reliable request routing across diverse LLM platforms.

FreeLLMAPI is an open-source router that aggregates multiple LLM providers behind a single API. To prevent quota exhaustion and ensure fair usage across different upstream services, the system implements a sophisticated rate limiting strategy. This article examines how the codebase in tashfeenahmed/freellmapi manages request throttling, token consumption, and automatic failover when limits are reached.

The Multi-Layer Rate Limiting Architecture

The rate limiting system operates across five distinct layers, each targeting different quota dimensions from individual requests to provider-wide daily aggregates.

Per-Minute and Per-Day Request Caps (RPM/RPD)

At the most granular level, FreeLLMAPI tracks requests per minute (RPM) and requests per day (RPD) for every unique provider → key → model combination.

The system maintains sliding-window counters in an in-memory windows map, backed by persistent storage in the SQLite rate_limit_usage table. This dual approach ensures that quota state survives server restarts while maintaining high-performance lookups during request routing.

According to the source code in server/src/services/ratelimit.ts, the router checks these limits via the canMakeRequest() function before selecting a key for any model.

Token-Based Limits (TPM/TPD)

Beyond request counts, FreeLLMAPI enforces tokens per minute (TPM) and tokens per day (TPD) limits. These constraints account for the total token consumption (prompt + completion) rather than just the number of API calls.

The tracking mechanism mirrors the request-based approach—using the same sliding-window infrastructure—but increments counters based on token counts instead of timestamps. The canUseTokens() function in server/src/services/ratelimit.ts validates these limits before routing, accepting an estimatedTokens parameter to preemptively check capacity.

Provider-Wide Daily Caps

Some providers, notably OpenRouter's free tier, enforce a single daily quota shared across all models on an account. FreeLLMAPI handles this through the canUseProvider() check.

These caps are configured via environment variables following the pattern PROVIDER_DAILY_REQUEST_CAP_<PLATFORM>, with fallback defaults defined in DEFAULT_PROVIDER_DAILY_REQUEST_CAPS. The system aggregates usage across every model for the same key using the providerDailyRequestCount tracker, ensuring that switching models within the same provider does not circumvent daily limits.

# Override OpenRouter's daily cap in your .env file

PROVIDER_DAILY_REQUEST_CAP_OPENROUTER=50

# Set to 0 to disable the cap entirely

PROVIDER_DAILY_REQUEST_CAP_OPENROUTER=0

Cooldown and Error Handling

When a provider returns specific HTTP status codes—429 (Rate Limit), 402 (Payment Required), or 403 (Model Forbidden)—FreeLLMAPI places the key/model combination into a cooldown state.

The cooldowns map stores expiry timestamps in memory, with persistent backup in the rate_limit_cooldowns table. The duration follows an escalating backoff strategy: 2 minutes → 10 minutes → 1 hour → 24 hours, based on how many times the key has been throttled in the last 24 hours.

The isOnCooldown() function checks this state during key selection, and getNextCooldownDuration() calculates the appropriate penalty period.

Dynamic Quota Observation

FreeLLMAPI adapts to provider-specific limit changes by parsing response headers. The parseQuotaObservationsFromResponse() function in server/src/services/provider-quota.ts extracts official limits, timestamps, and retry-after values from headers like x-ratelimit-limit-requests.

These observations populate the provider_quota_state and provider_quota_observations tables, creating a feedback loop where the router learns and respects the provider's current quota configuration rather than relying solely on static configuration.

How the Router Enforces Limits

The request routing logic in server/src/services/router.ts implements a sequential validation pipeline when selecting keys for a model. The router iterates through available keys and applies the following checks in order:

if (isOnCooldown(entry.platform, entry.model_id, kid)) continue;
if (!canUseProvider(entry.platform, kid)) continue;                 // Provider-wide daily cap
if (!canMakeRequest(entry.platform, entry.model_id, kid, limits)) continue; // RPM/RPD
if (!canUseTokens(entry.platform, entry.model_id, kid, estimatedTokens, limits)) continue; // TPM/TPD

If any check fails, the router skips that key and attempts the next available one. When all keys for a specific model are exhausted, the system falls back to the next model in the chain. Only when the entire fallback chain is depleted does FreeLLMAPI throw a RouteError (HTTP 429), signaling that the client must wait for quota reset.

Implementation Examples

Checking Rate Limit Status Before Routing

When implementing custom routing logic or extending the selector, use the public helper functions from server/src/services/ratelimit.ts:

import {
  canMakeRequest,
  canUseTokens,
  canUseProvider,
  isOnCooldown,
} from './ratelimit';

// Limits are derived from the model configuration
const limits = {
  rpm: entry.rpm_limit,
  rpd: entry.rpd_limit,
  tpm: entry.tpm_limit,
  tpd: entry.tpd_limit,
};

// Comprehensive check before using a key
if (
  isOnCooldown(entry.platform, entry.model_id, key.id) ||
  !canUseProvider(entry.platform, key.id) ||
  !canMakeRequest(entry.platform, entry.model_id, key.id, limits) ||
  !canUseTokens(entry.platform, entry.model_id, key.id, estimatedTokens, limits)
) {
  continue; // Skip to next key
}

Recording Usage After Successful Requests

After a successful API call, update the counters to maintain accurate sliding-window state:

import { recordRequest, recordTokens } from './ratelimit';

// Increment request counter
recordRequest(platform, modelId, keyId);

// Update token consumption (prompt + completion)
recordTokens(platform, modelId, keyId, tokens);

Calculating Cooldown Duration

To determine how long a key should be benched after hitting a rate limit:

import { getNextCooldownDuration } from './ratelimit';

const nextCooldownMs = getNextCooldownDuration(platform, modelId, keyId);
// Returns: 120000 (2min), 600000 (10min), 3600000 (1hr), or 86400000 (24hr)

Summary

• Multi-layer protection: FreeLLMAPI enforces limits at the request, token, and provider-wide levels using server/src/services/ratelimit.ts.
• Sliding-window tracking: RPM/RPD and TPM/TPD counters use in-memory windows maps with SQLite persistence via rate_limit_usage.
• Provider-wide daily caps: Configurable via PROVIDER_DAILY_REQUEST_CAP_<PLATFORM> environment variables to handle shared quotas across models.
• Intelligent cooldown: HTTP 429/402/403 responses trigger escalating cooldowns (2min → 24hr) stored in rate_limit_cooldowns.
• Dynamic learning: server/src/services/provider-quota.ts parses response headers to adapt to provider-specific limits in real-time.
• Router integration: server/src/services/router.ts sequences limit checks before key selection, enabling automatic fallback when quotas are exhausted.

Frequently Asked Questions

How does FreeLLMAPI handle rate limit errors from providers?

When a provider returns HTTP 429, 402, or 403, FreeLLMAPI immediately places the specific key/model combination into a cooldown state using the setCooldown mechanism. The duration escalates based on recent throttle history: starting at 2 minutes and increasing to 10 minutes, 1 hour, or 24 hours for repeated violations. This protects both the upstream service and your account status while allowing the router to seamlessly fall back to alternative keys or models.

Can I configure different daily limits for different providers?

Yes. FreeLLMAPI supports provider-specific daily request caps through environment variables. Set PROVIDER_DAILY_REQUEST_CAP_<PLATFORM> (e.g., PROVIDER_DAILY_REQUEST_CAP_OPENROUTER) to override defaults. Setting the value to 0 disables the cap entirely for that platform. These settings are read during the canUseProvider() check in server/src/services/ratelimit.ts.

Where is rate limit data stored and does it persist across restarts?

Rate limit state uses a hybrid storage approach. Active counters live in the in-memory windows map for performance, while the SQLite tables rate_limit_usage and rate_limit_cooldowns provide persistence. This ensures that if the server restarts, the system retains accurate quota consumption data and active cooldown states, preventing accidental quota violations on startup.

How does the router decide which key to use when multiple are available?

The router in server/src/services/router.ts evaluates keys sequentially using four checks: isOnCooldown(), canUseProvider(), canMakeRequest(), and canUseTokens(). The first key passing all validations is selected. If no keys pass for a given model, the router proceeds to the next model in the fallback chain. Only when all models are exhausted does it return a 429 error to the client.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}