How FreeLLMAPI Handles Rate Limiting Across Different Providers

FreeLLMAPI implements a multi-layered rate limiting system that tracks requests and tokens per provider, model, and API key using sliding-window counters, while dynamically observing provider quotas and applying escalating cooldowns to prevent quota exhaustion.

FreeLLMAPI (tashfeenahmed/freellmapi) manages rate limiting across different providers through a sophisticated routing layer that respects contractual quotas while maximizing availability. The system tracks usage at the provider-key-model level, maintains persistent counters in SQLite, and automatically falls back to alternative models when limits are reached.

Multi-Layered Rate Limiting Architecture

The rate limiting system operates across five distinct layers to protect both the service and upstream providers from overconsumption. Each layer targets specific quota types, from per-minute request bursts to daily provider-wide aggregates. The architecture lives primarily in server/src/services/ratelimit.ts and server/src/services/provider-quota.ts.

Per-Minute and Per-Day Request and Token Caps

The core enforcement mechanism tracks Requests Per Minute (RPM), Requests Per Day (RPD), Tokens Per Minute (TPM), and Tokens Per Day (TPD) for every provider-key-model combination.

Sliding-window counters stored in memory (windows map) provide real-time tracking, while the SQLite rate_limit_usage table ensures persistence across restarts. When the router evaluates a key, it calls canMakeRequest() for request quotas and canUseTokens() for token quotas, both defined in server/src/services/ratelimit.ts.

import {
  canMakeRequest,
  canUseTokens,
} from './ratelimit';

const limits = {
  rpm: entry.rpm_limit,
  rpd: entry.rpd_limit,
  tpm: entry.tpm_limit,
  tpd: entry.tpd_limit,
};

if (!canMakeRequest(entry.platform, entry.model_id, key.id, limits)) {
  continue; // Skip to next key
}
if (!canUseTokens(entry.platform, entry.model_id, key.id, estimatedTokens, limits)) {
  continue;
}

Provider-Wide Daily Caps

Some providers, such as OpenRouter's free tier, enforce a single daily request quota shared across all models on an account. FreeLLMAPI handles this through configurable environment variables and the canUseProvider() function.

The system reads caps from PROVIDER_DAILY_REQUEST_CAP_<PLATFORM> variables, falling back to DEFAULT_PROVIDER_DAILY_REQUEST_CAPS. The providerDailyRequestCount aggregates usage across every model for the same key, stored in the rate limit tables.

# Override OpenRouter daily cap

PROVIDER_DAILY_REQUEST_CAP_OPENROUTER=50

# Disable the cap entirely

PROVIDER_DAILY_REQUEST_CAP_OPENROUTER=0

Dynamic Quota Observation from Response Headers

When providers return quota information in HTTP headers (e.g., x-ratelimit-limit-requests), FreeLLMAPI learns and adapts to actual upstream limits. The parseQuotaObservationsFromResponse function in server/src/services/provider-quota.ts extracts limits, timestamps, and retry-after values.

These observations populate provider_quota_state and provider_quota_observations tables, feeding back into the router's decision-making to ensure the system respects provider-side constraints even when they differ from configured defaults.

Cooldown Mechanisms and Error Handling

The system implements exponential backoff through escalating cooldown periods when providers return specific error codes. When a key receives a 429 (rate limit), 402 (payment required), or 403 (model forbidden) response, it enters a cooldown state.

The isOnCooldown() function checks against an in-memory cooldowns map backed by the rate_limit_cooldowns SQLite table. Cooldown durations escalate based on 24-hour throttle frequency: 2 minutes → 10 minutes → 1 hour → 24 hours.

import { getNextCooldownDuration } from './ratelimit';

const nextMs = getNextCooldownDuration(platform, modelId, keyId);
// The router automatically applies cooldowns on 429 errors

Router Integration and Fallback Logic

The request router (server/src/services/router.ts) orchestrates rate limiting checks during key selection. The selectKeyForModel function validates keys sequentially using the rate limiting helpers.

if (isOnCooldown(entry.platform, entry.model_id, kid)) continue;
if (!canUseProvider(entry.platform, kid)) continue;
if (!canMakeRequest(entry.platform, entry.model_id, kid, limits)) continue;
if (!canUseTokens(entry.platform, entry.model_id, kid, estimatedTokens, limits)) continue;

If all keys for a model are exhausted, the router proceeds to the next model in the fallback chain. When the entire chain is depleted, it throws a RouteError with status 429, signaling clients to wait for limit resets.

Recording Usage

After successful requests, the system updates counters to maintain accurate sliding windows. The recordRequest() function increments request counts, while recordTokens() updates token consumption.

import { recordRequest, recordTokens } from './ratelimit';

// After successful provider call
recordRequest(platform, modelId, keyId);

// After receiving response with token count
recordTokens(platform, modelId, keyId, tokens);

Summary

• FreeLLMAPI enforces rate limiting at the provider-key-model granularity using sliding-window counters persisted in SQLite.
• The system tracks both request volume (RPM/RPD) and token consumption (TPM/TPD) through helpers in server/src/services/ratelimit.ts.
• Provider-wide daily caps accommodate shared quotas across models via environment variables and the canUseProvider() function.
• Dynamic quota observation parses provider response headers to learn actual limits in real-time via server/src/services/provider-quota.ts.
• Escalating cooldowns (2 minutes to 24 hours) protect against repeated rate limit errors while the router automatically falls back to alternative models.

Frequently Asked Questions

How does FreeLLMAPI handle different rate limit tiers between providers?

The system normalizes limits through the limits object passed to checking functions, with each provider-key-model combination maintaining independent counters. Provider-specific daily caps are configured via PROVIDER_DAILY_REQUEST_CAP_<PLATFORM> environment variables, allowing custom thresholds for each platform's unique constraints.

What happens when a provider returns a 429 error?

When a provider returns HTTP 429 (rate limit), 402 (payment required), or 403 (model forbidden), FreeLLMAPI immediately places the key on cooldown using an escalating duration pattern. The cooldown starts at 2 minutes and increases to 10 minutes, 1 hour, or 24 hours based on how many times the key has been throttled in the previous 24-hour window.

Can rate limit data survive server restarts?

Yes. While sliding-window counters operate in memory for performance, FreeLLMAPI persists rate limit usage and cooldown states to SQLite tables (rate_limit_usage and rate_limit_cooldowns). This ensures that quota tracking remains accurate across application restarts or deployments.

How does the router decide which model to use when primary keys are rate limited?

The router evaluates keys sequentially using isOnCooldown(), canUseProvider(), canMakeRequest(), and canUseTokens(). If all keys for a model fail these checks, the router skips to the next model in the fallback chain. Only when the entire chain is exhausted does it return a 429 error to the client.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}