How FreeLLMAPI Manages RPM, RPD, TPM, and TPD Rate Limits

FreeLLMAPI enforces per-model and per-provider quotas using a sliding-window counter persisted in SQLite (with an in-memory fallback), applying pre-flight checks and routing guard-rails to prevent limit violations.

FreeLLMAPI is an open-source routing layer that aggregates multiple LLM providers while enforcing strict FreeLLMAPI rate limits to prevent quota exhaustion. The system tracks both request velocity (RPM/RPD) and token consumption (TPM/TPD) across per-model and provider-wide windows, storing usage data in SQLite with graceful degradation to in-memory counters when the database is unavailable.

Per-Model Rate Limiting (RPM, RPD, TPM, TPD)

Each request or token grant is recorded in the rate_limit_usage table, which stores the platform, model ID, key ID, the kind ('request' or 'tokens'), the token count, and a timestamp. The core validation logic lives in server/src/services/ratelimit.ts.

Request Validation Functions

The entry points for rate limit checks are canMakeRequest and canUseTokens (lines 12–17 and surrounding implementation). These functions query sliding-window counters before allowing traffic to upstream providers.

export function canMakeRequest(
  platform: string,
  modelId: string,
  keyId: number,
  limits: { rpm: number | null; rpd: number | null; tpm: number | null; tpd: number | null },
): boolean {
  const now = Date.now();
  const inFlight = provisionalRequests(platform, modelId, keyId, now);
  if (limits.rpm !== null && requestCount(platform, modelId, keyId, MINUTE, now) + inFlight >= limits.rpm) return false;
  if (limits.rpd !== null && requestCount(platform, modelId, keyId, DAY, now) + inFlight >= limits.rpd) return false;
  return true;
}

export function canUseTokens(
  platform: string,
  modelId: string,
  keyId: number,
  estimatedTokens: number,
  limits: { tpm: number | null; tpd: number | null },
): boolean {
  const now = Date.now();
  const inFlight = provisionalTokens(platform, modelId, keyId, now);
  if (limits.tpm !== null) {
    const used = tokenCount(platform, modelId, keyId, MINUTE, now);
    if (used + inFlight + estimatedTokens > limits.tpm) return false;
  }
  if (limits.tpd !== null) {
    const used = tokenCount(platform, modelId, keyId, DAY, now);
    if (used + inFlight + estimatedTokens > limits.tpd) return false;
  }
  return true;
}

Sliding-Window Implementation

The window definitions use MINUTE = 60_000ms for per-minute limits and msSinceUtcMidnight for per-day limits to align with UTC boundaries. The requestCount and tokenCount helpers first attempt to read from persisted SQLite counters (countPersistedRequests / sumPersistedTokens) and transparently fall back to in-memory sliding windows (memoryRequestCount / memoryTokenCount) if the database is unavailable.

Provider-Wide Quota Caps

Some providers (such as OpenRouter’s free tier) expose a single quota shared across all models. FreeLLMAPI enforces these global caps separately from per-model limits.

Environment-Based Configuration

Default caps are defined in server/src/services/ratelimit.ts (lines 61–78) and can be overridden via environment variables:

• PROVIDER_DAILY_REQUEST_CAP_<PLATFORM>
• PROVIDER_MINUTE_REQUEST_CAP_<PLATFORM>
• PROVIDER_DAILY_TOKEN_CAP_<PLATFORM>

Aggregate Enforcement Functions

The functions getProviderDailyRequestCap, getProviderMinuteRequestCap, and getProviderDailyTokenCap return the configured limits. These feed into canUseProvider, canUseProviderMinute, and canUseProviderTokens, which aggregate usage across all models belonging to the same account/key rather than checking a single model in isolation.

Concurrency Guardrails

FreeLLMAPI also limits the number of in-flight requests per key, which protects providers that meter concurrency rather than per-minute throughput. The limit is read from MAX_CONCURRENT_REQUESTS_PER_KEY_<PLATFORM> (with a generic fallback) in server/src/services/ratelimit.ts (lines 98–106) and enforced via canUseKeyConcurrency.

Routing Guardrails and Fractional Utilization

To avoid abrupt cutoffs, the scoring engine uses fractional utilization to gently demote keys as they approach limits. This is implemented in server/src/services/ratelimit.ts (lines 73–84) via buildWindowUsageSnapshot, which performs a single grouped query over the last minute and day for all keys, then caches the result for a 5-second TTL.

The modelWindowUsedFraction function calculates pressure by comparing current usage against limits. This value feeds into rateWindowHeadroomFactor in server/src/services/scoring.ts (lines 94–104), which applies a smooth penalty to a model’s score as its quota consumes, preserving throughput for healthier alternatives rather than blocking traffic only after the limit is hit.

Cooldown Escalation After HTTP 429

When a provider returns a 429 status code, FreeLLMAPI places the model+key combination on a cooldown. The logic in getCooldownDecisionForLimit (server/src/services/ratelimit.ts, lines 66–78) consults a ladder (COOLDOWN_DURATIONS) that escalates up to 24 hours when the daily quota appears exhausted, or falls back to a 90-second bench for transient per-minute failures. This prevents hammering providers that have already signaled exhaustion.

Summary

• Dual-layer persistence: Rate limit counters are stored in SQLite with an automatic in-memory fallback for high availability.
• Per-model and provider-wide enforcement: Separate tracks validate limits for individual models and aggregate quotas across provider accounts.
• Sliding-window logic: RPM/TPM use 60-second windows; RPD/TPD align to UTC midnight boundaries.
• Pre-flight validation: canMakeRequest and canUseTokens reject requests before they reach upstream providers, conserving quota.
• Graceful degradation: Routing guardrails use fractional utilization (modelWindowUsedFraction) and headroom factors to smoothly steer traffic away from saturated keys.
• Exponential backoff: 429 responses trigger escalating cooldowns up to 24 hours to handle daily limit exhaustion.

Frequently Asked Questions

How does FreeLLMAPI handle database failures during rate limit checks?

When the SQLite database is unavailable, the requestCount and tokenCount functions automatically fall back to in-memory sliding-window counters (memoryRequestCount and memoryTokenCount). This ensures that rate limiting continues to function even during database outages, although counts will reset if the server process restarts.

What is the difference between per-model and provider-wide rate limits in FreeLLMAPI?

Per-model limits (enforced by canMakeRequest and canUseTokens) track usage for a specific model ID and key combination, while provider-wide caps (enforced by canUseProvider and related functions) aggregate usage across all models belonging to the same provider account. Provider limits are configured via environment variables like PROVIDER_DAILY_REQUEST_CAP_<PLATFORM> and protect shared quotas such as OpenRouter’s free tier.

How does the routing guardrail prevent quota exhaustion before limits are hit?

The router calls modelWindowUsedFraction to calculate a utilization percentage between 0 and 1. This feeds into rateWindowHeadroomFactor in server/src/services/scoring.ts, which applies a diminishing multiplier to the model’s selection score as utilization increases. By the time a key reaches its RPM or TPM limit, its score has degraded sufficiently that the router prefers alternative keys, preventing hard rejections.

What happens when a provider returns a 429 error?

The system invokes getCooldownDecisionForLimit to determine a backoff duration. Based on the COOLDOWN_DURATIONS ladder, the model+key is benched for 90 seconds if the error appears transient, or up to 24 hours if the daily quota is exhausted. Subsequent routing decisions consult this cooldown state via setCooldown and reject traffic to that key until the period expires.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}