How FreeLLMAPI Learns Provider Rate Limits: Dynamic Quota Detection

FreeLLMAPI learns provider rate limits by inspecting HTTP response headers and status codes, then maintains a live sliding-window model that penalizes throttled keys during request routing.

Unlike traditional API gateways that rely on static configuration files, FreeLLMAPI treats rate limits as dynamic, observable properties. The system builds a real-time model of each provider key’s quota by intercepting traffic between the router and upstream LLM providers. According to the tashfeenahmed/freellmapi source code, this adaptive approach requires zero manual configuration while automatically steering traffic away from exhausted keys.

The Three-Stage Learning Pipeline

The learning mechanism consists of three tightly coupled components that operate on every request. Each stage transforms raw HTTP responses into structured quota knowledge that influences routing decisions.

Detecting Rate-Limit Signals in HTTP Responses

Every response from a provider passes through parseQuotaObservationsFromResponse in server/src/services/provider-quota.ts. This function inspects HTTP status codes—specifically 429 (Too Many Requests) and 402 (Payment Required)—along with provider-specific headers like x-ratelimit-limit-requests.

The detection logic uses a HEADER_SPECS map to normalize different provider implementations. When a platform like OpenRouter, Groq, or Cerebras returns quota metadata, the system extracts values using parseHeaderNumber and parseResetAtFromHeader to determine when limits reset.

Recording Observations to SQLite

Extracted data is converted into a QuotaObservationInput object and persisted via recordQuotaObservation. The storage layer assigns a confidence score to each observation based on its source:

• Header-based observations receive confidence 1.0 when explicit limit headers are present
• Retry-After headers yield confidence 1.0 (or 0.8 if parsed from error bodies)
• Error-code learning (429/402 without headers) receives confidence 0.55
• Probe observations for shared-pool platforms receive confidence 0.1

This weighted approach allows the scoring engine to trust explicit headers over inferred signals while still maintaining visibility into opaque providers.

Enforcing Limits with Sliding-Window Tracking

The server/src/services/ratelimit.ts module maintains lightweight in-memory sliding windows for each key. It merges persisted usage rows from the SQLite rate_limit_usage table with live counters to compute current requestCount and tokenCount.

The rateLimitFactor function transforms these raw counts into a penalty multiplier between 0 (fully blocked) and 1 (no penalty). When the router evaluates models in server/src/services/scoring.ts, it combines this factor with reliability and speed scores to deprioritize keys approaching their limits.

How the System Learns from Different Provider Signals

FreeLLMAPI employs multiple learning strategies to handle the heterogeneous rate-limiting implementations across the LLM provider ecosystem.

Header-Based Learning with HEADER_SPECS

For platforms that expose structured quota headers, the HEADER_SPECS table maps header names to request or token limits. The maybeAddObservation function parses these values and stores them with maximum confidence. This covers providers like OpenRouter that return rich metadata about remaining requests and reset timestamps.

Retry-After Heuristics

When a response includes a Retry-After header, parseRetryAfterMs (shared with providers/base.ts) converts the value into milliseconds. The system records a rate-limit observation with source: 'header' and uses the timestamp to calculate when the key will become viable again. This prevents the router from selecting keys that are explicitly tellings clients to back off.

Error-Code Learning for 429/402 Responses

Some providers signal quota exhaustion solely through HTTP status codes without helpful headers. In these cases, parseQuotaObservationsFromResponse creates an observation with source: 'error_body' and confidence 0.55. If a Retry-After value is absent, resetAt remains null, and the system relies on the sliding-window counters to estimate recovery time based on historical patterns.

Probe-Based Fallback for Shared Pools

When a provider operates on a shared-pool model (multiple keys drawing from a single quota), the system cannot learn limits from individual key responses. Instead, upon receiving a successful 200 response with no quota headers, FreeLLMAPI injects a low-confidence probe observation (confidence 0.1). This ensures the sliding-window tracker still monitors aggregate usage, preventing a single shared pool from being overrun by concurrent requests.

End-to-End Request Flow

The complete lifecycle of a rate-limit observation follows this path:

1. The router dispatches a request to the upstream provider.
2. The response enters parseQuotaObservationsFromResponse for header analysis.
3. Observations are persisted to SQLite via recordQuotaObservation.
4. Usage counters increment in rate_limit_usage through recordUsage.
5. The scoring module calls getPenalty to retrieve the current penalty and applies rateLimitFactor.
6. The final score calculation in combineScore weighs the rate-limit factor against latency and reliability metrics.
7. The router selects the highest-scoring model, automatically avoiding keys that recently returned 429 errors or exceeded their sliding-window quotas.

Inspecting Rate-Limit Factors in Code

You can observe the learning mechanism in action using the following patterns from the codebase:

// Trigger quota observation parsing manually for debugging
import { parseQuotaObservationsFromResponse } from './server/src/services/provider-quota';

const response = await fetch('https://api.openrouter.ai/v1/chat/completions', {
  method: 'POST',
  headers: { 'Authorization': 'Bearer sk-or-v1-...' },
  body: JSON.stringify({ model: 'gpt-4o-mini', messages: [] })
});

const observations = parseQuotaObservationsFromResponse(response, {
  platform: 'openrouter',
  modelId: 'gpt-4o-mini',
  keyId: 42
});

console.log('Detected quota signals:', observations);

// Check the current rate-limit penalty for a specific key
import { getPenalty } from './server/src/services/router';
import { rateLimitFactor } from './server/src/services/scoring';

const penalty = getPenalty(7); // model database ID
const factor = rateLimitFactor(penalty);

console.log(`Key penalty: ${penalty}, Rate limit factor: ${factor}`);
// factor closer to 0 indicates heavy throttling; 1.0 is fully available

Summary

• Dynamic detection in provider-quota.ts parses HTTP headers and status codes to identify rate limits without hardcoded configuration.
• Confidence-weighted observations distinguish between explicit provider headers (confidence 1.0), error codes (confidence 0.55), and heuristic probes (confidence 0.1).
• Sliding-window tracking in ratelimit.ts maintains per-key request and token counters in SQLite with automatic pruning of stale entries.
• Adaptive scoring converts raw penalties into multipliers that automatically demote throttled keys, ensuring the router selects viable providers.
• Zero-configuration learning allows the system to adapt to new providers immediately by observing their specific rate-limiting behaviors.

Frequently Asked Questions

What happens if a provider returns a 429 status but no Retry-After header?

FreeLLMAPI records an observation with source: 'error_body' and confidence 0.55, then relies on the sliding-window counter to estimate when the key will recover. The system treats the key as rate-limited but does not block it indefinitely; instead, the rateLimitFactor gradually increases as time passes without additional errors.

How does the system handle providers that use shared API key pools?

For shared-pool platforms, FreeLLMAPI injects a "probe" observation with confidence 0.1 whenever a successful 200 response arrives without quota headers. This ensures the sliding-window tracker monitors aggregate usage across the pool, preventing cascade failures when multiple requests exhaust a shared quota simultaneously.

Where does FreeLLMAPI store the learned rate-limit data?

Persistent quota observations and usage counters live in a SQLite database, specifically in the rate_limit_usage table, while the ratelimit.ts service maintains live sliding windows in memory. This hybrid approach provides durability across restarts while enabling microsecond-fast lookups during request routing.

Can developers manually override the learned rate limits?

The source code references the rateLimitFactor function in server/src/services/scoring.ts as the primary interface for incorporating penalties into routing decisions. While the learning pipeline is automatic, the penalty value fed into this function could theoretically be adjusted at the router level, though the codebase emphasizes passive observation over manual configuration.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}