How FreeLLMAPI Handles Rate Limiting at Provider and Proxy Levels

FreeLLMAPI implements a two-tier rate-limiting system that uses SQLite-backed sliding windows to enforce provider quotas while dynamically penalizing rate-limited endpoints to protect internal proxy infrastructure from overload.

FreeLLMAPI is an open-source LLM aggregation service that routes requests across multiple AI providers. To prevent exhausting platform quotas and shielding its proxy infrastructure from saturation, the system employs a sophisticated two-tier rate-limiting strategy that tracks usage at both the provider level and the proxy level.

Provider-Level Rate Limiting

FreeLLMAPI enforces strict quotas for each upstream LLM provider using a sliding-window algorithm that tracks every request by platform, model, and API key.

Sliding-Window Tracking in ratelimit.ts

The core implementation resides in server/src/services/ratelimit.ts, which provides a SQLite-backed sliding-window tracker. The system records every request with metadata including timestamp, token count, and request kind using INSERT INTO rate_limit_usage … operations. To prevent database bloat, old entries are automatically pruned after 24 hours using DELETE FROM rate_limit_usage WHERE created_at_ms <= ?.

This store tracks usage against the RPM (requests per minute), TPM (tokens per minute), RPD (requests per day), and TPD (tokens per day) limits defined in the Model type in shared/types.ts via fields like rpmLimit, tpmLimit, rpdLimit, and tpdLimit.

Cooldown and Token Budgeting

Helper functions exported from ratelimit.ts—including canMakeRequest, canUseTokens, isOnCooldown, and canUseProvider—consult the usage history to determine if a new request would exceed the model's configured limits. These functions check the sliding window before allowing traffic to hit the upstream provider, ensuring compliance with the provider's advertised constraints.

Dynamic Penalties for Rate Limit Violations

When a provider returns a 429 "Too Many Requests" response, the router in server/src/services/router.ts registers a penalty for the offending model using rateLimitPenalties.set(modelDbId, { count: 1, lastHit: now, penalty: PENALTY_PER_429 }). This penalty is later folded into a rate-limit factor (rateLimitFactor) calculated in server/src/services/scoring.ts, which depresses the model's score in the routing algorithm. The scheduler consequently prefers alternative keys until the cooldown expires.

Proxy-Level Rate Limiting

In addition to direct provider connections, FreeLLMAPI aggregates certain platforms—such as AI Horde, Routeway, and BazaarLink—through third-party proxy endpoints that impose their own throughput constraints (e.g., approximately 5 RPM observed for Routeway).

Unified Tracking for Proxy Providers

The system treats these proxies as regular providers within the same ratelimit.ts infrastructure. Each proxy key is represented as a Model entry (see the Platform enum in shared/types.ts where entries like 'routeway' and 'aihorde' are defined). Their limits are encoded in the model definition using the same rpmLimit, tpmLimit, and related fields, making them subject to identical sliding-window checks and cooldown logic.

Fallback Handling and Proxy Saturation

When a proxy reports a 429 error, the router applies the same penalty mechanism used for direct providers. The rate-limit factor reduces the proxy's weight in the combineScore function, and the fallback chain implemented in router.ts automatically drops the rate-limited slot from the active panel. This ensures that a saturated proxy does not block traffic to healthy models.

Integration in the Routing Pipeline

The rate-limiting system operates at three distinct phases in the request lifecycle:

1. Pre-filtering: Upon receiving a request, router.ts calls canMakeRequest and canUseTokens to exclude models currently over quota before any network traffic is initiated.

2. Scoring: Remaining candidate models are scored using the combineScore function, which multiplies rateLimitFactor(penalty) into the final score alongside other metrics like latency and cost.

3. Selection and Penalty Recording: The scheduler selects the highest-scoring model. If the subsequent API call returns a 429, router.ts records the penalty, ensuring the next request cycle deprioritizes that model.

Code Examples

Checking Provider Quotas Before Requesting

// Example: checking provider quota before issuing a request
import { canMakeRequest, canUseTokens } from './services/ratelimit.js';

async function requestModel(modelId: number, tokens: number) {
  if (!canMakeRequest(modelId)) {
    throw new Error('Model is on cooldown – rate limit exceeded');
  }
  if (!canUseTokens(modelId, tokens)) {
    throw new Error('Token budget exceeded for this model');
  }
  // …perform the actual API call…
}

Handling 429 Responses in the Router

// Example: handling a 429 response in the router
import { recordRateLimitPenalty } from './services/router.js';

async function callProvider(provider, payload) {
  try {
    return await provider.invoke(payload);
  } catch (e) {
    if (e.status === 429) {
      recordRateLimitPenalty(provider.modelDbId); // updates the penalty map
    }
    throw e;
  }
}

Summary

• FreeLLMAPI uses a SQLite-backed sliding-window tracker in server/src/services/ratelimit.ts to enforce RPM, TPM, RPD, and TPD limits at the provider level, with automatic pruning of entries older than 24 hours.
• Dynamic penalties applied in server/src/services/router.ts automatically deprioritize models that return 429 errors, while server/src/services/scoring.ts computes the rateLimitFactor that affects routing decisions.
• The system treats third-party proxies like AI Horde and Routeway as first-class citizens with identical rate-limiting semantics, storing their constraints in shared/types.ts.
• The routing pipeline integrates rate-limit checks during pre-filtering, scoring via combineScore, and post-error penalty phases to ensure optimal model selection.

Frequently Asked Questions

How does FreeLLMAPI store rate limit usage data?

FreeLLMAPI stores rate limit usage in a SQLite database via the server/src/services/ratelimit.ts module. The system inserts records with timestamps, token counts, and request metadata into a rate_limit_usage table, then prunes entries older than 24 hours using DELETE FROM rate_limit_usage WHERE created_at_ms <= ? to maintain performance.

What happens when a provider returns a 429 error?

When a provider returns a 429 "Too Many Requests" response, the router in server/src/services/router.ts records a penalty for that specific model using an internal rateLimitPenalties map. This penalty feeds into a rateLimitFactor calculated in server/src/services/scoring.ts that reduces the model's score in subsequent routing decisions, effectively deprioritizing it until the cooldown period expires.

Does FreeLLMAPI handle rate limiting differently for proxy aggregators like AI Horde?

No, FreeLLMAPI applies the same rate-limiting architecture to proxy aggregators as it does to direct providers. Platforms like AI Horde and Routeway are defined in the Platform enum in shared/types.ts and configured with standard RPM/TPM limits. The same sliding-window tracker in ratelimit.ts and penalty system in router.ts governs both proxy and direct provider traffic.

Which functions should I call to check rate limits before making a request?

Before initiating a request, call canMakeRequest(modelId) to verify the request won't exceed RPM/RPD limits, and canUseTokens(modelId, tokenCount) to ensure the token budget (TPM/TPD) isn't exhausted. Both functions are exported from server/src/services/ratelimit.ts and consult the SQLite-backed sliding window to provide accurate quota checks.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}