# How FreeLLMAPI Handles Rate Limiting at Provider and Proxy Levels

> FreeLLMAPI uses a two-tier rate-limiting system with SQLite sliding windows to manage provider quotas and penalize overloaded endpoints, ensuring proxy stability.

- Repository: [Tashfeen/freellmapi](https://github.com/tashfeenahmed/freellmapi)
- Tags: how-to-guide
- Published: 2026-06-29

---

**FreeLLMAPI implements a two-tier rate-limiting system that uses SQLite-backed sliding windows to enforce provider quotas while dynamically penalizing rate-limited endpoints to protect internal proxy infrastructure from overload.**

FreeLLMAPI is an open-source LLM aggregation service that routes requests across multiple AI providers. To prevent exhausting platform quotas and shielding its proxy infrastructure from saturation, the system employs a sophisticated **two-tier rate-limiting strategy** that tracks usage at both the provider level and the proxy level.

## Provider-Level Rate Limiting

FreeLLMAPI enforces strict quotas for each upstream LLM provider using a sliding-window algorithm that tracks every request by platform, model, and API key.

### Sliding-Window Tracking in ratelimit.ts

The core implementation resides in [`server/src/services/ratelimit.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/ratelimit.ts), which provides a **SQLite-backed sliding-window tracker**. The system records every request with metadata including timestamp, token count, and request kind using `INSERT INTO rate_limit_usage …` operations. To prevent database bloat, old entries are automatically pruned after 24 hours using `DELETE FROM rate_limit_usage WHERE created_at_ms <= ?`.

This store tracks usage against the **RPM** (requests per minute), **TPM** (tokens per minute), **RPD** (requests per day), and **TPD** (tokens per day) limits defined in the `Model` type in [`shared/types.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/shared/types.ts) via fields like `rpmLimit`, `tpmLimit`, `rpdLimit`, and `tpdLimit`.

### Cooldown and Token Budgeting

Helper functions exported from [`ratelimit.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/ratelimit.ts)—including `canMakeRequest`, `canUseTokens`, `isOnCooldown`, and `canUseProvider`—consult the usage history to determine if a new request would exceed the model's configured limits. These functions check the sliding window before allowing traffic to hit the upstream provider, ensuring compliance with the provider's advertised constraints.

### Dynamic Penalties for Rate Limit Violations

When a provider returns a 429 "Too Many Requests" response, the router in [`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts) registers a penalty for the offending model using `rateLimitPenalties.set(modelDbId, { count: 1, lastHit: now, penalty: PENALTY_PER_429 })`. This penalty is later folded into a **rate-limit factor** (`rateLimitFactor`) calculated in [`server/src/services/scoring.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/scoring.ts), which depresses the model's score in the routing algorithm. The scheduler consequently prefers alternative keys until the cooldown expires.

## Proxy-Level Rate Limiting

In addition to direct provider connections, FreeLLMAPI aggregates certain platforms—such as **AI Horde**, **Routeway**, and **BazaarLink**—through third-party proxy endpoints that impose their own throughput constraints (e.g., approximately 5 RPM observed for Routeway).

### Unified Tracking for Proxy Providers

The system treats these proxies as regular providers within the same [`ratelimit.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/ratelimit.ts) infrastructure. Each proxy key is represented as a `Model` entry (see the `Platform` enum in [`shared/types.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/shared/types.ts) where entries like `'routeway'` and `'aihorde'` are defined). Their limits are encoded in the model definition using the same `rpmLimit`, `tpmLimit`, and related fields, making them subject to identical sliding-window checks and cooldown logic.

### Fallback Handling and Proxy Saturation

When a proxy reports a 429 error, the router applies the same penalty mechanism used for direct providers. The **rate-limit factor** reduces the proxy's weight in the `combineScore` function, and the fallback chain implemented in [`router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/router.ts) automatically drops the rate-limited slot from the active panel. This ensures that a saturated proxy does not block traffic to healthy models.

## Integration in the Routing Pipeline

The rate-limiting system operates at three distinct phases in the request lifecycle:

1. **Pre-filtering**: Upon receiving a request, [`router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/router.ts) calls `canMakeRequest` and `canUseTokens` to exclude models currently over quota before any network traffic is initiated.

2. **Scoring**: Remaining candidate models are scored using the `combineScore` function, which multiplies `rateLimitFactor(penalty)` into the final score alongside other metrics like latency and cost.

3. **Selection and Penalty Recording**: The scheduler selects the highest-scoring model. If the subsequent API call returns a 429, [`router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/router.ts) records the penalty, ensuring the next request cycle deprioritizes that model.

## Code Examples

### Checking Provider Quotas Before Requesting

```typescript
// Example: checking provider quota before issuing a request
import { canMakeRequest, canUseTokens } from './services/ratelimit.js';

async function requestModel(modelId: number, tokens: number) {
  if (!canMakeRequest(modelId)) {
    throw new Error('Model is on cooldown – rate limit exceeded');
  }
  if (!canUseTokens(modelId, tokens)) {
    throw new Error('Token budget exceeded for this model');
  }
  // …perform the actual API call…
}

```

### Handling 429 Responses in the Router

```typescript
// Example: handling a 429 response in the router
import { recordRateLimitPenalty } from './services/router.js';

async function callProvider(provider, payload) {
  try {
    return await provider.invoke(payload);
  } catch (e) {
    if (e.status === 429) {
      recordRateLimitPenalty(provider.modelDbId); // updates the penalty map
    }
    throw e;
  }
}

```

## Summary

- FreeLLMAPI uses a **SQLite-backed sliding-window tracker** in [`server/src/services/ratelimit.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/ratelimit.ts) to enforce RPM, TPM, RPD, and TPD limits at the provider level, with automatic pruning of entries older than 24 hours.
- **Dynamic penalties** applied in [`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts) automatically deprioritize models that return 429 errors, while [`server/src/services/scoring.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/scoring.ts) computes the `rateLimitFactor` that affects routing decisions.
- The system treats third-party proxies like **AI Horde** and **Routeway** as first-class citizens with identical rate-limiting semantics, storing their constraints in [`shared/types.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/shared/types.ts).
- The **routing pipeline** integrates rate-limit checks during pre-filtering, scoring via `combineScore`, and post-error penalty phases to ensure optimal model selection.

## Frequently Asked Questions

### How does FreeLLMAPI store rate limit usage data?

FreeLLMAPI stores rate limit usage in a SQLite database via the [`server/src/services/ratelimit.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/ratelimit.ts) module. The system inserts records with timestamps, token counts, and request metadata into a `rate_limit_usage` table, then prunes entries older than 24 hours using `DELETE FROM rate_limit_usage WHERE created_at_ms <= ?` to maintain performance.

### What happens when a provider returns a 429 error?

When a provider returns a 429 "Too Many Requests" response, the router in [`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts) records a penalty for that specific model using an internal `rateLimitPenalties` map. This penalty feeds into a `rateLimitFactor` calculated in [`server/src/services/scoring.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/scoring.ts) that reduces the model's score in subsequent routing decisions, effectively deprioritizing it until the cooldown period expires.

### Does FreeLLMAPI handle rate limiting differently for proxy aggregators like AI Horde?

No, FreeLLMAPI applies the same rate-limiting architecture to proxy aggregators as it does to direct providers. Platforms like **AI Horde** and **Routeway** are defined in the `Platform` enum in [`shared/types.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/shared/types.ts) and configured with standard RPM/TPM limits. The same sliding-window tracker in [`ratelimit.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/ratelimit.ts) and penalty system in [`router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/router.ts) governs both proxy and direct provider traffic.

### Which functions should I call to check rate limits before making a request?

Before initiating a request, call `canMakeRequest(modelId)` to verify the request won't exceed RPM/RPD limits, and `canUseTokens(modelId, tokenCount)` to ensure the token budget (TPM/TPD) isn't exhausted. Both functions are exported from [`server/src/services/ratelimit.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/ratelimit.ts) and consult the SQLite-backed sliding window to provide accurate quota checks.