Rate Limiting System in Free-Claude-Code: Implementing Strict Rolling Windows

Free-Claude-Code implements a strict rolling-window rate limiting system through a singleton GlobalRateLimiter class that combines reactive blocking, proactive throttling using a monotonic timestamp deque, and concurrency caps to prevent API quota violations.

The Alishahryar1/free-claude-code repository enforces polite API consumption through a sophisticated throttling mechanism defined in providers/rate_limit.py. At its core sits the GlobalRateLimiter, a singleton that coordinates request pacing across all LLM providers using a strict rolling-window algorithm rather than fixed-interval buckets.

Core Architecture of the Global Rate Limiter

The Singleton Pattern for Global Coordination

The rate limiting system relies on GlobalRateLimiter.get_instance() to lazily construct a single shared instance on first invocation. This singleton design enables global coordination across all provider modules, ensuring that disparate API calls respect a unified quota regardless of which specific provider (OpenAI, LMStudio, NVIDIA NIM) initiates the request.

Three-Layer Defense Strategy

The implementation combines three complementary mechanisms to enforce quota compliance:

• Reactive Blocking: Uses self._blocked_until and wait_if_blocked() to pause execution when a provider returns a 429 error. The set_blocked(delay) method records a future timestamp, and subsequent calls to wait_if_blocked() sleep until that time passes.
• Proactive Throttling: Implements a strict rolling window using _request_times: deque[float] and _acquire_proactive_slot(). This guarantees no more than rate_limit requests occur within any sliding interval of length rate_window.
• Concurrency Caps: An asyncio.Semaphore managed through concurrency_slot() limits simultaneous open connections (e.g., SSE streams), preventing resource oversubscription.

How the Rolling Window Algorithm Works

The Timestamp Deque

Inside providers/rate_limit.py, the limiter maintains _request_times, a collections.deque storing monotonic timestamps of successful requests. Before granting a new request slot, the algorithm trims this deque to remove entries older than now - self._rate_window.

Slot Acquisition Logic

The _acquire_proactive_slot() method executes the core rolling-window logic under an asyncio.Lock:

1. Window Pruning: Removes timestamps falling outside the current window.
2. Limit Check: Grants the slot immediately if fewer than self._rate_limit timestamps remain.
3. Calculated Delay: If the limit is reached, computes the exact sleep time required for the oldest timestamp to expire: (oldest + self._rate_window) - now.
4. Non-Blocking Sleep: Releases the lock before sleeping, allowing other coroutines to queue.

This approach eliminates burst behavior at window boundaries, ensuring strict quota enforcement where any arbitrary interval of length rate_window contains at most rate_limit requests.

Reactive Blocking and Retry Handling

Handling 429 Errors with set_blocked

When a provider returns a RateLimitError, the set_blocked(delay) method records a future timestamp in self._blocked_until. The wait_if_blocked() coroutine checks this timestamp before proceeding, implementing an immediate circuit-breaker pattern that prevents retry storms.

Exponential Back-off with execute_with_retry

The execute_with_retry() helper combines proactive throttling with reactive recovery:

async def execute_with_retry(self, func, max_retries=3):
    for attempt in range(max_retries):
        await self.wait_if_blocked()  # Check reactive + proactive limits

        try:
            return await func()
        except RateLimitError:
            delay = calculate_backoff(attempt)
            self.set_blocked(delay)  # Trigger reactive block

            await asyncio.sleep(delay)

This wrapper ensures that 429 responses trigger both the rolling-window adjustment and exponential back-off before subsequent attempts.

Concurrency Control with Async Semaphores

Independent of the sliding window, the concurrency_slot() async context manager acquires self._concurrency_sem, an asyncio.Semaphore initialized with max_concurrency. This limits simultaneously open provider streams—critical for SSE connections or batch operations—ensuring that rate limits are not circumvented through parallel request flooding.

Practical Implementation Examples

Basic Provider Integration

from providers.rate_limit import GlobalRateLimiter

limiter = GlobalRateLimiter.get_instance(rate_limit=40, rate_window=60)

async def call_api(payload):
    # Wait for both reactive blocks and proactive slots

    await limiter.wait_if_blocked()
    response = await http_client.post("/v1/chat", json=payload)
    return response

Automatic Retry Wrapping

async def chat(messages):
    async def do_request():
        return await http_client.post("/v1/chat", json={"messages": messages})

    limiter = GlobalRateLimiter.get_instance()
    # Retries 3 times on 429 with back-off

    return await limiter.execute_with_retry(do_request, max_retries=3)

Limiting Concurrent SSE Streams

limiter = GlobalRateLimiter.get_instance(max_concurrency=2)

async def stream_events():
    async with limiter.concurrency_slot():
        async for event in sse_client.iter_events():
            process(event)  # Only 2 streams run in parallel

Validating Rolling-Window Guarantees

import asyncio
import time
from providers.rate_limit import GlobalRateLimiter

async def demo():
    GlobalRateLimiter.reset_instance()
    lim = GlobalRateLimiter.get_instance(rate_limit=2, rate_window=0.5)
    
    timestamps = []
    
    async def acquire():
        await lim.wait_if_blocked()
        timestamps.append(time.monotonic())
    
    await asyncio.gather(*(acquire() for _ in range(5)))
    timestamps.sort()
    print(timestamps)  # Observe ≥0.5s spacing between every 2nd request

Summary

• GlobalRateLimiter in providers/rate_limit.py acts as a singleton coordinator for all API requests.
• Strict rolling windows use a deque of monotonic timestamps to guarantee no more than rate_limit requests occur within any rate_window interval.
• _acquire_proactive_slot() calculates precise sleep times based on the oldest timestamp in the window, preventing burst behavior.
• Reactive blocking via set_blocked() and wait_if_blocked() handles 429 errors with immediate back-off.
• Concurrency limits are enforced independently through an asyncio.Semaphore via concurrency_slot().
• Comprehensive tests in tests/providers/test_provider_rate_limit.py validate singleton semantics, zero-value guards, and strict window enforcement.

Frequently Asked Questions

What is a strict rolling window in rate limiting?

A strict rolling window (or sliding window) tracks the exact timestamps of recent requests and enforces that no more than rate_limit requests have occurred within the past rate_window seconds. Unlike fixed windows that reset at interval boundaries, this approach prevents burst attacks at window edges and guarantees quota compliance for any arbitrary time slice of the specified duration.

How does GlobalRateLimiter handle 429 rate limit errors?

When a provider returns a 429 status, the code calls set_blocked(delay), which sets self._blocked_until to a future timestamp. The wait_if_blocked() method checks this timestamp before each request and pauses execution until the block expires. This reactive mechanism works alongside the proactive rolling window to prevent retry loops while the API quota recovers.

Why use a deque instead of a fixed bucket for the rolling window?

The _request_times deque stores actual request timestamps, allowing the algorithm to calculate the exact moment the oldest request falls outside the window. Fixed-token buckets allow bursts at refill time, whereas the deque-based approach enforces strict pacing by requiring the oldest timestamp to age out before granting new slots, matching the semantics of commercial APIs like OpenAI's rate limits.

Can the concurrency limit be adjusted independently of the rate window?

Yes. The max_concurrency parameter controls the asyncio.Semaphore initialized separately from rate_limit and rate_window. You can configure GlobalRateLimiter.get_instance(rate_limit=20, rate_window=60, max_concurrency=5) to allow five simultaneous connections while still enforcing only 20 requests per minute, preventing resource exhaustion without affecting the rolling-window throttle.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}