How free-claude-code Implements Concurrency Limiting with PROVIDER_MAX_CONCURRENCY

The PROVIDER_MAX_CONCURRENCY environment variable controls simultaneous LLM streams by configuring an asyncio.Semaphore inside the GlobalRateLimiter singleton, which blocks additional requests when the concurrent stream limit is reached.

The free-claude-code repository manages provider load through a centralized concurrency limiting system. By configuring the PROVIDER_MAX_CONCURRENCY setting, developers define exactly how many simultaneous streaming requests can execute across all configured providers. This prevents downstream API rate limit errors and ensures predictable resource usage through a shared semaphore mechanism.

Core Components of the Concurrency System

ProviderConfig Dataclass

In providers/base.py (line 21), the ProviderConfig dataclass stores the max_concurrency value (default 5). This configuration object is instantiated for every provider and passed during initialization, establishing the per-provider concurrency capacity that ultimately feeds into the global limiter.

Settings Integration

The Settings class in config/settings.py (lines 141‑144) binds the PROVIDER_MAX_CONCURRENCY environment variable to the provider_max_concurrency attribute. During application startup, this value is read from the environment and injected into each ProviderConfig, allowing dynamic configuration without code modification.

GlobalRateLimiter Singleton

The GlobalRateLimiter class in providers/rate_limit.py (lines 62‑70) implements the actual concurrency control. It creates an asyncio.Semaphore sized according to the max_concurrency parameter passed during instantiation. Because providers access this limiter through the get_instance() singleton pattern, all provider implementations share the same semaphore, creating a unified global limit rather than isolated per-provider caps.

The concurrency_slot Context Manager

Implemented in providers/rate_limit.py (lines 165‑176), the concurrency_slot async context manager handles semaphore acquisition. When a provider enters this context, it executes await self._concurrency_sem.acquire(), blocking until a permit becomes available. The semaphore releases automatically when the context exits, even if the stream raises an exception.

Execution Flow and Request Handling

The concurrency limiting mechanism operates through five distinct phases:

1. Application Startup: Settings reads PROVIDER_MAX_CONCURRENCY (defaulting to 5) and stores the value in provider_max_concurrency.
2. Provider Initialization: Each provider receives a ProviderConfig instance with max_concurrency set from the global settings.
3. Limiter Instantiation: The first provider calling GlobalRateLimiter.get_instance() creates the singleton with the configured concurrency cap. Subsequent providers reuse this instance, ensuring all share the same semaphore.
4. Stream Acquisition: Inside streaming methods like stream_response, providers wrap their logic in async with self._global_rate_limiter.concurrency_slot():. When the semaphore count reaches zero, subsequent calls pause until another stream completes.
5. Concurrent Execution: At any moment, no more than PROVIDER_MAX_CONCURRENCY streams remain active across all providers, preventing downstream API overload.

Configuration Examples

Setting the Environment Variable

Configure the global concurrency limit before launching the application:

export PROVIDER_MAX_CONCURRENCY=8
uv run your_app.py

This value propagates through Settings.provider_max_concurrency to all providers automatically.

Custom Provider Configuration

Override the default for specific provider instances programmatically:

from config import get_settings
from providers.base import ProviderConfig
from providers.openai_compat import OpenAICompatibleProvider

settings = get_settings()

custom_cfg = ProviderConfig(
    api_key="sk-...",
    base_url="https://api.openai.com/v1",
    rate_limit=settings.provider_rate_limit,
    rate_window=settings.provider_rate_window,
    max_concurrency=3,  # Override: limit this provider to 3 concurrent streams

    http_read_timeout=settings.http_read_timeout,
    http_write_timeout=settings.http_write_timeout,
    http_connect_timeout=settings.http_connect_timeout,
    enable_thinking=settings.enable_thinking,
    proxy="",
)

provider = OpenAICompatibleProvider(
    config=custom_cfg,
    provider_name="openai",
    base_url=custom_cfg.base_url,
    api_key=custom_cfg.api_key,
)

Internal Semaphore Usage

In providers/openai_compat.py (lines 96‑98), the streaming implementation acquires a concurrency slot before creating the remote connection:

async with self._global_rate_limiter.concurrency_slot():
    stream, body = await self._create_stream(body)
    # Stream processing continues...

The underlying concurrency_slot implementation in providers/rate_limit.py uses standard semaphore mechanics:

@asynccontextmanager
async def concurrency_slot(self) -> AsyncIterator[None]:
    """Async context manager that holds one concurrency slot for a stream."""
    await self._concurrency_sem.acquire()
    try:
        yield
    finally:
        self._concurrency_sem.release()

Testing and Validation

The test suite in tests/providers/test_provider_rate_limit.py verifies that invalid max_concurrency values raise ValueError and confirms the semaphore never exceeds the configured limit under concurrent load. You can observe the semaphore state directly:

limiter = GlobalRateLimiter(rate_limit=10, rate_window=60, max_concurrency=2)
assert limiter._concurrency_sem._value == 2  # Initial permits available

Summary

• PROVIDER_MAX_CONCURRENCY sets a process-wide cap on active streams, defaulting to 5.
• ProviderConfig in providers/base.py stores the concurrency limit for each provider instance.
• GlobalRateLimiter in providers/rate_limit.py manages an asyncio.Semaphore as a singleton shared across all providers.
• concurrency_slot blocks when limits are reached, queuing excess requests until capacity frees.
• Provider implementations in providers/openai_compat.py acquire slots before initiating remote connections.

Frequently Asked Questions

What happens when the concurrency limit is reached?

When PROVIDER_MAX_CONCURRENCY streams are active, subsequent requests block at await self._concurrency_sem.acquire() inside the concurrency_slot context manager. The coroutine pauses until another provider releases its slot, creating a natural backpressure mechanism that prevents overwhelming downstream APIs.

Can different providers have different concurrency limits?

While each ProviderConfig can specify its own max_concurrency, the GlobalRateLimiter singleton pattern means all providers typically share the same semaphore instance created from the first provider's configuration. To enforce different limits per provider, you would need to modify the instantiation logic to create separate limiter instances rather than using the shared singleton.

Where is the default concurrency value of 5 defined?

The default value originates in providers/base.py within the ProviderConfig definition (line 21), where max_concurrency defaults to 5. This value is overridden when Settings in config/settings.py injects the PROVIDER_MAX_CONCURRENCY environment variable during provider initialization.

How does the system handle semaphore cleanup if a stream crashes?

The concurrency_slot context manager in providers/rate_limit.py (lines 165‑176) uses a try...finally block to ensure self._concurrency_sem.release() executes regardless of whether the stream completes successfully or raises an exception. This guarantees that crashed or interrupted streams always return their permit to the semaphore pool, preventing resource leaks.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}