# How free-claude-code Implements Concurrency Limiting with PROVIDER_MAX_CONCURRENCY

> Leverage PROVIDER_MAX_CONCURRENCY in free-claude-code to limit concurrent LLM streams. Discover how asyncio Semaphore manages request blocking for efficient concurrency control.

- Repository: [Ali Khokhar/free-claude-code](https://github.com/Alishahryar1/free-claude-code)
- Tags: internals
- Published: 2026-04-24

---

**The `PROVIDER_MAX_CONCURRENCY` environment variable controls simultaneous LLM streams by configuring an `asyncio.Semaphore` inside the `GlobalRateLimiter` singleton, which blocks additional requests when the concurrent stream limit is reached.**

The free-claude-code repository manages provider load through a centralized concurrency limiting system. By configuring the `PROVIDER_MAX_CONCURRENCY` setting, developers define exactly how many simultaneous streaming requests can execute across all configured providers. This prevents downstream API rate limit errors and ensures predictable resource usage through a shared semaphore mechanism.

## Core Components of the Concurrency System

### ProviderConfig Dataclass

In [`providers/base.py`](https://github.com/Alishahryar1/free-claude-code/blob/main/providers/base.py) (line 21), the `ProviderConfig` dataclass stores the `max_concurrency` value (default 5). This configuration object is instantiated for every provider and passed during initialization, establishing the per-provider concurrency capacity that ultimately feeds into the global limiter.

### Settings Integration

The `Settings` class in [`config/settings.py`](https://github.com/Alishahryar1/free-claude-code/blob/main/config/settings.py) (lines 141‑144) binds the `PROVIDER_MAX_CONCURRENCY` environment variable to the `provider_max_concurrency` attribute. During application startup, this value is read from the environment and injected into each `ProviderConfig`, allowing dynamic configuration without code modification.

### GlobalRateLimiter Singleton

The `GlobalRateLimiter` class in [`providers/rate_limit.py`](https://github.com/Alishahryar1/free-claude-code/blob/main/providers/rate_limit.py) (lines 62‑70) implements the actual concurrency control. It creates an `asyncio.Semaphore` sized according to the `max_concurrency` parameter passed during instantiation. Because providers access this limiter through the `get_instance()` singleton pattern, all provider implementations share the same semaphore, creating a unified global limit rather than isolated per-provider caps.

### The concurrency_slot Context Manager

Implemented in [`providers/rate_limit.py`](https://github.com/Alishahryar1/free-claude-code/blob/main/providers/rate_limit.py) (lines 165‑176), the `concurrency_slot` async context manager handles semaphore acquisition. When a provider enters this context, it executes `await self._concurrency_sem.acquire()`, blocking until a permit becomes available. The semaphore releases automatically when the context exits, even if the stream raises an exception.

## Execution Flow and Request Handling

The concurrency limiting mechanism operates through five distinct phases:

1. **Application Startup**: `Settings` reads `PROVIDER_MAX_CONCURRENCY` (defaulting to 5) and stores the value in `provider_max_concurrency`.
2. **Provider Initialization**: Each provider receives a `ProviderConfig` instance with `max_concurrency` set from the global settings.
3. **Limiter Instantiation**: The first provider calling `GlobalRateLimiter.get_instance()` creates the singleton with the configured concurrency cap. Subsequent providers reuse this instance, ensuring all share the same semaphore.
4. **Stream Acquisition**: Inside streaming methods like `stream_response`, providers wrap their logic in `async with self._global_rate_limiter.concurrency_slot():`. When the semaphore count reaches zero, subsequent calls pause until another stream completes.
5. **Concurrent Execution**: At any moment, no more than `PROVIDER_MAX_CONCURRENCY` streams remain active across all providers, preventing downstream API overload.

## Configuration Examples

### Setting the Environment Variable

Configure the global concurrency limit before launching the application:

```bash
export PROVIDER_MAX_CONCURRENCY=8
uv run your_app.py

```

This value propagates through `Settings.provider_max_concurrency` to all providers automatically.

### Custom Provider Configuration

Override the default for specific provider instances programmatically:

```python
from config import get_settings
from providers.base import ProviderConfig
from providers.openai_compat import OpenAICompatibleProvider

settings = get_settings()

custom_cfg = ProviderConfig(
    api_key="sk-...",
    base_url="https://api.openai.com/v1",
    rate_limit=settings.provider_rate_limit,
    rate_window=settings.provider_rate_window,
    max_concurrency=3,  # Override: limit this provider to 3 concurrent streams

    http_read_timeout=settings.http_read_timeout,
    http_write_timeout=settings.http_write_timeout,
    http_connect_timeout=settings.http_connect_timeout,
    enable_thinking=settings.enable_thinking,
    proxy="",
)

provider = OpenAICompatibleProvider(
    config=custom_cfg,
    provider_name="openai",
    base_url=custom_cfg.base_url,
    api_key=custom_cfg.api_key,
)

```

### Internal Semaphore Usage

In [`providers/openai_compat.py`](https://github.com/Alishahryar1/free-claude-code/blob/main/providers/openai_compat.py) (lines 96‑98), the streaming implementation acquires a concurrency slot before creating the remote connection:

```python
async with self._global_rate_limiter.concurrency_slot():
    stream, body = await self._create_stream(body)
    # Stream processing continues...

```

The underlying `concurrency_slot` implementation in [`providers/rate_limit.py`](https://github.com/Alishahryar1/free-claude-code/blob/main/providers/rate_limit.py) uses standard semaphore mechanics:

```python
@asynccontextmanager
async def concurrency_slot(self) -> AsyncIterator[None]:
    """Async context manager that holds one concurrency slot for a stream."""
    await self._concurrency_sem.acquire()
    try:
        yield
    finally:
        self._concurrency_sem.release()

```

## Testing and Validation

The test suite in [`tests/providers/test_provider_rate_limit.py`](https://github.com/Alishahryar1/free-claude-code/blob/main/tests/providers/test_provider_rate_limit.py) verifies that invalid `max_concurrency` values raise `ValueError` and confirms the semaphore never exceeds the configured limit under concurrent load. You can observe the semaphore state directly:

```python
limiter = GlobalRateLimiter(rate_limit=10, rate_window=60, max_concurrency=2)
assert limiter._concurrency_sem._value == 2  # Initial permits available

```

## Summary

- **`PROVIDER_MAX_CONCURRENCY`** sets a process-wide cap on active streams, defaulting to 5.
- **`ProviderConfig`** in [`providers/base.py`](https://github.com/Alishahryar1/free-claude-code/blob/main/providers/base.py) stores the concurrency limit for each provider instance.
- **`GlobalRateLimiter`** in [`providers/rate_limit.py`](https://github.com/Alishahryar1/free-claude-code/blob/main/providers/rate_limit.py) manages an `asyncio.Semaphore` as a singleton shared across all providers.
- **`concurrency_slot`** blocks when limits are reached, queuing excess requests until capacity frees.
- **Provider implementations** in [`providers/openai_compat.py`](https://github.com/Alishahryar1/free-claude-code/blob/main/providers/openai_compat.py) acquire slots before initiating remote connections.

## Frequently Asked Questions

### What happens when the concurrency limit is reached?

When `PROVIDER_MAX_CONCURRENCY` streams are active, subsequent requests block at `await self._concurrency_sem.acquire()` inside the `concurrency_slot` context manager. The coroutine pauses until another provider releases its slot, creating a natural backpressure mechanism that prevents overwhelming downstream APIs.

### Can different providers have different concurrency limits?

While each `ProviderConfig` can specify its own `max_concurrency`, the `GlobalRateLimiter` singleton pattern means all providers typically share the same semaphore instance created from the first provider's configuration. To enforce different limits per provider, you would need to modify the instantiation logic to create separate limiter instances rather than using the shared singleton.

### Where is the default concurrency value of 5 defined?

The default value originates in [`providers/base.py`](https://github.com/Alishahryar1/free-claude-code/blob/main/providers/base.py) within the `ProviderConfig` definition (line 21), where `max_concurrency` defaults to 5. This value is overridden when `Settings` in [`config/settings.py`](https://github.com/Alishahryar1/free-claude-code/blob/main/config/settings.py) injects the `PROVIDER_MAX_CONCURRENCY` environment variable during provider initialization.

### How does the system handle semaphore cleanup if a stream crashes?

The `concurrency_slot` context manager in [`providers/rate_limit.py`](https://github.com/Alishahryar1/free-claude-code/blob/main/providers/rate_limit.py) (lines 165‑176) uses a `try...finally` block to ensure `self._concurrency_sem.release()` executes regardless of whether the stream completes successfully or raises an exception. This guarantees that crashed or interrupted streams always return their permit to the semaphore pool, preventing resource leaks.