# How to Configure Rate Limiting and Memory-Adaptive Dispatchers in Crawl4AI

> Learn to configure rate limiting and memory-adaptive dispatchers in Crawl4AI. Efficiently manage per-domain delays and prevent OOM crashes with this guide.

- Repository: [UncleCode/crawl4ai](https://github.com/unclecode/crawl4ai)
- Tags: how-to-guide
- Published: 2026-03-05

---

**Configure rate limiting and memory-adaptive dispatchers in Crawl4AI by instantiating a `RateLimiter` to manage per-domain request delays and passing it to a `MemoryAdaptiveDispatcher` that monitors system RAM via `psutil` to prevent out-of-memory crashes during large-scale crawls.**

The `crawl4ai` library decouples crawling logic from concurrency management through pluggable dispatcher components. By separating the "what" (crawler logic) from the "how" (job dispatching), you can fine-tune request throttling and safeguard system resources when processing hundreds of concurrent URLs.

## Understanding the Dispatcher Architecture

Crawl4AI provides two primary dispatcher classes in [`crawl4ai/async_dispatcher.py`](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/async_dispatcher.py) that work together to control execution flow:

- **`RateLimiter`**: Manages per-domain request pacing, exponential back-off on HTTP 429/503 responses, and retry limits (lines 28-85).
- **`MemoryAdaptiveDispatcher`**: Monitors real-time memory consumption, toggles "pressure mode" when thresholds are crossed, and re-queues tasks to prevent OOM errors (lines 48-66 for initialization, lines 75-104 for the monitor task).

The `RateLimiter` maintains domain-specific state using the `DomainState` class defined in [`crawl4ai/models.py`](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/models.py), while the `MemoryAdaptiveDispatcher` relies on `get_true_memory_usage_percent` from [`crawl4ai/utils.py`](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/utils.py) to poll system resources.

## Configuring the RateLimiter

The `RateLimiter` class inserts randomized delays between requests to the same domain and automatically adjusts timing when rate-limit responses are detected.

### Constructor Parameters

Instantiate the limiter with precise control over back-off behavior:

```python
from crawl4ai.async_dispatcher import RateLimiter

rl = RateLimiter(
    base_delay=(0.5, 2.0),      # Random delay between 0.5s and 2.0s

    max_delay=30.0,             # Cap delay at 30 seconds

    max_retries=5,              # Retry up to 5 times

    rate_limit_codes=[429, 503] # Consider these status codes as rate limits

)

```

The `wait_if_needed` method checks the domain's last request timestamp before allowing new connections, while `update_delay` implements exponential back-off when the server returns rate-limit codes (lines 69-78 in [`async_dispatcher.py`](https://github.com/unclecode/crawl4ai/blob/main/async_dispatcher.py)).

## Implementing Memory-Adaptive Dispatching

The `MemoryAdaptiveDispatcher` wraps your rate limiter and adds memory-aware task scheduling to protect host resources.

### Memory Threshold Parameters

Configure the dispatcher with three distinct RAM thresholds:

```python
from crawl4ai.async_dispatcher import MemoryAdaptiveDispatcher

dispatcher = MemoryAdaptiveDispatcher(
    memory_threshold_percent=85.0,      # Enter pressure mode

    critical_threshold_percent=95.0,    # Aggressive re-queuing

    recovery_threshold_percent=75.0,    # Exit pressure mode

    max_session_permit=20,              # Max concurrent sessions

    fairness_timeout=600.0,             # Prevent task starvation

    memory_wait_timeout=600.0,          # Hard timeout for pressure

    rate_limiter=rl                     # Your RateLimiter instance

)

```

When system RAM exceeds `memory_threshold_percent`, the dispatcher enters pressure mode and assigns higher priority to new tasks. Crossing `critical_threshold_percent` triggers immediate re-queuing of active tasks with incremented retry counts and negative priority values (lines 90-106).

### The Memory Monitor Task

The dispatcher spawns a background coroutine `_memory_monitor_task` that polls `psutil` every second (configurable via `check_interval`). This task toggles the `memory_pressure_mode` flag based on current usage relative to your configured thresholds.

## Complete Integration Example

Combine both dispatchers with `AsyncWebCrawler` to process URL batches safely:

```python
from crawl4ai import AsyncWebCrawler
from crawl4ai.config import CrawlerRunConfig
from crawl4ai.async_dispatcher import RateLimiter, MemoryAdaptiveDispatcher

# 1. Configure rate limiting

rate_limiter = RateLimiter(
    base_delay=(1.0, 2.0),
    max_delay=30.0,
    max_retries=3,
    rate_limit_codes=[429]
)

# 2. Configure memory-adaptive dispatching

dispatcher = MemoryAdaptiveDispatcher(
    memory_threshold_percent=80.0,
    critical_threshold_percent=92.0,
    max_session_permit=15,
    fairness_timeout=300.0,
    rate_limiter=rate_limiter
)

# 3. Initialize crawler

crawler = AsyncWebCrawler()
config = CrawlerRunConfig()

# 4. Process URLs with automatic throttling and memory protection

urls = ["https://example.com/page1", "https://example.com/page2"]

results = await dispatcher.run_urls(urls, crawler, config)

```

For streaming results, use `run_urls_stream` instead:

```python
async for result in dispatcher.run_urls_stream(urls, crawler, config):
    print(f"{result.url}: success={result.result.success}")

```

## Customizing Dispatcher Behavior

Tailor the dispatchers to your specific workload requirements:

- **Per-domain strategies**: Modify `rate_limit_codes` to include 403 responses or adjust `max_retries` for stubborn endpoints.
- **Aggressive memory handling**: Lower `memory_threshold_percent` to 70.0 for early conservation, or set `memory_wait_timeout=None` to allow indefinite waiting during pressure.
- **Fairness tuning**: Increase `fairness_timeout` for strict FIFO ordering, or decrease it to prioritize long-waiting URLs and prevent starvation.

You can also nest dispatchers by passing a `SemaphoreDispatcher` as the base dispatcher argument in higher-level configurations (see [`docs/examples/dispatcher_example.py`](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/dispatcher_example.py)).

## Summary

- **RateLimiter** in [`crawl4ai/async_dispatcher.py`](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/async_dispatcher.py) provides per-domain throttling with exponential back-off for HTTP 429/503 responses.
- **MemoryAdaptiveDispatcher** monitors RAM via `psutil`, toggles pressure mode at configurable thresholds, and re-queues tasks to prevent OOM crashes.
- Both dispatchers integrate seamlessly with `AsyncWebCrawler` through the `run_urls` and `run_urls_stream` methods.
- Configure `memory_threshold_percent`, `critical_threshold_percent`, and `recovery_threshold_percent` to define custom memory safety zones.
- The `_memory_monitor_task` coroutine polls system resources every second, ensuring real-time adaptation to memory pressure.

## Frequently Asked Questions

### What is the difference between RateLimiter and MemoryAdaptiveDispatcher?

**`RateLimiter`** manages request pacing and server-friendly back-off behaviors, tracking per-domain state to prevent overwhelming target servers. **`MemoryAdaptiveDispatcher`** manages local system resources, monitoring RAM usage to prevent your crawling process from consuming all available memory and crashing.

### How does the memory monitor detect pressure?

The dispatcher spawns a background `_memory_monitor_task` that calls `get_true_memory_usage_percent` from [`crawl4ai/utils.py`](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/utils.py) (which wraps `psutil`) every `check_interval` seconds. When reported RAM exceeds `memory_threshold_percent`, the dispatcher sets `memory_pressure_mode` to true, triggering priority adjustments and potential task re-queuing.

### Can I use RateLimiter without MemoryAdaptiveDispatcher?

Yes, the `RateLimiter` is a standalone class that can be used with other dispatcher types or custom async implementations. However, `MemoryAdaptiveDispatcher` requires a `rate_limiter` parameter to handle per-domain throttling while managing memory constraints, making the combination recommended for production workloads.

### What happens when the critical memory threshold is reached?

When RAM usage exceeds `critical_threshold_percent`, the dispatcher immediately re-queues the current task with a negative priority value (higher priority) and increments its retry count, logging "Requeued due to critical memory pressure". This prevents the crawler from holding memory-intensive resources while the system is under extreme pressure.