How to Configure Rate Limiting and Memory-Adaptive Dispatchers in Crawl4AI

Configure rate limiting and memory-adaptive dispatchers in Crawl4AI by instantiating a RateLimiter to manage per-domain request delays and passing it to a MemoryAdaptiveDispatcher that monitors system RAM via psutil to prevent out-of-memory crashes during large-scale crawls.

The crawl4ai library decouples crawling logic from concurrency management through pluggable dispatcher components. By separating the "what" (crawler logic) from the "how" (job dispatching), you can fine-tune request throttling and safeguard system resources when processing hundreds of concurrent URLs.

Understanding the Dispatcher Architecture

Crawl4AI provides two primary dispatcher classes in crawl4ai/async_dispatcher.py that work together to control execution flow:

• RateLimiter: Manages per-domain request pacing, exponential back-off on HTTP 429/503 responses, and retry limits (lines 28-85).
• MemoryAdaptiveDispatcher: Monitors real-time memory consumption, toggles "pressure mode" when thresholds are crossed, and re-queues tasks to prevent OOM errors (lines 48-66 for initialization, lines 75-104 for the monitor task).

The RateLimiter maintains domain-specific state using the DomainState class defined in crawl4ai/models.py, while the MemoryAdaptiveDispatcher relies on get_true_memory_usage_percent from crawl4ai/utils.py to poll system resources.

Configuring the RateLimiter

The RateLimiter class inserts randomized delays between requests to the same domain and automatically adjusts timing when rate-limit responses are detected.

Constructor Parameters

Instantiate the limiter with precise control over back-off behavior:

from crawl4ai.async_dispatcher import RateLimiter

rl = RateLimiter(
    base_delay=(0.5, 2.0),      # Random delay between 0.5s and 2.0s

    max_delay=30.0,             # Cap delay at 30 seconds

    max_retries=5,              # Retry up to 5 times

    rate_limit_codes=[429, 503] # Consider these status codes as rate limits

)

The wait_if_needed method checks the domain's last request timestamp before allowing new connections, while update_delay implements exponential back-off when the server returns rate-limit codes (lines 69-78 in async_dispatcher.py).

Implementing Memory-Adaptive Dispatching

The MemoryAdaptiveDispatcher wraps your rate limiter and adds memory-aware task scheduling to protect host resources.

Memory Threshold Parameters

Configure the dispatcher with three distinct RAM thresholds:

from crawl4ai.async_dispatcher import MemoryAdaptiveDispatcher

dispatcher = MemoryAdaptiveDispatcher(
    memory_threshold_percent=85.0,      # Enter pressure mode

    critical_threshold_percent=95.0,    # Aggressive re-queuing

    recovery_threshold_percent=75.0,    # Exit pressure mode

    max_session_permit=20,              # Max concurrent sessions

    fairness_timeout=600.0,             # Prevent task starvation

    memory_wait_timeout=600.0,          # Hard timeout for pressure

    rate_limiter=rl                     # Your RateLimiter instance

)

When system RAM exceeds memory_threshold_percent, the dispatcher enters pressure mode and assigns higher priority to new tasks. Crossing critical_threshold_percent triggers immediate re-queuing of active tasks with incremented retry counts and negative priority values (lines 90-106).

The Memory Monitor Task

The dispatcher spawns a background coroutine _memory_monitor_task that polls psutil every second (configurable via check_interval). This task toggles the memory_pressure_mode flag based on current usage relative to your configured thresholds.

Complete Integration Example

Combine both dispatchers with AsyncWebCrawler to process URL batches safely:

from crawl4ai import AsyncWebCrawler
from crawl4ai.config import CrawlerRunConfig
from crawl4ai.async_dispatcher import RateLimiter, MemoryAdaptiveDispatcher

# 1. Configure rate limiting

rate_limiter = RateLimiter(
    base_delay=(1.0, 2.0),
    max_delay=30.0,
    max_retries=3,
    rate_limit_codes=[429]
)

# 2. Configure memory-adaptive dispatching

dispatcher = MemoryAdaptiveDispatcher(
    memory_threshold_percent=80.0,
    critical_threshold_percent=92.0,
    max_session_permit=15,
    fairness_timeout=300.0,
    rate_limiter=rate_limiter
)

# 3. Initialize crawler

crawler = AsyncWebCrawler()
config = CrawlerRunConfig()

# 4. Process URLs with automatic throttling and memory protection

urls = ["https://example.com/page1", "https://example.com/page2"]

results = await dispatcher.run_urls(urls, crawler, config)

For streaming results, use run_urls_stream instead:

async for result in dispatcher.run_urls_stream(urls, crawler, config):
    print(f"{result.url}: success={result.result.success}")

Customizing Dispatcher Behavior

Tailor the dispatchers to your specific workload requirements:

• Per-domain strategies: Modify rate_limit_codes to include 403 responses or adjust max_retries for stubborn endpoints.
• Aggressive memory handling: Lower memory_threshold_percent to 70.0 for early conservation, or set memory_wait_timeout=None to allow indefinite waiting during pressure.
• Fairness tuning: Increase fairness_timeout for strict FIFO ordering, or decrease it to prioritize long-waiting URLs and prevent starvation.

You can also nest dispatchers by passing a SemaphoreDispatcher as the base dispatcher argument in higher-level configurations (see docs/examples/dispatcher_example.py).

Summary

• RateLimiter in crawl4ai/async_dispatcher.py provides per-domain throttling with exponential back-off for HTTP 429/503 responses.
• MemoryAdaptiveDispatcher monitors RAM via psutil, toggles pressure mode at configurable thresholds, and re-queues tasks to prevent OOM crashes.
• Both dispatchers integrate seamlessly with AsyncWebCrawler through the run_urls and run_urls_stream methods.
• Configure memory_threshold_percent, critical_threshold_percent, and recovery_threshold_percent to define custom memory safety zones.
• The _memory_monitor_task coroutine polls system resources every second, ensuring real-time adaptation to memory pressure.

Frequently Asked Questions

What is the difference between RateLimiter and MemoryAdaptiveDispatcher?

RateLimiter manages request pacing and server-friendly back-off behaviors, tracking per-domain state to prevent overwhelming target servers. MemoryAdaptiveDispatcher manages local system resources, monitoring RAM usage to prevent your crawling process from consuming all available memory and crashing.

How does the memory monitor detect pressure?

The dispatcher spawns a background _memory_monitor_task that calls get_true_memory_usage_percent from crawl4ai/utils.py (which wraps psutil) every check_interval seconds. When reported RAM exceeds memory_threshold_percent, the dispatcher sets memory_pressure_mode to true, triggering priority adjustments and potential task re-queuing.

Can I use RateLimiter without MemoryAdaptiveDispatcher?

Yes, the RateLimiter is a standalone class that can be used with other dispatcher types or custom async implementations. However, MemoryAdaptiveDispatcher requires a rate_limiter parameter to handle per-domain throttling while managing memory constraints, making the combination recommended for production workloads.

What happens when the critical memory threshold is reached?

When RAM usage exceeds critical_threshold_percent, the dispatcher immediately re-queues the current task with a negative priority value (higher priority) and increments its retry count, logging "Requeued due to critical memory pressure". This prevents the crawler from holding memory-intensive resources while the system is under extreme pressure.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}