# How to Handle Rate Limits with MediaCrawler: Configuration and Best Practices

> Learn to handle rate limits with MediaCrawler. Discover configurations for fixed spacing, automatic retries, and IP proxy rotation to optimize your web scraping.

- Repository: [程序员阿江-Relakkes/MediaCrawler](https://github.com/NanmiCoder/MediaCrawler)
- Tags: best-practices
- Published: 2026-06-28

---

**MediaCrawler handles rate limits through three complementary mechanisms: fixed request spacing via `CRAWLER_MAX_SLEEP_SEC`, automatic retry logic using the `tenacity` library, and IP proxy rotation via [`proxy_ip_pool.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/proxy_ip_pool.py).**

MediaCrawler is an open-source multi-platform crawler supporting XiaoHongShu, Zhihu, Weibo, Douyin, Bilibili, and Tieba. Understanding how to handle rate limits with MediaCrawler is essential for maintaining stable data collection without triggering platform bans. The architecture implements request throttling, intelligent retry policies, and proxy rotation to respect the rate-limiting policies of every target platform.

## Understanding MediaCrawler's Rate Limit Architecture

The crawler implements a three-layer defense against rate limiting. Each layer targets different stages of the HTTP request lifecycle, ensuring robust operation across diverse platform restrictions.

### Fixed Request Spacing with CRAWLER_MAX_SLEEP_SEC

The primary mechanism inserts a configurable pause after each HTTP request. In [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py), the global constant `CRAWLER_MAX_SLEEP_SEC` defines the default interval, set to 2 seconds.

Every platform implementation applies this throttle uniformly. For example, in [`media_platform/zhihu/core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/zhihu/core.py) lines 189-190, the code executes `await asyncio.sleep(config.CRAWLER_MAX_SLEEP_SEC)` after retrieving content. Similar implementations appear in [`media_platform/weibo/core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/weibo/core.py) (lines 187-188) and [`media_platform/tieba/core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/tieba/core.py) (lines 200-201).

This approach keeps the overall request frequency below platform limits by design.

### Retry Logic for Transient Failures

When a request returns HTTP 429 (Too Many Requests) or encounters temporary network errors, MediaCrawler automatically retries the operation. The `tenacity` library decorates low-level request functions with retry policies.

In [`media_platform/weibo/client.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/weibo/client.py) lines 72-73, the decorator `@retry(stop=stop_after_attempt(3), wait=wait_fixed(1))` ensures failed requests retry up to three times with a one-second fixed wait between attempts. Similar decorators appear in the Zhihu, XiaoHongShu, and Tieba client implementations.

After exhausting retry attempts, the exception propagates to the crawler core, which logs the failure and continues processing.

### IP Proxy Rotation

For scenarios requiring higher throughput, MediaCrawler supports rotating IP proxies to distribute requests across multiple addresses. The proxy configuration resides in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py) with flags including `ENABLE_IP_PROXY`, `IP_PROXY_PROVIDER_NAME`, and `STATIC_PROXY_URL`.

The actual pool implementation lives in [`proxy/proxy_ip_pool.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/proxy/proxy_ip_pool.py). The `ProxyMixin` class injects fresh proxies into the HTTP client created by [`tools/httpx_util.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/httpx_util.py), which configures the `httpx.AsyncClient` with appropriate proxy settings.

When the current IP encounters throttling, the pool automatically provides a new address for subsequent requests.

## Configuring Rate Limit Settings

MediaCrawler offers both static configuration and dynamic runtime adjustments to fine-tune rate limiting behavior.

### Global Configuration in base_config.py

Modify [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py) to set baseline behavior:

- Set `CRAWLER_MAX_SLEEP_SEC = 2` to change the global delay between requests
- Enable `ENABLE_IP_PROXY = True` to activate proxy rotation
- Configure `IP_PROXY_POOL_COUNT` to control the proxy pool size
- Specify `IP_PROXY_PROVIDER_NAME` using supported values like `kuaidaili`, `wandouhttp`, or `static`

These settings apply to all platform crawlers unless overridden at runtime.

### CLI Overrides for Runtime Adjustment

Pass custom limits when launching crawls to reduce request volume and avoid hitting rate limits. The command-line argument parser in [`cmd_arg/arg.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/cmd_arg/arg.py) (lines 197-266) processes these flags:

- `--crawler_max_notes_count`: Limits total notes fetched per run
- `--max_comments_count_singlenotes`: Caps comments collected per note

Example command:

```bash
python -m main --platform xhs --crawler_max_notes_count 100 --max_comments_count_singlenotes 20

```

## Implementing Custom Rate Limiting Strategies

For platform-specific requirements, implement custom logic in your crawler scripts.

Adjust the sleep interval dynamically for stricter platforms:

```python
if platform == "douyin":
    # Douyin enforces stricter limits; increase delay

    config.CRAWLER_MAX_SLEEP_SEC = 5

```

Customize retry policies for specific clients:

```python
from tenacity import retry, stop_after_attempt, wait_fixed

@retry(stop=stop_after_attempt(5), wait=wait_fixed(2))
async def fetch_note_page(self, note_id):
    resp = await self.http_client.get(f"https://weibo.com/note/{note_id}")
    resp.raise_for_status()
    return resp.json()

```

Run crawls with environment variables:

```bash
export CRAWLER_MAX_SLEEP_SEC=3
export ENABLE_IP_PROXY=True
export STATIC_PROXY_URL="http://user:pass@my-proxy:8080"
python -m main --platform zhihu

```

## Key Files for Rate Limit Management

Understanding the codebase structure helps when debugging rate limit issues:

- [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py): Global crawler settings including sleep intervals and proxy flags
- `media_platform/*/core.py`: Platform-specific crawling loops that invoke `asyncio.sleep()` after each request
- [`proxy/proxy_ip_pool.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/proxy/proxy_ip_pool.py): Rotating pool implementation for IP proxy management
- [`cmd_arg/arg.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/cmd_arg/arg.py): Command-line argument parser for runtime configuration
- [`tools/httpx_util.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/httpx_util.py): Factory for `httpx.AsyncClient` with proxy and SSL settings
- [`tests/test_api_limits.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tests/test_api_limits.py): Test suite validating limit-related CLI arguments

## Summary

MediaCrawler provides a robust framework for handling rate limits across Chinese social media platforms:

- **Fixed request spacing**: Control crawl speed via `CRAWLER_MAX_SLEEP_SEC` in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py) or CLI arguments
- **Automatic retry logic**: The `tenacity` library handles HTTP 429 and transient errors with configurable backoff strategies
- **IP proxy rotation**: Enable `ENABLE_IP_PROXY` and configure [`proxy_ip_pool.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/proxy_ip_pool.py) to distribute requests across multiple IP addresses
- **Runtime adjustment**: Use CLI flags like `--crawler_max_notes_count` to limit data volume and reduce request frequency

## Frequently Asked Questions

### How do I increase the delay between requests to avoid rate limits?

Set the `CRAWLER_MAX_SLEEP_SEC` environment variable or modify the value in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py). The default is 2 seconds, but you can increase it to 3-5 seconds for stricter platforms. Every platform core implementation in `media_platform/*/core.py` calls `await asyncio.sleep(config.CRAWLER_MAX_SLEEP_SEC)` after each request, so this change applies globally.

### What happens when MediaCrawler encounters an HTTP 429 error?

The `tenacity` decorator on client methods (visible in [`media_platform/weibo/client.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/weibo/client.py) and similar files) automatically retries the request up to three times with a one-second fixed wait. If all retries fail, the exception bubbles up to the crawler core, which logs the error and continues with the next item. You can customize the retry count by modifying the `stop_after_attempt` parameter in the decorator.

### Can I use proxies to bypass rate limits in MediaCrawler?

Yes. Enable proxy support by setting `ENABLE_IP_PROXY = True` in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py) and configure your provider settings (`IP_PROXY_PROVIDER_NAME`, `STATIC_PROXY_URL`, etc.). The [`proxy/proxy_ip_pool.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/proxy/proxy_ip_pool.py) module manages the rotation automatically, providing fresh IPs through the `ProxyMixin` class when the current address encounters throttling.

### How do I limit the number of requests per crawl to stay under rate limits?

Use the CLI arguments defined in [`cmd_arg/arg.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/cmd_arg/arg.py) to cap data collection: `--crawler_max_notes_count` limits total items fetched, while `--max_comments_count_singlenotes` restricts comments per item. These settings reduce the total request volume, making it easier to stay within platform limits without modifying the sleep interval.