How to Handle Rate Limits with MediaCrawler: Configuration and Best Practices

MediaCrawler handles rate limits through three complementary mechanisms: fixed request spacing via CRAWLER_MAX_SLEEP_SEC, automatic retry logic using the tenacity library, and IP proxy rotation via proxy_ip_pool.py.

MediaCrawler is an open-source multi-platform crawler supporting XiaoHongShu, Zhihu, Weibo, Douyin, Bilibili, and Tieba. Understanding how to handle rate limits with MediaCrawler is essential for maintaining stable data collection without triggering platform bans. The architecture implements request throttling, intelligent retry policies, and proxy rotation to respect the rate-limiting policies of every target platform.

Understanding MediaCrawler's Rate Limit Architecture

The crawler implements a three-layer defense against rate limiting. Each layer targets different stages of the HTTP request lifecycle, ensuring robust operation across diverse platform restrictions.

Fixed Request Spacing with CRAWLER_MAX_SLEEP_SEC

The primary mechanism inserts a configurable pause after each HTTP request. In config/base_config.py, the global constant CRAWLER_MAX_SLEEP_SEC defines the default interval, set to 2 seconds.

Every platform implementation applies this throttle uniformly. For example, in media_platform/zhihu/core.py lines 189-190, the code executes await asyncio.sleep(config.CRAWLER_MAX_SLEEP_SEC) after retrieving content. Similar implementations appear in media_platform/weibo/core.py (lines 187-188) and media_platform/tieba/core.py (lines 200-201).

This approach keeps the overall request frequency below platform limits by design.

Retry Logic for Transient Failures

When a request returns HTTP 429 (Too Many Requests) or encounters temporary network errors, MediaCrawler automatically retries the operation. The tenacity library decorates low-level request functions with retry policies.

In media_platform/weibo/client.py lines 72-73, the decorator @retry(stop=stop_after_attempt(3), wait=wait_fixed(1)) ensures failed requests retry up to three times with a one-second fixed wait between attempts. Similar decorators appear in the Zhihu, XiaoHongShu, and Tieba client implementations.

After exhausting retry attempts, the exception propagates to the crawler core, which logs the failure and continues processing.

IP Proxy Rotation

For scenarios requiring higher throughput, MediaCrawler supports rotating IP proxies to distribute requests across multiple addresses. The proxy configuration resides in config/base_config.py with flags including ENABLE_IP_PROXY, IP_PROXY_PROVIDER_NAME, and STATIC_PROXY_URL.

The actual pool implementation lives in proxy/proxy_ip_pool.py. The ProxyMixin class injects fresh proxies into the HTTP client created by tools/httpx_util.py, which configures the httpx.AsyncClient with appropriate proxy settings.

When the current IP encounters throttling, the pool automatically provides a new address for subsequent requests.

Configuring Rate Limit Settings

MediaCrawler offers both static configuration and dynamic runtime adjustments to fine-tune rate limiting behavior.

Global Configuration in base_config.py

Modify config/base_config.py to set baseline behavior:

• Set CRAWLER_MAX_SLEEP_SEC = 2 to change the global delay between requests
• Enable ENABLE_IP_PROXY = True to activate proxy rotation
• Configure IP_PROXY_POOL_COUNT to control the proxy pool size
• Specify IP_PROXY_PROVIDER_NAME using supported values like kuaidaili, wandouhttp, or static

These settings apply to all platform crawlers unless overridden at runtime.

CLI Overrides for Runtime Adjustment

Pass custom limits when launching crawls to reduce request volume and avoid hitting rate limits. The command-line argument parser in cmd_arg/arg.py (lines 197-266) processes these flags:

• --crawler_max_notes_count: Limits total notes fetched per run
• --max_comments_count_singlenotes: Caps comments collected per note

Example command:

python -m main --platform xhs --crawler_max_notes_count 100 --max_comments_count_singlenotes 20

Implementing Custom Rate Limiting Strategies

For platform-specific requirements, implement custom logic in your crawler scripts.

Adjust the sleep interval dynamically for stricter platforms:

if platform == "douyin":
    # Douyin enforces stricter limits; increase delay

    config.CRAWLER_MAX_SLEEP_SEC = 5

Customize retry policies for specific clients:

from tenacity import retry, stop_after_attempt, wait_fixed

@retry(stop=stop_after_attempt(5), wait=wait_fixed(2))
async def fetch_note_page(self, note_id):
    resp = await self.http_client.get(f"https://weibo.com/note/{note_id}")
    resp.raise_for_status()
    return resp.json()

Run crawls with environment variables:

export CRAWLER_MAX_SLEEP_SEC=3
export ENABLE_IP_PROXY=True
export STATIC_PROXY_URL="http://user:pass@my-proxy:8080"
python -m main --platform zhihu

Key Files for Rate Limit Management

Understanding the codebase structure helps when debugging rate limit issues:

• config/base_config.py: Global crawler settings including sleep intervals and proxy flags
• media_platform/*/core.py: Platform-specific crawling loops that invoke asyncio.sleep() after each request
• proxy/proxy_ip_pool.py: Rotating pool implementation for IP proxy management
• cmd_arg/arg.py: Command-line argument parser for runtime configuration
• tools/httpx_util.py: Factory for httpx.AsyncClient with proxy and SSL settings
• tests/test_api_limits.py: Test suite validating limit-related CLI arguments

Summary

MediaCrawler provides a robust framework for handling rate limits across Chinese social media platforms:

• Fixed request spacing: Control crawl speed via CRAWLER_MAX_SLEEP_SEC in config/base_config.py or CLI arguments
• Automatic retry logic: The tenacity library handles HTTP 429 and transient errors with configurable backoff strategies
• IP proxy rotation: Enable ENABLE_IP_PROXY and configure proxy_ip_pool.py to distribute requests across multiple IP addresses
• Runtime adjustment: Use CLI flags like --crawler_max_notes_count to limit data volume and reduce request frequency

Frequently Asked Questions

How do I increase the delay between requests to avoid rate limits?

Set the CRAWLER_MAX_SLEEP_SEC environment variable or modify the value in config/base_config.py. The default is 2 seconds, but you can increase it to 3-5 seconds for stricter platforms. Every platform core implementation in media_platform/*/core.py calls await asyncio.sleep(config.CRAWLER_MAX_SLEEP_SEC) after each request, so this change applies globally.

What happens when MediaCrawler encounters an HTTP 429 error?

The tenacity decorator on client methods (visible in media_platform/weibo/client.py and similar files) automatically retries the request up to three times with a one-second fixed wait. If all retries fail, the exception bubbles up to the crawler core, which logs the error and continues with the next item. You can customize the retry count by modifying the stop_after_attempt parameter in the decorator.

Can I use proxies to bypass rate limits in MediaCrawler?

Yes. Enable proxy support by setting ENABLE_IP_PROXY = True in config/base_config.py and configure your provider settings (IP_PROXY_PROVIDER_NAME, STATIC_PROXY_URL, etc.). The proxy/proxy_ip_pool.py module manages the rotation automatically, providing fresh IPs through the ProxyMixin class when the current address encounters throttling.

How do I limit the number of requests per crawl to stay under rate limits?

Use the CLI arguments defined in cmd_arg/arg.py to cap data collection: --crawler_max_notes_count limits total items fetched, while --max_comments_count_singlenotes restricts comments per item. These settings reduce the total request volume, making it easier to stay within platform limits without modifying the sleep interval.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}