How to Configure Rate Limiting and Sleep Intervals in MediaCrawler

MediaCrawler controls request throttling through the CRAWLER_MAX_SLEEP_SEC constant in config/base_config.py, which all platform-specific crawlers import to enforce consistent sleep intervals between HTTP requests.

MediaCrawler is an open-source multi-platform scraping framework supporting Zhihu, XiaoHongShu, Weibo, Douyin, and other Chinese social media sites. To avoid triggering server-side rate limits or IP bans, the project implements a centralized sleep mechanism that applies uniformly across all crawler implementations. Understanding how to configure rate limiting and sleep intervals in MediaCrawler ensures reliable data extraction without service interruptions.

The Central Configuration in base_config.py

The CRAWLER_MAX_SLEEP_SEC Constant

All throttle timing in MediaCrawler stems from a single constant defined in config/base_config.py. The default configuration sets a 2-second delay between requests:

# config/base_config.py

CRAWLER_MAX_SLEEP_SEC = 2

This value represents the maximum sleep duration in seconds that each crawler will wait before proceeding to the next request. The constant is located at lines 136-138 in the source file.

How Crawlers Enforce Sleep Intervals

Every platform-specific crawler imports the shared configuration module and invokes await asyncio.sleep(config.CRAWLER_MAX_SLEEP_SEC) at logical pause points. In media_platform/zhihu/core.py, the crawler pauses after fetching each page:

# media_platform/zhihu/core.py

await asyncio.sleep(config.CRAWLER_MAX_SLEEP_SEC)
utils.logger.info(
    f"[ZhihuCrawler.search] Sleeping for {config.CRAWLER_MAX_SLEEP_SEC} seconds after page {page-1}"
)

This pattern repeats identically across all supported platforms. The XiaoHongShu (media_platform/xhs/core.py), Weibo (media_platform/weibo/core.py), Tieba (media_platform/tieba/core.py), Kuaishou (media_platform/kuaishou/core.py), Douyin (media_platform/douyin/core.py), and Bilibili (media_platform/bilibili/core.py) crawlers all reference the same config.CRAWLER_MAX_SLEEP_SEC constant to maintain consistent throttling behavior.

Adjusting the Global Sleep Interval

To change the request frequency, modify the CRAWLER_MAX_SLEEP_SEC value in config/base_config.py. For example, to increase the delay to 5 seconds between requests:

# config/base_config.py

CRAWLER_MAX_SLEEP_SEC = 5

Set this to any positive float or integer based on your target platform's rate limits and your account's trust level.

Advanced Throttling Configuration

Beyond the basic sleep interval, MediaCrawler provides several complementary mechanisms to control request throughput.

Limiting Concurrent Crawlers

The MAX_CONCURRENCY_NUM setting in config/base_config.py controls how many crawlers execute simultaneously. The default is 1, but lowering this value (or keeping it at 1) ensures sequential processing that respects the sleep interval.

Capping Request Volume

You can indirectly reduce request rates by limiting the total items fetched:

• CRAWLER_MAX_NOTES_COUNT: Restricts the total number of posts scraped
• CRAWLER_MAX_COMMENTS_COUNT_SINGLENOTES: Limits comments retrieved per post

These settings reduce the absolute number of HTTP requests, decreasing the likelihood of detection regardless of sleep intervals.

Proxy Rotation

Enable ENABLE_IP_PROXY in the configuration and configure a proxy pool to distribute requests across multiple IP addresses. This mitigates server-side rate limiting by preventing any single IP from hitting request thresholds.

Runtime Configuration Overrides

If you cannot modify the source files directly, override the constant at runtime by importing the configuration module and reassigning the variable before starting the crawler. Since MediaCrawler loads configuration on import, ensure this reassignment occurs before the first await asyncio.sleep call:

# my_custom_config.py

from MediaCrawler.config import base_config as cfg

# Override the global sleep interval

cfg.CRAWLER_MAX_SLEEP_SEC = 3  # 3 seconds between requests

# Now start the crawler as usual

from MediaCrawler.main import run_crawler
run_crawler()

This approach allows environment-specific configurations without altering the repository's tracked files.

Summary

• Centralized control: All rate limiting depends on CRAWLER_MAX_SLEEP_SEC in config/base_config.py
• Uniform enforcement: Every platform crawler calls await asyncio.sleep(config.CRAWLER_MAX_SLEEP_SEC) between requests
• Simple adjustment: Change the integer value in base_config.py to increase or decrease delays
• Complementary limits: Use MAX_CONCURRENCY_NUM, CRAWLER_MAX_NOTES_COUNT, and proxy settings for fine-grained control
• Runtime flexibility: Override configuration values by importing and modifying the config module before execution

Frequently Asked Questions

Where is the sleep interval defined in MediaCrawler?

The sleep interval is defined as CRAWLER_MAX_SLEEP_SEC in config/base_config.py with a default value of 2 seconds. All platform-specific crawlers import this constant to ensure consistent throttling across the entire application.

Can I set different sleep intervals for different platforms?

Currently, MediaCrawler uses a single global constant for all platforms. To implement platform-specific delays, you would need to modify the individual crawler files in media_platform/<platform>/core.py to use a different sleep value or conditional logic based on the platform.

How do I prevent getting banned while scraping?

Combine the CRAWLER_MAX_SLEEP_SEC setting with ENABLE_IP_PROXY to rotate IP addresses, reduce MAX_CONCURRENCY_NUM to 1, and limit CRAWLER_MAX_NOTES_COUNT to keep total requests below platform thresholds. Longer sleep intervals (5-10 seconds) significantly reduce ban risk.

Why does the crawler sleep even when I set a low value?

The asyncio.sleep call executes regardless of value, but if you set CRAWLER_MAX_SLEEP_SEC to 0 or a very low decimal (e.g., 0.1), the pause will be minimal. Ensure you restarted the crawler after modifying config/base_config.py, as configuration is loaded at import time and cached for the session.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}