# How to Configure Rate Limiting and Sleep Intervals in MediaCrawler

> Learn how to configure rate limiting and sleep intervals in MediaCrawler. Control request throttling efficiently using CRAWLER_MAX_SLEEP_SEC for consistent intervals.

- Repository: [程序员阿江-Relakkes/MediaCrawler](https://github.com/NanmiCoder/MediaCrawler)
- Tags: how-to-guide
- Published: 2026-06-27

---

**MediaCrawler controls request throttling through the `CRAWLER_MAX_SLEEP_SEC` constant in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py), which all platform-specific crawlers import to enforce consistent sleep intervals between HTTP requests.**

MediaCrawler is an open-source multi-platform scraping framework supporting Zhihu, XiaoHongShu, Weibo, Douyin, and other Chinese social media sites. To avoid triggering server-side rate limits or IP bans, the project implements a centralized sleep mechanism that applies uniformly across all crawler implementations. Understanding how to configure rate limiting and sleep intervals in MediaCrawler ensures reliable data extraction without service interruptions.

## The Central Configuration in base_config.py

### The CRAWLER_MAX_SLEEP_SEC Constant

All throttle timing in MediaCrawler stems from a single constant defined in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py). The default configuration sets a 2-second delay between requests:

```python

# config/base_config.py

CRAWLER_MAX_SLEEP_SEC = 2

```

This value represents the maximum sleep duration in seconds that each crawler will wait before proceeding to the next request. The constant is located at lines 136-138 in the source file.

## How Crawlers Enforce Sleep Intervals

Every platform-specific crawler imports the shared configuration module and invokes `await asyncio.sleep(config.CRAWLER_MAX_SLEEP_SEC)` at logical pause points. In [`media_platform/zhihu/core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/zhihu/core.py), the crawler pauses after fetching each page:

```python

# media_platform/zhihu/core.py

await asyncio.sleep(config.CRAWLER_MAX_SLEEP_SEC)
utils.logger.info(
    f"[ZhihuCrawler.search] Sleeping for {config.CRAWLER_MAX_SLEEP_SEC} seconds after page {page-1}"
)

```

This pattern repeats identically across all supported platforms. The XiaoHongShu ([`media_platform/xhs/core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/xhs/core.py)), Weibo ([`media_platform/weibo/core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/weibo/core.py)), Tieba ([`media_platform/tieba/core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/tieba/core.py)), Kuaishou ([`media_platform/kuaishou/core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/kuaishou/core.py)), Douyin ([`media_platform/douyin/core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/douyin/core.py)), and Bilibili ([`media_platform/bilibili/core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/bilibili/core.py)) crawlers all reference the same `config.CRAWLER_MAX_SLEEP_SEC` constant to maintain consistent throttling behavior.

## Adjusting the Global Sleep Interval

To change the request frequency, modify the `CRAWLER_MAX_SLEEP_SEC` value in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py). For example, to increase the delay to 5 seconds between requests:

```python

# config/base_config.py

CRAWLER_MAX_SLEEP_SEC = 5

```

Set this to any positive float or integer based on your target platform's rate limits and your account's trust level.

## Advanced Throttling Configuration

Beyond the basic sleep interval, MediaCrawler provides several complementary mechanisms to control request throughput.

### Limiting Concurrent Crawlers

The `MAX_CONCURRENCY_NUM` setting in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py) controls how many crawlers execute simultaneously. The default is 1, but lowering this value (or keeping it at 1) ensures sequential processing that respects the sleep interval.

### Capping Request Volume

You can indirectly reduce request rates by limiting the total items fetched:

- **CRAWLER_MAX_NOTES_COUNT**: Restricts the total number of posts scraped
- **CRAWLER_MAX_COMMENTS_COUNT_SINGLENOTES**: Limits comments retrieved per post

These settings reduce the absolute number of HTTP requests, decreasing the likelihood of detection regardless of sleep intervals.

### Proxy Rotation

Enable `ENABLE_IP_PROXY` in the configuration and configure a proxy pool to distribute requests across multiple IP addresses. This mitigates server-side rate limiting by preventing any single IP from hitting request thresholds.

## Runtime Configuration Overrides

If you cannot modify the source files directly, override the constant at runtime by importing the configuration module and reassigning the variable before starting the crawler. Since MediaCrawler loads configuration on import, ensure this reassignment occurs before the first `await asyncio.sleep` call:

```python

# my_custom_config.py

from MediaCrawler.config import base_config as cfg

# Override the global sleep interval

cfg.CRAWLER_MAX_SLEEP_SEC = 3  # 3 seconds between requests

# Now start the crawler as usual

from MediaCrawler.main import run_crawler
run_crawler()

```

This approach allows environment-specific configurations without altering the repository's tracked files.

## Summary

- **Centralized control**: All rate limiting depends on `CRAWLER_MAX_SLEEP_SEC` in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py)
- **Uniform enforcement**: Every platform crawler calls `await asyncio.sleep(config.CRAWLER_MAX_SLEEP_SEC)` between requests
- **Simple adjustment**: Change the integer value in [`base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/base_config.py) to increase or decrease delays
- **Complementary limits**: Use `MAX_CONCURRENCY_NUM`, `CRAWLER_MAX_NOTES_COUNT`, and proxy settings for fine-grained control
- **Runtime flexibility**: Override configuration values by importing and modifying the config module before execution

## Frequently Asked Questions

### Where is the sleep interval defined in MediaCrawler?

The sleep interval is defined as `CRAWLER_MAX_SLEEP_SEC` in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py) with a default value of 2 seconds. All platform-specific crawlers import this constant to ensure consistent throttling across the entire application.

### Can I set different sleep intervals for different platforms?

Currently, MediaCrawler uses a single global constant for all platforms. To implement platform-specific delays, you would need to modify the individual crawler files in `media_platform/<platform>/core.py` to use a different sleep value or conditional logic based on the platform.

### How do I prevent getting banned while scraping?

Combine the `CRAWLER_MAX_SLEEP_SEC` setting with `ENABLE_IP_PROXY` to rotate IP addresses, reduce `MAX_CONCURRENCY_NUM` to 1, and limit `CRAWLER_MAX_NOTES_COUNT` to keep total requests below platform thresholds. Longer sleep intervals (5-10 seconds) significantly reduce ban risk.

### Why does the crawler sleep even when I set a low value?

The `asyncio.sleep` call executes regardless of value, but if you set `CRAWLER_MAX_SLEEP_SEC` to 0 or a very low decimal (e.g., 0.1), the pause will be minimal. Ensure you restarted the crawler after modifying [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py), as configuration is loaded at import time and cached for the session.