# How to Configure Rate Limiting and Sleep Intervals Using CRAWLER_MAX_SLEEP_SEC in MediaCrawler

> Learn to configure rate limiting and sleep intervals in MediaCrawler by adjusting CRAWLER_MAX_SLEEP_SEC. Control pause duration between HTTP requests for better crawling. Find out how in this guide.

- Repository: [程序员阿江-Relakkes/MediaCrawler](https://github.com/NanmiCoder/MediaCrawler)
- Tags: how-to-guide
- Published: 2026-06-29

---

**Set the `CRAWLER_MAX_SLEEP_SEC` constant in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py) to control the pause duration between HTTP requests across all crawler platforms.**

MediaCrawler implements request throttling through a unified sleep mechanism that prevents API rate limit violations and mimics human browsing patterns. The `CRAWLER_MAX_SLEEP_SEC` configuration value serves as the single source of truth for pacing across every supported platform including Zhihu, XiaoHongShu, Tieba, and Weibo. Understanding how to configure this parameter ensures optimal scraping speeds while minimizing the risk of IP bans or service blocks.

## Understanding CRAWLER_MAX_SLEEP_SEC

The global constant **`CRAWLER_MAX_SLEEP_SEC`** defines the default interval (in seconds) that the crawler waits between consecutive HTTP requests. Defined at line 137 in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py), this value propagates to all platform-specific implementations through a shared configuration object.

When the crawler executes, it calls `await asyncio.sleep(config.CRAWLER_MAX_SLEEP_SEC)` after each network operation. This asynchronous yield returns control to the event loop, allowing other coroutines to process while enforcing the mandatory delay. The implementation guarantees consistent pacing behavior regardless of which social media platform the crawler targets.

## How Rate Limiting Works Under the Hood

MediaCrawler's rate limiting operates through three distinct phases:

1. **Configuration Loading** – At import time, Python executes [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py), binding `CRAWLER_MAX_SLEEP_SEC` to the global config namespace.
2. **Request Execution** – Platform crawlers perform HTTP operations using `aiohttp` or similar async libraries.
3. **Throttling Application** – Immediately following each response, the crawler invokes `asyncio.sleep()` using the configured interval before processing the next item.

This pattern appears consistently across the codebase. For example, in [`media_platform/zhihu/core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/zhihu/core.py) at lines 188-191, the Zhihu crawler explicitly logs the sleep duration:

```python
await asyncio.sleep(config.CRAWLER_MAX_SLEEP_SEC)
utils.logger.info(f"[ZhihuCrawler.search] Sleeping for "
                  f"{config.CRAWLER_MAX_SLEEP_SEC} seconds after page {page-1}")

```

Identical implementations exist in [`media_platform/tieba/core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/tieba/core.py) (lines 200-201) and [`media_platform/weibo/core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/weibo/core.py) (lines 187-188), ensuring uniform behavior across platforms.

## Configuring Sleep Intervals

### Modifying the Global Configuration

The simplest method to adjust rate limiting involves editing the source configuration file directly. Open [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py) and modify line 137:

```python

# config/base_config.py

CRAWLER_MAX_SLEEP_SEC = 5  # Pauses 5 seconds between requests

```

This change applies universally to all crawler instances upon the next execution. Values typically range from 1 (aggressive) to 10+ seconds (conservative), depending on the target platform's tolerance.

### Runtime Override via CLI Arguments

For dynamic adjustment without code modification, override the configuration at startup through the main entry point. The following pattern allows command-line specification of sleep intervals:

```python

# main.py (entry point implementation)

import argparse
import config

parser = argparse.ArgumentParser()
parser.add_argument("--sleep-sec", type=int, 
                    help="Override the default crawler sleep interval")
args = parser.parse_args()

if args.sleep_sec:
    config.CRAWLER_MAX_SLEEP_SEC = args.sleep_sec

```

Execute the crawler with custom timing:

```bash
python main.py --sleep-sec 3

```

This approach assigns `3` to `config.CRAWLER_MAX_SLEEP_SEC`, overriding the default value before any crawler tasks initialize.

## Platform-Specific Implementation Examples

Each media platform follows the same throttling pattern while maintaining separate logging contexts. The following examples demonstrate the consistency:

**Zhihu Implementation** ([`media_platform/zhihu/core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/zhihu/core.py)):

```python
await asyncio.sleep(config.CRAWLER_MAX_SLEEP_SEC)

```

**Tieba Implementation** ([`media_platform/tieba/core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/tieba/core.py)):

```python
await asyncio.sleep(config.CRAWLER_MAX_SLEEP_SEC)

```

**Weibo Implementation** ([`media_platform/weibo/core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/weibo/core.py)):

```python
await asyncio.sleep(config.CRAWLER_MAX_SLEEP_SEC)

```

All platforms reference the identical config object, ensuring that changing `CRAWLER_MAX_SLEEP_SEC` in one location affects the entire application uniformly.

## Balancing Throughput and API Safety

The sleep interval interacts with other configuration limits to shape overall system throughput. MediaCrawler respects maximum content boundaries defined by **`CRAWLER_MAX_NOTES_COUNT`** and **`CRAWLER_MAX_COMMENTS_COUNT_SINGLENOTES`**.

When combined with higher sleep values, these limits create natural rate caps that prevent excessive API consumption. For instance, crawling 100 notes with a 5-second sleep interval generates a minimum total execution time of 500 seconds (approximately 8.3 minutes), keeping request volumes well within typical platform rate limits.

The logging system records every sleep event with the exact duration, enabling audit trails for debugging throttling behavior or verifying compliance with target service terms.

## Summary

- **`CRAWLER_MAX_SLEEP_SEC`** in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py) controls the global delay between HTTP requests
- The crawler uses `await asyncio.sleep(config.CRAWLER_MAX_SLEEP_SEC)` after each network call across all platforms
- Modify the constant directly in [`base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/base_config.py) or override it programmatically at runtime
- Sleep intervals work in conjunction with content limits (`CRAWLER_MAX_NOTES_COUNT`) to manage total throughput
- All sleep events are logged with duration details for operational transparency

## Frequently Asked Questions

### What is the default value for CRAWLER_MAX_SLEEP_SEC in MediaCrawler?

The default value varies by version, but the constant is defined at line 137 of [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py). Users should inspect this file directly, as the repository maintainers may adjust the baseline to reflect current platform tolerances. Typical defaults range between 1 and 3 seconds.

### Can I set different sleep intervals for different platforms?

While `CRAWLER_MAX_SLEEP_SEC` is global, you can implement platform-specific overrides by conditionally checking the crawler type before the sleep call. However, the stock MediaCrawler implementation uses a unified constant to ensure consistent behavior across all `media_platform/` modules. Forking the repository allows customization of individual platform files like [`zhihu/core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/zhihu/core.py) or [`tieba/core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tieba/core.py) to use distinct timing variables.

### How does CRAWLER_MAX_SLEEP_SEC interact with randomization or jitter?

The base implementation uses a fixed interval. To add randomization (recommended for avoiding detection patterns), wrap the sleep call with `random.uniform()`:

```python
import random
await asyncio.sleep(random.uniform(1, config.CRAWLER_MAX_SLEEP_SEC))

```

This modification requires editing the specific platform core files in `media_platform/`, as the stock configuration does not include built-in jitter support.

### Will reducing CRAWLER_MAX_SLEEP_SEC to 0 improve performance?

Setting the value to 0 eliminates intentional delays, significantly increasing request rates and substantially raising the risk of IP bans or account suspension. Most platforms implement automatic rate limiting that returns HTTP 429 (Too Many Requests) or temporary blocks when detecting rapid sequential requests. Maintain intervals of at least 1-2 seconds for production scraping tasks.