# How to Configure Concurrent Crawling in MediaCrawler

> Learn how to configure concurrent crawling in MediaCrawler. Adjust the MAX_CONCURRENCY_NUM setting to speed up your crawling process efficiently.

- Repository: [程序员阿江-Relakkes/MediaCrawler](https://github.com/NanmiCoder/MediaCrawler)
- Tags: how-to-guide
- Published: 2026-06-27

---

**MediaCrawler controls parallel crawling through the `MAX_CONCURRENCY_NUM` constant in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py), which defaults to 1 and can be overridden via CLI or programmatically.**

The NanmiCoder/MediaCrawler repository provides a flexible concurrency model that lets you tune the number of simultaneous HTTP requests across all supported platforms. By adjusting a single configuration value, you can balance crawling speed against API rate limits and system resources. This guide explains how to configure concurrent crawling using the actual source code implementation.

## Understanding the Concurrency Model

MediaCrawler implements concurrency control using Python's **`asyncio.Semaphore`**. Every platform-specific crawler creates a semaphore instance from the global `MAX_CONCURRENCY_NUM` value, ensuring that simultaneous I/O tasks never exceed your configured limit.

According to the source code in [`media_platform/zhihu/core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/zhihu/core.py) (line 216), the semaphore wraps asynchronous fetch operations:

```python

# media_platform/zhihu/core.py

semaphore = asyncio.Semaphore(config.MAX_CONCURRENCY_NUM)

async def fetch_page(...):
    async with semaphore:  # limits concurrent fetches

        ...

```

This pattern applies consistently across all platform implementations in the repository.

## Configuration Methods

You have three ways to set the concurrency level, depending on whether you want a permanent default, a temporary override, or programmatic control.

### Method 1: Edit the Base Configuration File

The default concurrency value lives in **[`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py)** at line 105. By default, it is set to `1`, meaning only one crawler runs at a time:

```python

# config/base_config.py

MAX_CONCURRENCY_NUM = 1  # ← default concurrency

```

To permanently increase the default, modify this constant directly:

```python

# config/base_config.py

MAX_CONCURRENCY_NUM = 8  # change from 1 to 8

```

This change affects all future runs, whether invoked via CLI, API, or test suites.

### Method 2: Use the Command-Line Flag

For temporary overrides, use the **`--max_concurrency_num`** flag when invoking the crawler. The argument parser in [`cmd_arg/arg.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/cmd_arg/arg.py) (lines 286-362) handles this input and updates the global config:

```python

# cmd_arg/arg.py

max_concurrency_num: Annotated[
    int,
    typer.Option(
        "--max_concurrency_num",
        help="Maximum number of concurrent crawlers",
    ),
] = config.MAX_CONCURRENCY_NUM

# ...

config.MAX_CONCURRENCY_NUM = max_concurrency_num

```

Run the crawler with custom concurrency:

```bash
python -m MediaCrawler crawl zhihu --max_concurrency_num 5

```

### Method 3: Programmatic Configuration

When integrating MediaCrawler into another Python application, set the constant before starting the crawl:

```python
from MediaCrawler.cmd_arg import arg
from MediaCrawler import config

# Set the concurrency before starting any crawler

config.MAX_CONCURRENCY_NUM = 4
arg.run()  # your usual entry point

```

## How Concurrency is Enforced in the Source Code

The enforcement mechanism relies on the semaphore pattern found in platform core files. In [`media_platform/zhihu/core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/zhihu/core.py) at line 216, the crawler initializes a semaphore using the global config value:

```python
semaphore = asyncio.Semaphore(config.MAX_CONCURRENCY_NUM)

```

Each asynchronous operation that performs I/O (such as `fetch_page`) acquires the semaphore before executing:

```python
async with semaphore:
    # HTTP request logic here

    response = await client.get(url)

```

This ensures that no more than `MAX_CONCURRENCY_NUM` requests are in flight simultaneously, regardless of how many platforms you are crawling.

## Practical Examples

**Running with default (single-threaded) behavior:**

```bash
python -m MediaCrawler crawl zhihu

```

**Running with 5 concurrent crawlers via CLI:**

```bash
python -m MediaCrawler crawl zhihu --max_concurrency_num 5

```

**Setting permanent high concurrency in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py):**

```python

# config/base_config.py

MAX_CONCURRENCY_NUM = 10

```

**Key files involved in concurrency control:**

- [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py) – Holds the global `MAX_CONCURRENCY_NUM` default (line 105)
- [`cmd_arg/arg.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/cmd_arg/arg.py) – Parses the `--max_concurrency_num` CLI option (lines 286-362)
- [`media_platform/zhihu/core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/zhihu/core.py) – Applies the semaphore to limit concurrent fetches (line 216)

## Summary

- **MAX_CONCURRENCY_NUM** in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py) controls the maximum number of simultaneous crawlers, defaulting to `1`.
- Use the **`--max_concurrency_num`** CLI flag to override the default for a single execution without modifying source files.
- The underlying implementation uses **`asyncio.Semaphore`** in platform-specific core files to enforce the limit.
- Adjust this value to balance crawling speed against API rate limits and your system's network capacity.

## Frequently Asked Questions

### What is the default concurrency level in MediaCrawler?

The default concurrency level is **1**, defined in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py) at line 105. This conservative default ensures that new users do not accidentally overwhelm target servers or trigger rate limits while testing the tool.

### How does MediaCrawler limit concurrent requests?

MediaCrawler uses Python's **`asyncio.Semaphore`** primitive. Each platform crawler creates a semaphore initialized with `MAX_CONCURRENCY_NUM` and wraps HTTP requests in `async with semaphore:` blocks, as seen in [`media_platform/zhihu/core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/zhihu/core.py) at line 216. This guarantees that no more than the configured number of requests execute simultaneously.

### Can I set different concurrency levels for different platforms?

No, the current architecture uses a single global constant `MAX_CONCURRENCY_NUM` defined in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py). All platform crawlers reference this same value when creating their semaphores. To use different limits for different platforms, you must run separate instances with different configuration values or modify the platform-specific code to use a different semaphore initialization.

### What happens if I set MAX_CONCURRENCY_NUM too high?

Setting the value too high can overwhelm target servers, trigger IP-based rate limiting or bans, and exhaust local system resources (file descriptors, memory). The repository documentation recommends starting with low values (1-5) and gradually increasing based on the target platform's rate limit policies and your hardware capabilities.