How to Configure Concurrent Crawling in MediaCrawler

MediaCrawler controls parallel crawling through the MAX_CONCURRENCY_NUM constant in config/base_config.py, which defaults to 1 and can be overridden via CLI or programmatically.

The NanmiCoder/MediaCrawler repository provides a flexible concurrency model that lets you tune the number of simultaneous HTTP requests across all supported platforms. By adjusting a single configuration value, you can balance crawling speed against API rate limits and system resources. This guide explains how to configure concurrent crawling using the actual source code implementation.

Understanding the Concurrency Model

MediaCrawler implements concurrency control using Python's asyncio.Semaphore. Every platform-specific crawler creates a semaphore instance from the global MAX_CONCURRENCY_NUM value, ensuring that simultaneous I/O tasks never exceed your configured limit.

According to the source code in media_platform/zhihu/core.py (line 216), the semaphore wraps asynchronous fetch operations:

# media_platform/zhihu/core.py

semaphore = asyncio.Semaphore(config.MAX_CONCURRENCY_NUM)

async def fetch_page(...):
    async with semaphore:  # limits concurrent fetches

        ...

This pattern applies consistently across all platform implementations in the repository.

Configuration Methods

You have three ways to set the concurrency level, depending on whether you want a permanent default, a temporary override, or programmatic control.

Method 1: Edit the Base Configuration File

The default concurrency value lives in config/base_config.py at line 105. By default, it is set to 1, meaning only one crawler runs at a time:

# config/base_config.py

MAX_CONCURRENCY_NUM = 1  # ← default concurrency

To permanently increase the default, modify this constant directly:

# config/base_config.py

MAX_CONCURRENCY_NUM = 8  # change from 1 to 8

This change affects all future runs, whether invoked via CLI, API, or test suites.

Method 2: Use the Command-Line Flag

For temporary overrides, use the --max_concurrency_num flag when invoking the crawler. The argument parser in cmd_arg/arg.py (lines 286-362) handles this input and updates the global config:

# cmd_arg/arg.py

max_concurrency_num: Annotated[
    int,
    typer.Option(
        "--max_concurrency_num",
        help="Maximum number of concurrent crawlers",
    ),
] = config.MAX_CONCURRENCY_NUM

# ...

config.MAX_CONCURRENCY_NUM = max_concurrency_num

Run the crawler with custom concurrency:

python -m MediaCrawler crawl zhihu --max_concurrency_num 5

Method 3: Programmatic Configuration

When integrating MediaCrawler into another Python application, set the constant before starting the crawl:

from MediaCrawler.cmd_arg import arg
from MediaCrawler import config

# Set the concurrency before starting any crawler

config.MAX_CONCURRENCY_NUM = 4
arg.run()  # your usual entry point

How Concurrency is Enforced in the Source Code

The enforcement mechanism relies on the semaphore pattern found in platform core files. In media_platform/zhihu/core.py at line 216, the crawler initializes a semaphore using the global config value:

semaphore = asyncio.Semaphore(config.MAX_CONCURRENCY_NUM)

Each asynchronous operation that performs I/O (such as fetch_page) acquires the semaphore before executing:

async with semaphore:
    # HTTP request logic here

    response = await client.get(url)

This ensures that no more than MAX_CONCURRENCY_NUM requests are in flight simultaneously, regardless of how many platforms you are crawling.

Practical Examples

Running with default (single-threaded) behavior:

python -m MediaCrawler crawl zhihu

Running with 5 concurrent crawlers via CLI:

python -m MediaCrawler crawl zhihu --max_concurrency_num 5

Setting permanent high concurrency in config/base_config.py:

# config/base_config.py

MAX_CONCURRENCY_NUM = 10

Key files involved in concurrency control:

• config/base_config.py – Holds the global MAX_CONCURRENCY_NUM default (line 105)
• cmd_arg/arg.py – Parses the --max_concurrency_num CLI option (lines 286-362)
• media_platform/zhihu/core.py – Applies the semaphore to limit concurrent fetches (line 216)

Summary

• MAX_CONCURRENCY_NUM in config/base_config.py controls the maximum number of simultaneous crawlers, defaulting to 1.
• Use the --max_concurrency_num CLI flag to override the default for a single execution without modifying source files.
• The underlying implementation uses asyncio.Semaphore in platform-specific core files to enforce the limit.
• Adjust this value to balance crawling speed against API rate limits and your system's network capacity.

Frequently Asked Questions

What is the default concurrency level in MediaCrawler?

The default concurrency level is 1, defined in config/base_config.py at line 105. This conservative default ensures that new users do not accidentally overwhelm target servers or trigger rate limits while testing the tool.

How does MediaCrawler limit concurrent requests?

MediaCrawler uses Python's asyncio.Semaphore primitive. Each platform crawler creates a semaphore initialized with MAX_CONCURRENCY_NUM and wraps HTTP requests in async with semaphore: blocks, as seen in media_platform/zhihu/core.py at line 216. This guarantees that no more than the configured number of requests execute simultaneously.

Can I set different concurrency levels for different platforms?

No, the current architecture uses a single global constant MAX_CONCURRENCY_NUM defined in config/base_config.py. All platform crawlers reference this same value when creating their semaphores. To use different limits for different platforms, you must run separate instances with different configuration values or modify the platform-specific code to use a different semaphore initialization.

What happens if I set MAX_CONCURRENCY_NUM too high?

Setting the value too high can overwhelm target servers, trigger IP-based rate limiting or bans, and exhaust local system resources (file descriptors, memory). The repository documentation recommends starting with low values (1-5) and gradually increasing based on the target platform's rate limit policies and your hardware capabilities.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}