# How to Customize MediaCrawler's Behavior: A Complete Configuration Guide

> Customize MediaCrawler behavior easily. Learn how to configure this powerful crawling framework using config values, command-line arguments, or environment variables without code changes.

- Repository: [程序员阿江-Relakkes/MediaCrawler](https://github.com/NanmiCoder/MediaCrawler)
- Tags: how-to-guide
- Published: 2026-06-28

---

**MediaCrawler is a modular, configurable crawling framework that allows you to customize its behavior through configuration values, command-line arguments, or environment variables without modifying any source code.**

MediaCrawler, maintained by NanmiCoder, is an open-source multi-platform content crawler designed for flexibility. The framework separates configuration from implementation, enabling you to customize the crawler's behavior for different social media platforms, authentication methods, and data storage formats. All runtime parameters are centralized in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py) and can be overridden via the CLI interface defined in [`cmd_arg/arg.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/cmd_arg/arg.py).

## Understanding the Configuration Architecture

The customization system relies on five interconnected components that determine how the crawler executes at runtime:

- **Global configuration** ([`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py)): Defines default values for platform selection, keywords, login types, proxy settings, CDP mode, and storage options.

- **CLI argument parser** ([`cmd_arg/arg.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/cmd_arg/arg.py)): Implements Typer-based command-line flags that override defaults at runtime.

- **CrawlerFactory** ([`main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/main.py)): Maps platform codes like `xhs`, `dy`, or `zhihu` to concrete crawler classes at lines 50-59.

- **AbstractCrawler** ([`base/base_crawler.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/base/base_crawler.py)): Defines the async interface (`start`, `search`, `launch_browser`) that all platform implementations must follow.

- **Store factories** (`store/<platform>/__init__.py`): Select data persistence backends based on the `SAVE_DATA_OPTION` configuration.

## Customization Methods

You can customize MediaCrawler through four primary interfaces without altering core logic:

### Configuration File Editing

Modify [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py) to set persistent defaults for your specific use case. This file contains all configuration variables as module-level constants.

### Command-Line Arguments

Override any configuration value at runtime using flags defined in [`cmd_arg/arg.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/cmd_arg/arg.py). All global config options have corresponding CLI flags.

### Environment Variables

For containerized deployments, set environment variables matching the configuration variable names to inject values at runtime.

### Programmatic Configuration

Import the config module in Python scripts and set attributes before invoking the main function for dynamic configuration.

## Key Customization Points

### 1. Platform Selection

Control which social media platform to crawl by setting `config.PLATFORM` or using the `--platform` flag. Supported codes include `xhs` (XiaoHongShu), `dy` (Douyin), `ks` (Kuaishou), `bili` (Bilibili), `wb` (Weibo), and `zhihu`. The `CrawlerFactory` in [`main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/main.py) resolves these strings to classes in `media_platform/<platform>/core.py`.

### 2. Crawling Mode

Set `config.CRAWLER_TYPE` to define the extraction strategy:

- **`search`**: Crawl content based on keyword queries.
- **`detail`**: Fetch specific content details by ID.
- **`creator`**: Crawl all content from specific user profiles.

Platform implementations like `ZhihuCrawler` read this value in their `start` method to determine whether to invoke `search`, `get_specified_notes`, or `get_creators_and_notes`.

### 3. Keywords and Pagination

Configure search scope using:

- **`config.KEYWORDS`**: Comma-separated string of search terms (e.g., `"machine learning,AI research"`).
- **`config.START_PAGE`**: Initial page number for pagination.
- **`config.CRAWLER_MAX_NOTES_COUNT`**: Maximum items to fetch per run.

The crawler splits keywords and iterates through them in platform-specific implementations like [`media_platform/zhihu/core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/zhihu/core.py).

### 4. Authentication Methods

Control login behavior via `config.LOGIN_TYPE` or `--lt`:

- **`qrcode`**: Scan QR code to authenticate.
- **`phone`**: Mobile phone number login.
- **`cookie`**: Use existing session cookies.

Platform-specific login classes (e.g., `ZhiHuLogin`, `XiaoHongShuLogin`) respect this setting during browser initialization.

### 5. Proxy Configuration

Enable IP rotation by setting `config.ENABLE_IP_PROXY = True` and selecting a provider:

- **`config.IP_PROXY_PROVIDER_NAME`**: Choose `kuaidaili`, `wandouhttp`, or `static`.
- **`config.STATIC_PROXY_URL`**: Set your proxy endpoint for static configuration.

The proxy pool is created via `create_ip_pool` and passed to Playwright/HTTPX in [`tools/httpx_util.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/httpx_util.py).

### 6. Browser Automation Mode

Switch between standard Playwright and Chrome DevTools Protocol:

- **`config.ENABLE_CDP_MODE`**: Enable CDP for connecting to existing Chrome instances.
- **`config.CDP_CONNECT_EXISTING`**: Connect to a running browser rather than launching new.
- **`config.CUSTOM_BROWSER_PATH`**: Specify a custom Chrome executable path.

CDP mode is implemented in [`tools/cdp_browser.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/cdp_browser.py) and helps bypass anti-scraping measures.

### 7. Concurrency and Throttling

Prevent rate limiting by adjusting:

- **`config.MAX_CONCURRENCY_NUM`**: Limit simultaneous asynchronous tasks.
- **`config.CRAWLER_MAX_SLEEP_SEC`**: Seconds to sleep between page loads.

### 8. Data Storage Options

Control output format via `config.SAVE_DATA_OPTION`:

- **`jsonl`**: JSON Lines format.
- **`csv`**: Comma-separated values.
- **`sqlite`**: SQLite database.
- **`mongo`**: MongoDB collection.
- **`excel`**: Excel spreadsheet.

The storage factory in `store/<platform>/__init__.py` (e.g., [`store/zhihu/__init__.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/zhihu/__init__.py)) instantiates the appropriate saver class. Redirect output directories using `config.SAVE_DATA_PATH`.

### 9. Optional Features

Toggle additional data extraction:

- **`config.ENABLE_GET_COMMENTS`**: Extract comment threads.
- **`config.ENABLE_GET_SUB_COMMENTS`**: Extract reply comments.
- **`config.ENABLE_GET_MEIDAS`**: Download media files.
- **`config.ENABLE_GET_WORDCLOUD`**: Generate word clouds from content.
- **`config.CUSTOM_WORDS`**: Supply custom vocabulary for word cloud generation.

## Practical Implementation Examples

Basic execution using defaults:

```bash
python -m main

```

Override platform, keywords, and storage format:

```bash
python -m main --platform zhihu \
    --keywords "machine learning,AI research" \
    --type search \
    --save_data_option jsonl \
    --headless

```

Use CDP mode with existing Chrome instance:

```bash
python -m main --platform bili \
    --type detail \
    --enable_cdp_mode \
    --cdp_connect_existing \
    --headless=False

```

Enable static proxy and limit concurrency:

```bash
python -m main --platform ks \
    --static_proxy_url http://myproxy:3128 \
    --max_concurrency 3 \
    --type creator

```

SQLite persistence with word cloud generation:

```bash
python -m main --platform wb \
    --save_data_option sqlite \
    --enable_get_wordcloud \
    --type search

```

Programmatic configuration in Python:

```python
import config
from main import main
import asyncio

# Customize the crawler's behavior programmatically

config.PLATFORM = "dy"
config.KEYWORDS = "golang,devops"
config.CRAWLER_TYPE = "detail"
config.HEADLESS = False
config.ENABLE_IP_PROXY = True
config.IP_PROXY_PROVIDER_NAME = "static"
config.STATIC_PROXY_URL = "http://user:pass@proxy.example.com:8080"

asyncio.run(main())

```

## Summary

- **MediaCrawler** exposes all behavioral controls through [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py) and command-line arguments defined in [`cmd_arg/arg.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/cmd_arg/arg.py).
- The **CrawlerFactory** in [`main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/main.py) dynamically instantiates platform-specific crawlers based on the `PLATFORM` configuration value.
- Storage backends are selected via factory patterns in `store/<platform>/__init__.py` based on `SAVE_DATA_OPTION`.
- **CDP mode** in [`tools/cdp_browser.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/cdp_browser.py) provides enhanced anti-detection capabilities for complex scraping scenarios.
- Runtime customization requires no source code modifications when using CLI flags, environment variables, or programmatic configuration.

## Frequently Asked Questions

### Can I customize the crawler's behavior without modifying the source code?

Yes, MediaCrawler is designed to be fully configurable through external mechanisms. You can adjust all parameters via command-line arguments, environment variables, or by editing [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py) without touching the core crawling logic in `media_platform/` or `base/` directories.

### How do I switch between different social media platforms?

Set the `PLATFORM` configuration value to a supported platform code such as `xhs` for XiaoHongShu, `dy` for Douyin, `zhihu` for Zhihu, or `bili` for Bilibili. The `CrawlerFactory` class in [`main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/main.py) automatically resolves this string to the appropriate crawler class in `media_platform/<platform>/core.py`.

### What is the difference between standard Playwright mode and CDP mode?

Standard Playwright mode launches a fresh browser instance controlled by Playwright's API, while CDP (Chrome DevTools Protocol) mode connects to an existing Chrome installation or remote browser instance. Enable CDP mode by setting `ENABLE_CDP_MODE=True` for better anti-detection capabilities and to reuse existing browser sessions with persistent logins.

### How do I change where scraped data is saved?

Modify the `SAVE_DATA_OPTION` configuration value to select between `jsonl`, `csv`, `sqlite`, `mongo`, or `excel` formats. The storage factory in `store/<platform>/__init__.py` reads this value and instantiates the appropriate saver class. You can also redirect output directories using `SAVE_DATA_PATH` to organize data by project or date.