How to Customize MediaCrawler's Behavior: A Complete Configuration Guide

MediaCrawler is a modular, configurable crawling framework that allows you to customize its behavior through configuration values, command-line arguments, or environment variables without modifying any source code.

MediaCrawler, maintained by NanmiCoder, is an open-source multi-platform content crawler designed for flexibility. The framework separates configuration from implementation, enabling you to customize the crawler's behavior for different social media platforms, authentication methods, and data storage formats. All runtime parameters are centralized in config/base_config.py and can be overridden via the CLI interface defined in cmd_arg/arg.py.

Understanding the Configuration Architecture

The customization system relies on five interconnected components that determine how the crawler executes at runtime:

• Global configuration (config/base_config.py): Defines default values for platform selection, keywords, login types, proxy settings, CDP mode, and storage options.

• CLI argument parser (cmd_arg/arg.py): Implements Typer-based command-line flags that override defaults at runtime.

• CrawlerFactory (main.py): Maps platform codes like xhs, dy, or zhihu to concrete crawler classes at lines 50-59.

• AbstractCrawler (base/base_crawler.py): Defines the async interface (start, search, launch_browser) that all platform implementations must follow.

• Store factories (store/<platform>/__init__.py): Select data persistence backends based on the SAVE_DATA_OPTION configuration.

Customization Methods

You can customize MediaCrawler through four primary interfaces without altering core logic:

Configuration File Editing

Modify config/base_config.py to set persistent defaults for your specific use case. This file contains all configuration variables as module-level constants.

Command-Line Arguments

Override any configuration value at runtime using flags defined in cmd_arg/arg.py. All global config options have corresponding CLI flags.

Environment Variables

For containerized deployments, set environment variables matching the configuration variable names to inject values at runtime.

Programmatic Configuration

Import the config module in Python scripts and set attributes before invoking the main function for dynamic configuration.

Key Customization Points

1. Platform Selection

Control which social media platform to crawl by setting config.PLATFORM or using the --platform flag. Supported codes include xhs (XiaoHongShu), dy (Douyin), ks (Kuaishou), bili (Bilibili), wb (Weibo), and zhihu. The CrawlerFactory in main.py resolves these strings to classes in media_platform/<platform>/core.py.

2. Crawling Mode

Set config.CRAWLER_TYPE to define the extraction strategy:

• search: Crawl content based on keyword queries.
• detail: Fetch specific content details by ID.
• creator: Crawl all content from specific user profiles.

Platform implementations like ZhihuCrawler read this value in their start method to determine whether to invoke search, get_specified_notes, or get_creators_and_notes.

3. Keywords and Pagination

Configure search scope using:

• config.KEYWORDS: Comma-separated string of search terms (e.g., "machine learning,AI research").
• config.START_PAGE: Initial page number for pagination.
• config.CRAWLER_MAX_NOTES_COUNT: Maximum items to fetch per run.

The crawler splits keywords and iterates through them in platform-specific implementations like media_platform/zhihu/core.py.

4. Authentication Methods

Control login behavior via config.LOGIN_TYPE or --lt:

• qrcode: Scan QR code to authenticate.
• phone: Mobile phone number login.
• cookie: Use existing session cookies.

Platform-specific login classes (e.g., ZhiHuLogin, XiaoHongShuLogin) respect this setting during browser initialization.

5. Proxy Configuration

Enable IP rotation by setting config.ENABLE_IP_PROXY = True and selecting a provider:

• config.IP_PROXY_PROVIDER_NAME: Choose kuaidaili, wandouhttp, or static.
• config.STATIC_PROXY_URL: Set your proxy endpoint for static configuration.

The proxy pool is created via create_ip_pool and passed to Playwright/HTTPX in tools/httpx_util.py.

6. Browser Automation Mode

Switch between standard Playwright and Chrome DevTools Protocol:

• config.ENABLE_CDP_MODE: Enable CDP for connecting to existing Chrome instances.
• config.CDP_CONNECT_EXISTING: Connect to a running browser rather than launching new.
• config.CUSTOM_BROWSER_PATH: Specify a custom Chrome executable path.

CDP mode is implemented in tools/cdp_browser.py and helps bypass anti-scraping measures.

7. Concurrency and Throttling

Prevent rate limiting by adjusting:

• config.MAX_CONCURRENCY_NUM: Limit simultaneous asynchronous tasks.
• config.CRAWLER_MAX_SLEEP_SEC: Seconds to sleep between page loads.

8. Data Storage Options

Control output format via config.SAVE_DATA_OPTION:

• jsonl: JSON Lines format.
• csv: Comma-separated values.
• sqlite: SQLite database.
• mongo: MongoDB collection.
• excel: Excel spreadsheet.

The storage factory in store/<platform>/__init__.py (e.g., store/zhihu/__init__.py) instantiates the appropriate saver class. Redirect output directories using config.SAVE_DATA_PATH.

9. Optional Features

Toggle additional data extraction:

• config.ENABLE_GET_COMMENTS: Extract comment threads.
• config.ENABLE_GET_SUB_COMMENTS: Extract reply comments.
• config.ENABLE_GET_MEIDAS: Download media files.
• config.ENABLE_GET_WORDCLOUD: Generate word clouds from content.
• config.CUSTOM_WORDS: Supply custom vocabulary for word cloud generation.

Practical Implementation Examples

Basic execution using defaults:

python -m main

Override platform, keywords, and storage format:

python -m main --platform zhihu \
    --keywords "machine learning,AI research" \
    --type search \
    --save_data_option jsonl \
    --headless

Use CDP mode with existing Chrome instance:

python -m main --platform bili \
    --type detail \
    --enable_cdp_mode \
    --cdp_connect_existing \
    --headless=False

Enable static proxy and limit concurrency:

python -m main --platform ks \
    --static_proxy_url http://myproxy:3128 \
    --max_concurrency 3 \
    --type creator

SQLite persistence with word cloud generation:

python -m main --platform wb \
    --save_data_option sqlite \
    --enable_get_wordcloud \
    --type search

Programmatic configuration in Python:

import config
from main import main
import asyncio

# Customize the crawler's behavior programmatically

config.PLATFORM = "dy"
config.KEYWORDS = "golang,devops"
config.CRAWLER_TYPE = "detail"
config.HEADLESS = False
config.ENABLE_IP_PROXY = True
config.IP_PROXY_PROVIDER_NAME = "static"
config.STATIC_PROXY_URL = "http://user:pass@proxy.example.com:8080"

asyncio.run(main())

Summary

• MediaCrawler exposes all behavioral controls through config/base_config.py and command-line arguments defined in cmd_arg/arg.py.
• The CrawlerFactory in main.py dynamically instantiates platform-specific crawlers based on the PLATFORM configuration value.
• Storage backends are selected via factory patterns in store/<platform>/__init__.py based on SAVE_DATA_OPTION.
• CDP mode in tools/cdp_browser.py provides enhanced anti-detection capabilities for complex scraping scenarios.
• Runtime customization requires no source code modifications when using CLI flags, environment variables, or programmatic configuration.

Frequently Asked Questions

Can I customize the crawler's behavior without modifying the source code?

Yes, MediaCrawler is designed to be fully configurable through external mechanisms. You can adjust all parameters via command-line arguments, environment variables, or by editing config/base_config.py without touching the core crawling logic in media_platform/ or base/ directories.

How do I switch between different social media platforms?

Set the PLATFORM configuration value to a supported platform code such as xhs for XiaoHongShu, dy for Douyin, zhihu for Zhihu, or bili for Bilibili. The CrawlerFactory class in main.py automatically resolves this string to the appropriate crawler class in media_platform/<platform>/core.py.

What is the difference between standard Playwright mode and CDP mode?

Standard Playwright mode launches a fresh browser instance controlled by Playwright's API, while CDP (Chrome DevTools Protocol) mode connects to an existing Chrome installation or remote browser instance. Enable CDP mode by setting ENABLE_CDP_MODE=True for better anti-detection capabilities and to reuse existing browser sessions with persistent logins.

How do I change where scraped data is saved?

Modify the SAVE_DATA_OPTION configuration value to select between jsonl, csv, sqlite, mongo, or excel formats. The storage factory in store/<platform>/__init__.py reads this value and instantiates the appropriate saver class. You can also redirect output directories using SAVE_DATA_PATH to organize data by project or date.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}