How to Configure MediaCrawler for a Specific Platform: Complete Setup Guide

You configure MediaCrawler for a specific platform by creating a Python configuration file in config/<platform>_config.py that overrides the global defaults in config/base_config.py, then launching the crawler with the --platform argument to load the targeted settings.

MediaCrawler is a modular social media scraping framework that supports platforms like Xiaohongshu, Douyin, and Bilibili through a layered configuration system. To target a specific platform, you must understand how the tool resolves settings from base defaults to platform-specific overrides. This guide explains the exact file paths, configuration classes, and command-line arguments required to customize the crawler for any supported site.

Understanding the Configuration Hierarchy

MediaCrawler resolves its final runtime configuration through a strict three-tier hierarchy defined across the config/ directory. According to the source code, the system loads and merges settings in this order:

1. Base configuration (config/base_config.py) provides global defaults for headless mode, Chrome DevTools Protocol (CDP) settings, and data storage options.
2. Platform-specific configuration (config/<platform>_config.py) overrides defaults for the selected target (e.g., xhs_config.py for Xiaohongshu, dy_config.py for Douyin).
3. CLI arguments (cmd_arg/arg.py) and environment variables apply final runtime overrides before the crawler core (base/base_crawler.py) initializes the platform client.

The Config dataclass exposed in each module is imported by main.py and instantiated based on the --platform argument passed at runtime.

Step-by-Step Platform Configuration

1. Create the Platform Configuration File

Create a new file at config/<platform>_config.py following the lowercase naming convention used by existing platforms. For example, to configure Xiaohongshu support, you would edit config/xhs_config.py:

# config/xhs_config.py

from dataclasses import dataclass

@dataclass
class Config:
    PLATFORM: str = "xhs"
    LOGIN_TYPE: str = "qrcode"  # Options: qrcode, phone, cookie

    HEADLESS: bool = True
    PROXY: str = ""
    FETCH_COMMENTS: bool = True
    SAVE_DATA_OPTION: str = "json"

2. Define Authentication and Connection Parameters

Set the LOGIN_TYPE field to specify the authentication method. Valid options are qrcode, phone, or cookie as implemented in the platform client modules. Additionally, configure the PROXY string and HEADLESS boolean to control browser visibility and network routing for this specific platform without affecting global defaults.

3. Launch with the Platform Flag

Execute the crawler by specifying the platform code that matches your configuration file prefix:

uv run main.py --platform xhs --lt qrcode --type search

The --platform argument must match the filename prefix (e.g., xhs for xhs_config.py). The cmd_arg/arg.py module parses these arguments and merges them with the configuration dataclass before instantiating the client in media_platform/<platform>/client.py.

Runtime Overrides and Storage Options

You can override any configuration value at runtime without modifying the source files. The CLI supports several key parameters that interact with the configuration system:

• --platform: Selects the target platform configuration file.
• --lt or --login-type: Overrides the LOGIN_TYPE field.
• --type: Specifies crawler operation mode (search, detail, or creator).
• --save_data_option: Changes the storage backend (json, csv, excel, or db).
• --headless: Overrides the headless browser setting.

For example, to export Xiaohongshu data to Excel using a visible browser window:

uv run main.py --platform xhs --lt qrcode --type search --save_data_option excel --headless false

This triggers the ExcelStoreBase implementation in store/excel_store_base.py to write results to data/xhs/xhs_search_<timestamp>.xlsx.

Extending Configuration to New Platforms

To configure MediaCrawler for a platform not yet in the repository, implement the following file structure:

1. Configuration: Add config/<new_platform>_config.py with a Config dataclass defining LOGIN_TYPE, HEADLESS, and platform-specific endpoints.
2. Data Models: Define Pydantic schemas in model/m_<platform>.py (e.g., model/m_xhs.py) to validate scraped data structures.
3. Client Implementation: Create media_platform/<platform>/client.py implementing the login(), fetch_posts(), and fetch_comments() methods.
4. Registration: The loader automatically discovers configuration files ending with _config.py in the config/ directory, so no manual registry edit is required unless adding custom dispatch logic in media_platform/__init__.py.

Summary

• MediaCrawler resolves configuration through a hierarchy: config/base_config.py → config/<platform>_config.py → CLI arguments parsed by cmd_arg/arg.py.
• Create platform-specific settings in config/<platform>_config.py using the Config dataclass pattern.
• Control authentication methods via the LOGIN_TYPE field with options qrcode, phone, or cookie.
• Launch crawls using uv run main.py --platform <code> --lt <method> --type <mode>.
• Override storage backends at runtime with --save_data_option to target ExcelStoreBase or database configurations defined in config/db_config.py.

Frequently Asked Questions

What filename convention should I use for platform configuration files?

Name your file config/<platform>_config.py where <platform> is a lowercase code matching the --platform argument. For example, use config/dy_config.py for Douyin and invoke with --platform dy. The system automatically discovers and loads any file ending with _config.py in the config/ directory.

How do I switch between QR code and cookie authentication?

Set the LOGIN_TYPE field in your platform configuration file to "qrcode", "phone", or "cookie", or override at runtime using the --lt argument. For example: uv run main.py --platform xhs --lt cookie --type search.

Can I use a proxy for only one platform while keeping others direct?

Yes. Define the PROXY field in the specific platform configuration file (e.g., config/dy_config.py) rather than in base_config.py. The system loads only the active platform's configuration at runtime, so proxy settings in other platform files remain inactive.

Where does the configuration get merged with command-line arguments?

The merging logic resides in cmd_arg/arg.py, which parses CLI flags and overlays them onto the Config dataclass loaded from the platform-specific module. This occurs before the crawler core (base/base_crawler.py) initializes the platform client (media_platform/<platform>/client.py).

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}