How to Install MediaCrawler: Complete Setup Guide for the NanmiCoder Repository

Install uv and Node.js, clone the NanmiCoder/MediaCrawler repository, run uv sync to install Python dependencies, and optionally execute uv run playwright install if you disable CDP mode.

MediaCrawler is a multi-platform media content scraper that leverages modern Python tooling and browser automation. According to the NanmiCoder/MediaCrawler source code, the project uses uv for fast dependency resolution and supports both Chrome DevTools Protocol (CDP) and standard Playwright modes for browser automation. This guide covers the complete installation process based on the repository's README.md and configuration files.

Prerequisites for MediaCrawler Installation

Before cloning the repository, install three core dependencies that power the crawler's Python environment and platform-specific scripts.

Install uv (Python Package Manager)

The project relies on uv for deterministic environment management. Install uv from the official documentation, then verify it is available in your PATH:

uv --version

Install Node.js

Node.js version 16 or higher is required for platform-specific signing scripts, particularly for sites like Douyin and Zhihu. Download and install Node.js from the official website.

Install Google Chrome (Optional)

For CDP mode (the default configuration in config/base_config.py), install Google Chrome version 144 or higher. CDP mode connects to an existing Chrome instance rather than launching a new browser, eliminating the need for separate Playwright browser downloads.

Installing MediaCrawler Step-by-Step

Clone the Repository

Use git to clone the source code and navigate into the project directory:

git clone https://github.com/NanmiCoder/MediaCrawler.git
cd MediaCrawler

Sync Python Dependencies with uv

Run uv sync to create a virtual environment (under .venv/ by default) and install all packages defined in pyproject.toml:

uv sync

This command resolves dependencies including internal modules such as cache/, tools/, and the various platform models like model/m_xiaohongshu.py.

Install Playwright Browser Binaries (Conditional)

If you set ENABLE_CDP_MODE = False in config/base_config.py, you must download the browser binaries that Playwright requires:

uv run playwright install

When CDP mode is enabled (the default), the crawler uses tools/cdp_browser.py to connect to your existing Chrome instance via the Chrome DevTools Protocol, making this step optional.

Verify Your Installation

Confirm the CLI is wired correctly by displaying the help text:

uv run main.py --help

You should see usage information for the entry point defined in main.py, which parses arguments and dispatches to platform-specific crawlers.

Alternative Installation Method (venv + pip)

If uv is unavailable in your environment, use the legacy workflow documented in the repository. Create a virtual environment and install from requirements.txt:

python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

pip install -r requirements.txt
playwright install
python main.py --help

This mirrors the uv workflow but uses pip for environments where uv cannot be installed.

Summary

• Install prerequisites: uv, Node.js (≥16), and optionally Google Chrome (≥144) for CDP mode.
• Clone and sync: Execute git clone followed by uv sync to install Python dependencies from pyproject.toml.
• Handle browser binaries: Run uv run playwright install only when ENABLE_CDP_MODE is set to False in config/base_config.py.
• Verify: Use uv run main.py --help to confirm successful installation.
• Alternative: Use venv + pip with requirements.txt if uv is unavailable.

Frequently Asked Questions

Do I need to install Playwright if I use CDP mode?

No. When ENABLE_CDP_MODE = True in config/base_config.py, MediaCrawler uses tools/cdp_browser.py to connect to an existing Chrome instance via the Chrome DevTools Protocol. You only need to run uv run playwright install when CDP mode is disabled, which triggers tools/browser_launcher.py to manage Playwright browsers.

What is the difference between uv and pip for installing MediaCrawler?

uv is the recommended Python package manager for this repository, providing fast dependency resolution and deterministic environments through pyproject.toml. pip with requirements.txt remains supported as a fallback workflow, but uv handles the virtual environment creation and synchronization automatically via the uv sync command.

Why is Node.js required for a Python project?

Node.js is required because certain platform-specific signing scripts (particularly for Douyin and Zhihu) rely on JavaScript execution. These scripts are invoked during the crawling process to generate valid request signatures and authentication tokens.

How do I switch between CDP mode and standard Playwright mode?

Edit config/base_config.py and change the ENABLE_CDP_MODE value. Set it to True to use CDP mode (connecting to an existing Chrome instance), or False to use standard Playwright mode where the crawler launches its own headless browser instances. If you disable CDP mode, ensure you have run uv run playwright install to download the necessary browser binaries.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}