How to Use MediaCrawler with Docker: Complete Containerization Guide

You can containerize MediaCrawler by building a custom Dockerfile that installs Python 3.11, Node.js, the uv package manager, and the project's locked dependencies, then executes the crawler via uv run main.py with your desired platform flags.

MediaCrawler is a versatile Python-based social media scraping tool maintained by NanmiCoder that supports platforms like XiaoHongShu, Douyin, and Bilibili. Since the repository does not ship an official Docker image, understanding how to use MediaCrawler with Docker requires constructing a custom container that handles its Python dependencies, Node.js signature generation requirements, and browser automation configuration. This guide provides a production-ready Dockerfile and execution patterns based on the actual project structure and dependency management workflow found in the source code.

Prerequisites for Containerizing MediaCrawler

Base Image and System Dependencies

MediaCrawler requires Python 3.11 and Node.js (>=16) for platform-specific signature generation. The build process also needs curl to fetch the uv installer, plus git and standard build tools (gcc, g++, make) for compiling any native extensions.

Understanding the uv Package Manager

The project uses uv as its package manager, relying on the uv.lock file in the repository root for deterministic dependency resolution. This approach replaces traditional pip requirements and ensures reproducible builds across different environments.

Building the MediaCrawler Docker Image

Create a multi-stage Dockerfile that optimizes the final image size while installing all necessary tooling:

# ---------- Dockerfile ----------

# 1️⃣ Use a slim Python base

FROM python:3.11-slim AS builder

# 2️⃣ Install OS utilities

RUN apt-get update && apt-get install -y --no-install-recommends \
    curl git ca-certificates make gcc g++ && \
    rm -rf /var/lib/apt/lists/*

# 3️⃣ Install Node.js (>=16) – using the official nodesource script

RUN curl -fsSL https://deb.nodesource.com/setup_20.x | bash - && \
    apt-get install -y nodejs && \
    rm -rf /var/lib/apt/lists/*

# 4️⃣ Install uv (the fast Python package manager)

RUN curl -LsSf https://install.uv.dev | python3 -

# 5️⃣ Set workdir and copy the project

WORKDIR /app
COPY . .

# 6️⃣ Cache the uv lock file and install Python deps

#    The lock file (`uv.lock`) is already present in the repo.

RUN /root/.local/bin/uv sync --no-cache

# 7️⃣ (Optional) Install Playwright browsers if you plan to use Playwright mode

#    Comment out if you always run in CDP mode.

# RUN /root/.local/bin/uv run playwright install

# 8️⃣ Runtime stage – keep it lightweight

FROM python:3.11-slim
WORKDIR /app

# Copy the virtual environment from the builder stage

COPY --from=builder /app/.venv /app/.venv
COPY --from=builder /app /app

# Ensure the venv is on PATH

ENV PATH="/app/.venv/bin:$PATH"

# Default command runs the crawler; you can override it at `docker run`

CMD ["uv", "run", "main.py", "--help"]

Build the image with:

docker build -t mediacrawler:latest .

Running MediaCrawler in Docker Containers

Basic Verification Commands

Verify the installation by checking the help output before running actual crawl jobs:

docker run --rm mediacrawler:latest

Platform-Specific Crawling Examples

Execute platform-specific crawls by passing arguments to main.py. For example, to search XiaoHongShu with QR code authentication:

docker run --rm mediacrawler:latest \
  uv run main.py --platform xhs --lt qrcode --type search

For fetching Bilibili post details:

docker run --rm mediacrawler:latest \
  uv run main.py --platform bilibili --lt qrcode --type detail

Persistent Data Storage with Volume Mounts

The crawler outputs data to CSV, JSON, Excel, SQLite, or MySQL as configured in config/base_config.py and documented in docs/data_storage_guide.md. Mount a host directory to preserve outputs:

docker run -v $(pwd)/output:/app/output --rm mediacrawler:latest \
  uv run main.py --platform weibo --lt qrcode --type search

This writes files to /app/output inside the container, mapping to ./output on your host.

Configuration and Browser Automation Modes

CDP vs Playwright Configuration

MediaCrawler defaults to CDP (Chrome DevTools Protocol) mode, which connects to an existing Chrome instance rather than downloading Playwright browsers. This is controlled by the ENABLE_CDP_MODE setting in config/base_config.py.

If you disable CDP mode and require Playwright, uncomment the browser installation line in the Dockerfile:

RUN /root/.local/bin/uv run playwright install

The CDP approach keeps container images smaller and avoids the overhead of managing Playwright browser binaries inside the container.

Summary

• MediaCrawler requires Python 3.11, Node.js, and the uv package manager for proper containerization.
• Use a multi-stage Dockerfile to keep the final image lean while installing system dependencies and syncing Python packages via uv sync.
• The entrypoint main.py accepts platform-specific flags like --platform xhs or --platform dy to control crawling behavior.
• Mount volumes to persist output data from the container's filesystem to your host machine.
• Choose between CDP mode (default, lightweight) or Playwright mode (requires additional browser installation) based on your config/base_config.py settings.

Frequently Asked Questions

Does MediaCrawler provide an official Docker image?

No, the NanmiCoder/MediaCrawler repository does not include an official Dockerfile or pre-built image. You must build your own container using the Dockerfile pattern provided in this guide, which installs the uv package manager, Node.js, and project dependencies from the uv.lock file.

Why does the container need Node.js if this is a Python project?

MediaCrawler uses Node.js for generating platform-specific signatures required by certain social media APIs, such as Douyin and TikTok. The JavaScript execution environment is essential for the crawler's authentication and request signing mechanisms to function correctly.

How do I save crawler output to my host machine?

Use Docker volume mounts to map the container's output directory to a host folder. For example, -v $(pwd)/output:/app/output ensures that CSV, JSON, or database files generated according to your config/base_config.py settings are persisted on your local filesystem after the container exits.

Can I run MediaCrawler in Docker without installing Playwright browsers?

Yes. By default, MediaCrawler uses CDP (Chrome DevTools Protocol) mode, which connects to an external Chrome instance rather than managing browsers internally. As long as ENABLE_CDP_MODE remains True in config/base_config.py, you can omit the playwright install step from your Dockerfile, resulting in a smaller container image.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}