How to Contribute to the MediaCrawler Open-Source Project

Contributing to MediaCrawler involves forking the repository, setting up the uv environment, and following the modular architecture defined in main.py and base/base_crawler.py to add features or fix bugs.

MediaCrawler is a modular, multi-platform web-scraping framework built around Playwright that supports platforms like XiaoHongShu, Douyin, and Zhihu. Understanding how to contribute to the MediaCrawler project requires familiarity with its factory-based architecture and the development workflow using uv for dependency management. This guide walks you through the codebase structure, environment setup, and submission process based on the actual implementation in the NanmiCoder/MediaCrawler repository.

Understanding the MediaCrawler Architecture

Before writing code, you must understand the seven-layer architecture that separates concerns between CLI parsing, configuration, abstraction, and platform-specific implementations.

CLI and Entry Point

The CrawlerFactory in main.py (lines 50-66) serves as the central dispatch mechanism. It maps platform identifiers (e.g., "xhs", "dy", "zhihu") to concrete crawler classes. When you add a new platform, you must register it in this factory dictionary to enable command-line invocation.

Configuration Layer

Centralized settings live in config/base_config.py. This file defines critical flags such as ENABLE_CDP_MODE, SAVE_DATA_OPTION, and database connection parameters. All platform-specific configurations inherit from or reference these base settings, ensuring consistent behavior across crawlers.

Abstract Crawler Interface

All platform implementations must inherit from AbstractCrawler defined in base/base_crawler.py. This base class enforces a common lifecycle including start(), stop(), and optional cdp_manager handling. Your crawler must implement these abstract methods to integrate with the framework's async execution model.

Platform Implementations

The media_platform/ directory contains concrete implementations for each supported site:

• media_platform/xhs.py (XiaoHongShu)
• media_platform/zhihu.py (Zhihu)
• media_platform/douyin.py (Douyin)
• media_platform/bilibili.py (Bilibili)

Each file contains a class that extends AbstractCrawler and implements platform-specific login and scraping logic.

Data Persistence

The store/ directory contains storage backends. Key files include store/excel_store_base.py and platform-specific implementations like store/zhihu/_store_impl.py and store/bilibili/_store_impl.py. These handle writing results to CSV, JSON, Excel, SQLite, or MySQL based on the SAVE_DATA_OPTION configuration.

Testing Infrastructure

The tests/ directory contains unit and integration tests (e.g., test_tieba_extractor.py, test_excel_store.py). The CI pipeline runs these tests automatically on every pull request using GitHub Actions.

Setting Up Your Development Environment

MediaCrawler uses uv for fast, reliable Python dependency management rather than traditional pip.

Fork and Clone

Start by creating your own fork on GitHub, then clone it locally:

git clone https://github.com/<your-username>/MediaCrawler.git
cd MediaCrawler

Install Dependencies

Install uv if you haven't already, then sync the environment:

curl -LsSf https://astral.sh/uv/install.sh | sh
uv sync

This installs Playwright and all other dependencies specified in the lock file.

Verify Baseline

Run the existing test suite to ensure your environment is healthy:

uv run pytest

All tests must pass before you begin making changes.

Implementing Contributions

Create a feature branch for your work:

git checkout -b feature/add-new-platform

Adding a New Platform

To add support for a new media site:

1. Create a new file in media_platform/ (e.g., media_platform/example.py)
2. Implement a class inheriting from AbstractCrawler
3. Register it in main.py within the CrawlerFactory.CRAWLERS dictionary (lines 50-66)

Here is a minimal skeleton implementation:

# File: media_platform/example.py

from base.base_crawler import AbstractCrawler

class ExampleCrawler(AbstractCrawler):
    async def start(self) -> None:
        print("[Example] Crawling started")
        await self.fetch_some_data()
        print("[Example] Crawling finished")

Register the crawler in main.py:

# Add to CrawlerFactory.CRAWLERS (lines 50-66)

CrawlerFactory.CRAWLERS["example"] = ExampleCrawler

Test your implementation locally:

uv run main.py --platform example --lt qrcode --type search

Fixing Bugs and Adding Tests

For bug fixes, locate the relevant test file in tests/ (e.g., test_excel_store.py for storage issues). Modify the implementation in store/*_impl.py or the relevant platform file, then ensure the test suite remains green. Add new test cases that cover your code paths using the tmp_path fixture for filesystem-related tests.

Documentation Improvements

The project specifically encourages contributions to the CDP mode documentation. The file docs/CDP模式使用指南.md contains a "贡献" (contribute) section (lines 80-84) welcoming improvements to examples, error handling guides, and browser support documentation.

Contribution Guidelines and Best Practices

Code Style and Linting

Follow PEP 8 standards. The repository uses ruff and black for linting and formatting. Run checks before committing:

uv run ruff .

Maintainers may require you to fix linting errors before merging.

Type Hints and API Consistency

All public APIs use type hints. Maintain this consistency when adding new methods or classes. The AbstractCrawler base class provides the type signatures your implementation must follow.

Testing Requirements

Every contribution must include appropriate test coverage. Unit tests should verify new logic, while integration tests validate platform-specific data extraction. Use existing tests in tests/ as templates for your assertions.

License Compliance

All contributions fall under the project's NON-COMMERCIAL LEARNING LICENSE 1.1. By submitting a pull request, you agree that your code will be distributed under these terms. Commercial use requires separate permission from the maintainers.

Summary

• Architecture: Understand the CrawlerFactory in main.py (lines 50-66) and the AbstractCrawler base class before writing code.
• Environment: Use uv sync to install dependencies and uv run pytest to verify changes.
• Implementation: New platforms require a class in media_platform/, registration in main.py, and corresponding tests.
• Quality: Follow PEP 8, maintain type hints, run ruff, and ensure all tests pass.
• Documentation: Update README.md or contribute to docs/CDP模式使用指南.md for CDP mode improvements.

Frequently Asked Questions

What is the fastest way to set up the MediaCrawler development environment?

Install uv using the official installer script, then run uv sync in the project root. This command installs Playwright and all Python dependencies in a single step. Verify your setup by running uv run pytest to ensure the baseline tests pass.

How do I add a new social media platform to MediaCrawler?

Create a Python file in media_platform/ that inherits from AbstractCrawler (defined in base/base_crawler.py), implement the required start() and stop() methods, then register your class in CrawlerFactory.CRAWLERS inside main.py (lines 50-66). Add corresponding tests in tests/ and update the configuration documentation if your platform requires new settings.

What testing is required before submitting a pull request?

You must run the full test suite using uv run pytest and ensure all tests pass. If you add new functionality, include unit tests that cover your code paths. For data storage changes, use the tmp_path fixture to test filesystem operations without side effects. The GitHub Actions CI will automatically run tests on your PR.

Where should I document changes to CDP mode functionality?

The docs/CDP模式使用指南.md file contains a dedicated "贡献" section (lines 80-84) specifically for CDP mode contributions. Add examples, improve error handling instructions, or document additional browser support in this file. You may also update the main README.md if your changes affect general usage patterns.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}