# How to Contribute to the MediaCrawler Open-Source Project

> Learn how to contribute to the MediaCrawler open-source project. Fork the repo, set up your environment, and follow the modular architecture to add new features or fix bugs.

- Repository: [程序员阿江-Relakkes/MediaCrawler](https://github.com/NanmiCoder/MediaCrawler)
- Tags: getting-started
- Published: 2026-06-28

---

**Contributing to MediaCrawler involves forking the repository, setting up the **uv** environment, and following the modular architecture defined in [`main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/main.py) and [`base/base_crawler.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/base/base_crawler.py) to add features or fix bugs.**

MediaCrawler is a modular, multi-platform web-scraping framework built around **Playwright** that supports platforms like XiaoHongShu, Douyin, and Zhihu. Understanding how to contribute to the MediaCrawler project requires familiarity with its factory-based architecture and the development workflow using **uv** for dependency management. This guide walks you through the codebase structure, environment setup, and submission process based on the actual implementation in the `NanmiCoder/MediaCrawler` repository.

## Understanding the MediaCrawler Architecture

Before writing code, you must understand the seven-layer architecture that separates concerns between CLI parsing, configuration, abstraction, and platform-specific implementations.

### CLI and Entry Point

The **`CrawlerFactory`** in [`main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/main.py) (lines 50-66) serves as the central dispatch mechanism. It maps platform identifiers (e.g., `"xhs"`, `"dy"`, `"zhihu"`) to concrete crawler classes. When you add a new platform, you must register it in this factory dictionary to enable command-line invocation.

### Configuration Layer

Centralized settings live in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py). This file defines critical flags such as `ENABLE_CDP_MODE`, `SAVE_DATA_OPTION`, and database connection parameters. All platform-specific configurations inherit from or reference these base settings, ensuring consistent behavior across crawlers.

### Abstract Crawler Interface

All platform implementations must inherit from **`AbstractCrawler`** defined in [`base/base_crawler.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/base/base_crawler.py). This base class enforces a common lifecycle including `start()`, `stop()`, and optional `cdp_manager` handling. Your crawler must implement these abstract methods to integrate with the framework's async execution model.

### Platform Implementations

The `media_platform/` directory contains concrete implementations for each supported site:

- [`media_platform/xhs.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/xhs.py) (XiaoHongShu)
- [`media_platform/zhihu.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/zhihu.py) (Zhihu)
- [`media_platform/douyin.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/douyin.py) (Douyin)
- [`media_platform/bilibili.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/bilibili.py) (Bilibili)

Each file contains a class that extends `AbstractCrawler` and implements platform-specific login and scraping logic.

### Data Persistence

The `store/` directory contains storage backends. Key files include [`store/excel_store_base.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/excel_store_base.py) and platform-specific implementations like [`store/zhihu/_store_impl.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/zhihu/_store_impl.py) and [`store/bilibili/_store_impl.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/bilibili/_store_impl.py). These handle writing results to CSV, JSON, Excel, SQLite, or MySQL based on the `SAVE_DATA_OPTION` configuration.

### Testing Infrastructure

The `tests/` directory contains unit and integration tests (e.g., [`test_tieba_extractor.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/test_tieba_extractor.py), [`test_excel_store.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/test_excel_store.py)). The CI pipeline runs these tests automatically on every pull request using GitHub Actions.

## Setting Up Your Development Environment

MediaCrawler uses **uv** for fast, reliable Python dependency management rather than traditional pip.

### Fork and Clone

Start by creating your own fork on GitHub, then clone it locally:

```bash
git clone https://github.com/<your-username>/MediaCrawler.git
cd MediaCrawler

```

### Install Dependencies

Install uv if you haven't already, then sync the environment:

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
uv sync

```

This installs Playwright and all other dependencies specified in the lock file.

### Verify Baseline

Run the existing test suite to ensure your environment is healthy:

```bash
uv run pytest

```

All tests must pass before you begin making changes.

## Implementing Contributions

Create a feature branch for your work:

```bash
git checkout -b feature/add-new-platform

```

### Adding a New Platform

To add support for a new media site:

1. Create a new file in `media_platform/` (e.g., [`media_platform/example.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/example.py))
2. Implement a class inheriting from `AbstractCrawler`
3. Register it in [`main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/main.py) within the `CrawlerFactory.CRAWLERS` dictionary (lines 50-66)

Here is a minimal skeleton implementation:

```python

# File: media_platform/example.py

from base.base_crawler import AbstractCrawler

class ExampleCrawler(AbstractCrawler):
    async def start(self) -> None:
        print("[Example] Crawling started")
        await self.fetch_some_data()
        print("[Example] Crawling finished")

```

Register the crawler in [`main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/main.py):

```python

# Add to CrawlerFactory.CRAWLERS (lines 50-66)

CrawlerFactory.CRAWLERS["example"] = ExampleCrawler

```

Test your implementation locally:

```bash
uv run main.py --platform example --lt qrcode --type search

```

### Fixing Bugs and Adding Tests

For bug fixes, locate the relevant test file in `tests/` (e.g., [`test_excel_store.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/test_excel_store.py) for storage issues). Modify the implementation in `store/*_impl.py` or the relevant platform file, then ensure the test suite remains green. Add new test cases that cover your code paths using the `tmp_path` fixture for filesystem-related tests.

### Documentation Improvements

The project specifically encourages contributions to the **CDP mode** documentation. The file `docs/CDP模式使用指南.md` contains a "贡献" (contribute) section (lines 80-84) welcoming improvements to examples, error handling guides, and browser support documentation.

## Contribution Guidelines and Best Practices

### Code Style and Linting

Follow **PEP 8** standards. The repository uses `ruff` and `black` for linting and formatting. Run checks before committing:

```bash
uv run ruff .

```

Maintainers may require you to fix linting errors before merging.

### Type Hints and API Consistency

All public APIs use type hints. Maintain this consistency when adding new methods or classes. The `AbstractCrawler` base class provides the type signatures your implementation must follow.

### Testing Requirements

Every contribution must include appropriate test coverage. Unit tests should verify new logic, while integration tests validate platform-specific data extraction. Use existing tests in `tests/` as templates for your assertions.

### License Compliance

All contributions fall under the project's **NON-COMMERCIAL LEARNING LICENSE 1.1**. By submitting a pull request, you agree that your code will be distributed under these terms. Commercial use requires separate permission from the maintainers.

## Summary

- **Architecture**: Understand the `CrawlerFactory` in [`main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/main.py) (lines 50-66) and the `AbstractCrawler` base class before writing code.
- **Environment**: Use `uv sync` to install dependencies and `uv run pytest` to verify changes.
- **Implementation**: New platforms require a class in `media_platform/`, registration in [`main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/main.py), and corresponding tests.
- **Quality**: Follow PEP 8, maintain type hints, run `ruff`, and ensure all tests pass.
- **Documentation**: Update [`README.md`](https://github.com/NanmiCoder/MediaCrawler/blob/main/README.md) or contribute to `docs/CDP模式使用指南.md` for CDP mode improvements.

## Frequently Asked Questions

### What is the fastest way to set up the MediaCrawler development environment?

Install **uv** using the official installer script, then run `uv sync` in the project root. This command installs Playwright and all Python dependencies in a single step. Verify your setup by running `uv run pytest` to ensure the baseline tests pass.

### How do I add a new social media platform to MediaCrawler?

Create a Python file in `media_platform/` that inherits from `AbstractCrawler` (defined in [`base/base_crawler.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/base/base_crawler.py)), implement the required `start()` and `stop()` methods, then register your class in `CrawlerFactory.CRAWLERS` inside [`main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/main.py) (lines 50-66). Add corresponding tests in `tests/` and update the configuration documentation if your platform requires new settings.

### What testing is required before submitting a pull request?

You must run the full test suite using `uv run pytest` and ensure all tests pass. If you add new functionality, include unit tests that cover your code paths. For data storage changes, use the `tmp_path` fixture to test filesystem operations without side effects. The GitHub Actions CI will automatically run tests on your PR.

### Where should I document changes to CDP mode functionality?

The `docs/CDP模式使用指南.md` file contains a dedicated "贡献" section (lines 80-84) specifically for CDP mode contributions. Add examples, improve error handling instructions, or document additional browser support in this file. You may also update the main [`README.md`](https://github.com/NanmiCoder/MediaCrawler/blob/main/README.md) if your changes affect general usage patterns.