# How to Extend MediaCrawler's Platform Support: A Complete Developer Guide

> Learn how to extend MediaCrawler's platform support by adding new platforms. This developer guide covers platform enumeration, configuration, store implementation, and factory registration. Boost your MediaCrawler integration t...

- Repository: [程序员阿江-Relakkes/MediaCrawler](https://github.com/NanmiCoder/MediaCrawler)
- Tags: how-to-guide
- Published: 2026-06-26

---

**To extend MediaCrawler's platform support, add a new member to PlatformEnum in [`cmd_arg/arg.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/cmd_arg/arg.py), create a platform-specific config module and package under `media_platform/`, implement the store layer in `store/<platform>/_store_impl.py`, and register the new platform in the factory map in [`main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/main.py).**

MediaCrawler is a popular open-source social media scraping framework maintained by NanmiCoder. Its modular architecture separates platform-specific logic from core crawling infrastructure, making it straightforward to extend MediaCrawler's platform support without modifying existing code. This guide walks through the exact file locations and integration points required to add a new social media platform to the crawler.

## Understanding the Plugin Architecture

MediaCrawler uses a clean, layered architecture that isolates platform specifics behind common interfaces. The system routes requests through seven distinct layers:

- **CLI / Option Parsing**: Converts the `--platform` flag into a concrete enum value via `PlatformEnum` in [`cmd_arg/arg.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/cmd_arg/arg.py) (lines 40-50)
- **Configuration**: Imports platform-specific settings through [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py)
- **Platform Package**: Implements crawling logic in `media_platform/<platform>/` including login, request signing, and data extraction
- **Store Layer**: Persists data via `store/<platform>/_store_impl.py` using `AsyncFileWriter` from [`tools/async_file_writer.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/async_file_writer.py)
- **Factory / Routing**: Maps enum values to concrete implementations in [`main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/main.py)
- **API Layer**: Exposes endpoints via [`api/routers/crawler.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/api/routers/crawler.py) (optional)
- **Tests**: Validates platform behavior in `tests/`

## Step-by-Step Guide to Extend MediaCrawler Platform Support

### Step 1: Register the Platform in the CLI Enum

Edit [`cmd_arg/arg.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/cmd_arg/arg.py) and add a new member to `PlatformEnum`. For example, to add a platform called "newsite":

```python
class PlatformEnum(str, Enum):
    XHS = "xhs"
    DOUYIN = "dy"
    # ... existing platforms ...

    NEWSITE = "newsite"  # New platform entry

```

### Step 2: Create Platform-Specific Configuration

Create a new config file at [`config/newsite_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/newsite_config.py) containing API URLs, headers, and signing parameters. Then import it in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py):

```python
from .newsite_config import *  # Add alongside other imports

```

### Step 3: Implement the Platform Package

Create a new directory at `media_platform/newsite/` with seven required files:

- [`login.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/login.py) - QR-code, phone, or cookie authentication using Playwright
- [`core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/core.py) - Low-level request helpers and signature generation
- [`client.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/client.py) - High-level interface exposing `search()`, `detail()`, and `creator()` methods
- [`help.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/help.py), [`field.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/field.py), [`exception.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/exception.py) - Data containers and error classes

The [`client.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/client.py) must expose a uniform API that the generic crawler expects.

### Step 4: Add the Store Implementation

Create [`store/newsite/_store_impl.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/newsite/_store_impl.py) by copying an existing implementation like [`store/zhihu/_store_impl.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/zhihu/_store_impl.py). Update the platform string and field logic:

```python
class NewsiteStore(BaseStore):
    def __init__(self):
        super().__init__()
        self.writer = AsyncFileWriter(platform="newsite", crawler_type=crawler_type_var.get())

    async def write_post(self, post):
        await self.writer.write_jsonl(post, "posts")

```

This automatically routes output to `data/newsite/`.

### Step 5: Wire the Factory Router

In [`main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/main.py), import the new package and extend the `PLATFORM_CLIENTS` dictionary:

```python
from media_platform import newsite as newsite_pkg

PLATFORM_CLIENTS = {
    PlatformEnum.XHS: xhs_pkg,
    PlatformEnum.DOUYIN: douyin_pkg,
    PlatformEnum.NEWSITE: newsite_pkg,  # New mapping

}

```

### Step 6: Expose via API (Optional)

If using the HTTP API, update [`api/routers/crawler.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/api/routers/crawler.py) to recognize the new platform value and forward requests to the appropriate client.

### Step 7: Write Unit Tests

Create [`tests/test_newsite_client.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tests/test_newsite_client.py) following patterns from existing tests like [`test_tieba_client_pagination.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/test_tieba_client_pagination.py). Mock HTTP responses and verify that the client returns correctly parsed objects.

## Platform Integration Code Examples

Below are the critical integration points for adding a "newsite" platform:

**CLI Enum Registration:**

```python

# cmd_arg/arg.py

class PlatformEnum(str, Enum):
    XHS = "xhs"
    DOUYIN = "dy"
    NEWSITE = "newsite"      # ← new entry

```

**Config Import:**

```python

# config/base_config.py

from .newsite_config import *   # ← add this line

```

**Client Skeleton:**

```python

# media_platform/newsite/client.py

class NewsiteClient:
    def __init__(self, login: NewsiteLogin, config: NewsiteConfig):
        self.login = login
        self.cfg = config

    async def search(self, keywords: str, page: int = 1):
        # Build request, sign if needed, fetch and parse

        pass

    async def detail(self, post_id: str):
        pass

    async def creator(self, creator_id: str):
        pass

```

**Store Implementation:**

```python

# store/newsite/_store_impl.py

class NewsiteStore(BaseStore):
    def __init__(self):
        super().__init__()
        self.writer = AsyncFileWriter(platform="newsite", crawler_type=crawler_type_var.get())

    async def write_post(self, post):
        await self.writer.write_jsonl(post, "posts")

```

**Factory Registration:**

```python

# main.py

from media_platform import newsite as newsite_pkg

PLATFORM_CLIENTS = {
    PlatformEnum.XHS: xhs_pkg,
    PlatformEnum.DOUYIN: douyin_pkg,
    PlatformEnum.NEWSITE: newsite_pkg,
}

```

## Key Files and Their Roles

- **[`cmd_arg/arg.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/cmd_arg/arg.py)** - Defines `PlatformEnum` (lines 40-50) that converts CLI flags to enum values
- **[`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py)** - Imports per-platform config modules and serves as the global configuration hub
- **`media_platform/<platform>/`** - Contains platform-specific crawling logic including [`login.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/login.py), [`core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/core.py), and [`client.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/client.py)
- **`store/<platform>/_store_impl.py`** - Handles data persistence and instantiates `AsyncFileWriter` from [`tools/async_file_writer.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/async_file_writer.py)
- **[`tools/async_file_writer.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/async_file_writer.py)** - Manages file writes under `data/<platform>/` directories
- **[`api/routers/crawler.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/api/routers/crawler.py)** - HTTP endpoints that forward requests to platform clients
- **`tests/`** - Unit test suites that should be duplicated for new platforms

## Summary

- MediaCrawler uses a seven-layer plugin architecture that isolates platform-specific code
- To extend MediaCrawler's platform support, modify `PlatformEnum` in [`cmd_arg/arg.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/cmd_arg/arg.py) and create the required package structure under `media_platform/`
- Implement the store layer in `store/<platform>/_store_impl.py` using `AsyncFileWriter` to ensure output routes to `data/<platform>/`
- Register the new platform in [`main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/main.py) by mapping the enum value to the package import
- The framework handles concurrency control, CSV/JSON export, and WebUI automatically once the integration is complete

## Frequently Asked Questions

### What is the minimum code required to add a new platform to MediaCrawler?

You must extend `PlatformEnum` in [`cmd_arg/arg.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/cmd_arg/arg.py), create a client class in `media_platform/<platform>/client.py` with `search()` and `detail()` methods, and add a corresponding entry to the factory map in [`main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/main.py). The store layer in `store/<platform>/_store_impl.py` is also required if you want to persist data using the built-in CSV/JSONL exporters.

### Does MediaCrawler require specific method signatures for platform clients?

Yes. According to the source code in `media_platform/<platform>/client.py`, the client must expose public methods like `search()`, `detail()`, and `creator()` that accept standardized parameters. The generic crawler in [`main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/main.py) invokes these methods polymorphically based on the platform enum value selected via the `--platform` CLI flag.

### Where does MediaCrawler store data for new platforms?

Data automatically writes to `data/<platform>/` subdirectories. This is handled by `AsyncFileWriter` in [`tools/async_file_writer.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/async_file_writer.py), which is instantiated in each platform's store implementation with the platform name parameter (e.g., `platform="newsite"`).

### Can I add platform support without modifying the API layer?

Yes. The API layer in [`api/routers/crawler.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/api/routers/crawler.py) is optional. If you only use the CLI entry point, you can skip API registration entirely. The CLI factory in [`main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/main.py) works independently of the HTTP API, though adding API support requires only a simple branch to recognize the new platform value and forward to the appropriate client.