How to Extend MediaCrawler's Platform Support: A Complete Developer Guide

To extend MediaCrawler's platform support, add a new member to PlatformEnum in cmd_arg/arg.py, create a platform-specific config module and package under media_platform/, implement the store layer in store/<platform>/_store_impl.py, and register the new platform in the factory map in main.py.

MediaCrawler is a popular open-source social media scraping framework maintained by NanmiCoder. Its modular architecture separates platform-specific logic from core crawling infrastructure, making it straightforward to extend MediaCrawler's platform support without modifying existing code. This guide walks through the exact file locations and integration points required to add a new social media platform to the crawler.

Understanding the Plugin Architecture

MediaCrawler uses a clean, layered architecture that isolates platform specifics behind common interfaces. The system routes requests through seven distinct layers:

• CLI / Option Parsing: Converts the --platform flag into a concrete enum value via PlatformEnum in cmd_arg/arg.py (lines 40-50)
• Configuration: Imports platform-specific settings through config/base_config.py
• Platform Package: Implements crawling logic in media_platform/<platform>/ including login, request signing, and data extraction
• Store Layer: Persists data via store/<platform>/_store_impl.py using AsyncFileWriter from tools/async_file_writer.py
• Factory / Routing: Maps enum values to concrete implementations in main.py
• API Layer: Exposes endpoints via api/routers/crawler.py (optional)
• Tests: Validates platform behavior in tests/

Step-by-Step Guide to Extend MediaCrawler Platform Support

Step 1: Register the Platform in the CLI Enum

Edit cmd_arg/arg.py and add a new member to PlatformEnum. For example, to add a platform called "newsite":

class PlatformEnum(str, Enum):
    XHS = "xhs"
    DOUYIN = "dy"
    # ... existing platforms ...

    NEWSITE = "newsite"  # New platform entry

Step 2: Create Platform-Specific Configuration

Create a new config file at config/newsite_config.py containing API URLs, headers, and signing parameters. Then import it in config/base_config.py:

from .newsite_config import *  # Add alongside other imports

Step 3: Implement the Platform Package

Create a new directory at media_platform/newsite/ with seven required files:

• login.py - QR-code, phone, or cookie authentication using Playwright
• core.py - Low-level request helpers and signature generation
• client.py - High-level interface exposing search(), detail(), and creator() methods
• help.py, field.py, exception.py - Data containers and error classes

The client.py must expose a uniform API that the generic crawler expects.

Step 4: Add the Store Implementation

Create store/newsite/_store_impl.py by copying an existing implementation like store/zhihu/_store_impl.py. Update the platform string and field logic:

class NewsiteStore(BaseStore):
    def __init__(self):
        super().__init__()
        self.writer = AsyncFileWriter(platform="newsite", crawler_type=crawler_type_var.get())

    async def write_post(self, post):
        await self.writer.write_jsonl(post, "posts")

This automatically routes output to data/newsite/.

Step 5: Wire the Factory Router

In main.py, import the new package and extend the PLATFORM_CLIENTS dictionary:

from media_platform import newsite as newsite_pkg

PLATFORM_CLIENTS = {
    PlatformEnum.XHS: xhs_pkg,
    PlatformEnum.DOUYIN: douyin_pkg,
    PlatformEnum.NEWSITE: newsite_pkg,  # New mapping

}

Step 6: Expose via API (Optional)

If using the HTTP API, update api/routers/crawler.py to recognize the new platform value and forward requests to the appropriate client.

Step 7: Write Unit Tests

Create tests/test_newsite_client.py following patterns from existing tests like test_tieba_client_pagination.py. Mock HTTP responses and verify that the client returns correctly parsed objects.

Platform Integration Code Examples

Below are the critical integration points for adding a "newsite" platform:

CLI Enum Registration:

# cmd_arg/arg.py

class PlatformEnum(str, Enum):
    XHS = "xhs"
    DOUYIN = "dy"
    NEWSITE = "newsite"      # ← new entry

Config Import:

# config/base_config.py

from .newsite_config import *   # ← add this line

Client Skeleton:

# media_platform/newsite/client.py

class NewsiteClient:
    def __init__(self, login: NewsiteLogin, config: NewsiteConfig):
        self.login = login
        self.cfg = config

    async def search(self, keywords: str, page: int = 1):
        # Build request, sign if needed, fetch and parse

        pass

    async def detail(self, post_id: str):
        pass

    async def creator(self, creator_id: str):
        pass

Store Implementation:

# store/newsite/_store_impl.py

class NewsiteStore(BaseStore):
    def __init__(self):
        super().__init__()
        self.writer = AsyncFileWriter(platform="newsite", crawler_type=crawler_type_var.get())

    async def write_post(self, post):
        await self.writer.write_jsonl(post, "posts")

Factory Registration:

# main.py

from media_platform import newsite as newsite_pkg

PLATFORM_CLIENTS = {
    PlatformEnum.XHS: xhs_pkg,
    PlatformEnum.DOUYIN: douyin_pkg,
    PlatformEnum.NEWSITE: newsite_pkg,
}

Key Files and Their Roles

• cmd_arg/arg.py - Defines PlatformEnum (lines 40-50) that converts CLI flags to enum values
• config/base_config.py - Imports per-platform config modules and serves as the global configuration hub
• media_platform/<platform>/ - Contains platform-specific crawling logic including login.py, core.py, and client.py
• store/<platform>/_store_impl.py - Handles data persistence and instantiates AsyncFileWriter from tools/async_file_writer.py
• tools/async_file_writer.py - Manages file writes under data/<platform>/ directories
• api/routers/crawler.py - HTTP endpoints that forward requests to platform clients
• tests/ - Unit test suites that should be duplicated for new platforms

Summary

• MediaCrawler uses a seven-layer plugin architecture that isolates platform-specific code
• To extend MediaCrawler's platform support, modify PlatformEnum in cmd_arg/arg.py and create the required package structure under media_platform/
• Implement the store layer in store/<platform>/_store_impl.py using AsyncFileWriter to ensure output routes to data/<platform>/
• Register the new platform in main.py by mapping the enum value to the package import
• The framework handles concurrency control, CSV/JSON export, and WebUI automatically once the integration is complete

Frequently Asked Questions

What is the minimum code required to add a new platform to MediaCrawler?

You must extend PlatformEnum in cmd_arg/arg.py, create a client class in media_platform/<platform>/client.py with search() and detail() methods, and add a corresponding entry to the factory map in main.py. The store layer in store/<platform>/_store_impl.py is also required if you want to persist data using the built-in CSV/JSONL exporters.

Does MediaCrawler require specific method signatures for platform clients?

Yes. According to the source code in media_platform/<platform>/client.py, the client must expose public methods like search(), detail(), and creator() that accept standardized parameters. The generic crawler in main.py invokes these methods polymorphically based on the platform enum value selected via the --platform CLI flag.

Where does MediaCrawler store data for new platforms?

Data automatically writes to data/<platform>/ subdirectories. This is handled by AsyncFileWriter in tools/async_file_writer.py, which is instantiated in each platform's store implementation with the platform name parameter (e.g., platform="newsite").

Can I add platform support without modifying the API layer?

Yes. The API layer in api/routers/crawler.py is optional. If you only use the CLI entry point, you can skip API registration entirely. The CLI factory in main.py works independently of the HTTP API, though adding API support requires only a simple branch to recognize the new platform value and forward to the appropriate client.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}