# Understanding Platform-Specific Features in NanmiCoder/MediaCrawler: Key Files and Architecture

> Explore platform-specific features in NanmiCoder/MediaCrawler. Discover key files like corepy, clientpy, and loginpy within isolated packages to understand architecture and customization.

- Repository: [程序员阿江-Relakkes/MediaCrawler](https://github.com/NanmiCoder/MediaCrawler)
- Tags: architecture
- Published: 2026-07-03

---

**The NanmiCoder/MediaCrawler repository organizes each social media platform into isolated packages under `media_platform/` containing five standard files—[`core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/core.py), [`client.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/client.py), [`login.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/login.py), [`field.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/field.py), and [`exception.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/exception.py)—that inherit from a shared abstract base in [`base/base_crawler.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/base/base_crawler.py) to handle authentication, API communication, and data extraction.**

MediaCrawler is a Python-based social media scraping framework that supports platforms including Weibo, Zhihu, Xiaohongshu, Douyin, Bilibili, Tieba, and Kuaishou through a modular plugin architecture. To understand how platform-specific features are implemented, you must examine the consistent file patterns within each `media_platform/<platform>/` directory and the shared abstractions that standardize their behavior. This guide identifies the essential source files that define how each platform handles browser automation, API requests, and data persistence.

## The Five-Layer Architecture Pattern

Each platform package follows an identical five-layer structure that separates concerns from authentication to data storage.

- **[`core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/core.py)**: Orchestrates the high-level workflow including searching, detail fetching, comment crawling, and media downloads. This file contains the concrete crawler class (e.g., `WeiboCrawler`) that inherits from `AbstractCrawler` and implements methods like `search()` and `get_specified_notes()`.

- **[`client.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/client.py)**: Wraps low-level HTTP and API interactions, handling headers, cookies, and platform-specific request logic using `httpx` or Playwright.

- **[`login.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/login.py)**: Manages authentication flows including QR-code scanning, cookie-based sessions, and mobile login methods.

- **[`field.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/field.py)**: Defines platform-specific constants, search-type enums (e.g., `SearchType`), and request parameter schemas.

- **[`exception.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/exception.py)**: Declares custom error types like `DataFetchError` that bubble up to the core logic for centralized handling.

## Abstract Base and Shared Infrastructure

The foundation of every platform implementation resides in [`base/base_crawler.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/base/base_crawler.py), which defines the **AbstractCrawler** interface. This abstract class mandates three core methods that all concrete crawlers must implement:

- `start()`: Entry point for the crawling workflow.
- `search()`: Handles query execution and result pagination.
- `launch_browser()`: Manages browser initialization using Chromium instances, often coordinated through [`tools/cdp_browser.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/cdp_browser.py) when `ENABLE_CDP_MODE` is active in the configuration.

By inheriting from this base, each platform crawler gains access to shared helper methods while implementing platform-specific logic for content extraction.

## Data Flow from Configuration to Persistence

Understanding the relationship between files requires tracing the execution flow through the system:

1. **Configuration → Crawler**: [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py) and platform-specific configs (e.g., [`config/weibo_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/weibo_config.py)) set switches like `ENABLE_CDP_MODE` and `WEIBO_SEARCH_TYPE`, which initialize the crawler behavior.

2. **Crawler → Client**: The [`core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/core.py) module instantiates a platform-specific client (e.g., `create_weibo_client`) with proper headers and authentication tokens.

3. **Client → API**: The client issues HTTP requests to platform endpoints and returns raw JSON structures.

4. **Core → Store**: Parsed data flows from [`core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/core.py) to platform-specific store functions (e.g., `store/weibo/update_weibo_note`) defined in [`store/__init__.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/__init__.py).

5. **Store → Persistence**: Store implementations in `store/<platform>/_store_impl.py` map dictionaries to ORM models in [`database/models.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/database/models.py) or write files directly via [`tools/async_file_writer.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/async_file_writer.py).

## Key Files by Functional Layer

### Core Infrastructure

- **[`base/base_crawler.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/base/base_crawler.py)**: Defines the `AbstractCrawler` contract and shared utilities used by all platform implementations.

### Weibo Implementation (Reference Platform)

The Weibo package exemplifies the standard structure:

- **[`media_platform/weibo/core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/weibo/core.py)**: Contains `WeiboCrawler` with complete workflow implementation including browser launch, search execution, and media handling.

- **[`media_platform/weibo/client.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/weibo/client.py)**: Low-level API wrapper for Weibo endpoints, handling GET requests for notes, comments, and image downloads.

- **[`media_platform/weibo/login.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/weibo/login.py)**: QR-code and cookie-based authentication flows.

- **[`media_platform/weibo/field.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/weibo/field.py)**: `SearchType` enum and request parameter definitions.

- **[`media_platform/weibo/exception.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/weibo/exception.py)**: Custom exceptions including `DataFetchError`.

### Other Platform Packages

Each platform mirrors the Weibo structure:

- **Zhihu**: [`media_platform/zhihu/core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/zhihu/core.py) and [`media_platform/zhihu/client.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/zhihu/client.py) handle answer and article extraction.
- **Xiaohongshu**: [`media_platform/xhs/core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/xhs/core.py) manages XHS-specific crawling logic.
- **Douyin**: [`media_platform/douyin/core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/douyin/core.py) for video and comment crawling.
- **Bilibili**: [`media_platform/bilibili/core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/bilibili/core.py) for user data and video extraction.
- **Tieba**: [`media_platform/tieba/core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/tieba/core.py) for post and comment crawling.
- **Kuaishou**: [`media_platform/kuaishou/core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/kuaishou/core.py) utilizing GraphQL queries.

### Storage Layer

- **[`store/__init__.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/__init__.py)**: Exposes platform-specific persistence functions like `update_weibo_note` and `update_zhihu_content`.

- **[`store/weibo/_store_impl.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/weibo/_store_impl.py)**: Weibo-specific storage logic for MongoDB, SQLite, and image handling.

- **[`store/zhihu/_store_impl.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/zhihu/_store_impl.py)**: Zhihu-specific persistence implementations.

### Configuration and Utilities

- **[`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py)**: Global switches including proxy settings, CDP mode, and concurrency limits.

- **[`tools/cdp_browser.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/cdp_browser.py)**: CDP-mode browser manager shared across all platforms when enabled.

- **[`tools/utils.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/utils.py)**: Helper functions for user-agent generation, logging, and cookie conversion.

## Practical Implementation Examples

The following examples demonstrate how to instantiate platform-specific crawlers using the abstract base and concrete implementations.

### Running the Weibo Crawler

This example mirrors the execution pattern found in [`api/main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/api/main.py), launching a search workflow programmatically:

```python

# example: run_weibo.py

import asyncio
from media_platform.weibo.core import WeiboCrawler
from config import config

async def main():
    # Configure search parameters

    config.CRAWLER_TYPE = "search"
    config.KEYWORDS = "AI,机器学习"
    
    # Initialize and start crawler

    await WeiboCrawler().start()

if __name__ == "__main__":
    asyncio.run(main())

```

**Execution flow**: `WeiboCrawler.start()` → `launch_browser` (or CDP) → `create_weibo_client` → `search()` → `weibo_store.update_weibo_note` → persistence.

### Implementing a Custom Crawler

To extend the framework with a new platform, inherit from the abstract base and implement the required methods:

```python
from base.base_crawler import AbstractCrawler

class CustomCrawler(AbstractCrawler):
    async def start(self):
        # Platform-specific entry logic

        pass
        
    async def search(self):
        # Search implementation

        pass
        
    async def launch_browser(self, chromium, proxy, ua, headless=True):
        # Browser initialization logic

        return await super().launch_browser(chromium, proxy, ua, headless)

# Concrete implementations like WeiboCrawler provide full working examples

# of this pattern in production use.

```

## Summary

- **MediaCrawler uses a plugin architecture** where each platform resides in `media_platform/<platform>/` with identical file structures.
- **Five standard files define each platform**: [`core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/core.py) (workflow), [`client.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/client.py) (API), [`login.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/login.py) (auth), [`field.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/field.py) (constants), and [`exception.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/exception.py) (errors).
- **All crawlers inherit from `AbstractCrawler`** in [`base/base_crawler.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/base/base_crawler.py), ensuring consistent interfaces for `start()`, `search()`, and `launch_browser()`.
- **Data flows through standardized stages**: Configuration → Crawler → Client → API → Store → Persistence, with platform-specific logic isolated at each layer.
- **Storage implementations are platform-specific** in `store/<platform>/_store_impl.py` to handle differing schemas (e.g., Weibo's `pics` vs. Zhihu's `question_id`).

## Frequently Asked Questions

### What is the purpose of [`core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/core.py) in each platform package?

The [`core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/core.py) file contains the concrete crawler class (e.g., `WeiboCrawler`) that orchestrates the entire extraction workflow. It inherits from `AbstractCrawler` and implements platform-specific methods for searching content, fetching details, downloading comments, and handling media files, while coordinating with the [`client.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/client.py) module for HTTP requests and the store layer for persistence.

### How does MediaCrawler handle different authentication methods across platforms?

Each platform implements its own [`login.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/login.py) module containing platform-specific authentication logic such as QR-code scanning, cookie-based sessions, or mobile login flows. The [`core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/core.py) module calls these login handlers during initialization before commencing data extraction, allowing Weibo, Zhihu, and other platforms to use completely different authentication mechanisms while maintaining a consistent interface.

### Where are platform-specific data schemas defined?

Platform-specific constants and enums are defined in each package's [`field.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/field.py) file, including search types and API parameter mappings. The actual data persistence schemas are handled in the store layer, specifically in `store/<platform>/_store_impl.py` files, which map the raw JSON structures to database models in [`database/models.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/database/models.py) or to file system outputs via [`tools/async_file_writer.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/async_file_writer.py).

### Can I add a new platform by copying an existing one?

Yes, the modular architecture enables rapid platform addition by creating a new directory under `media_platform/` with the five standard files ([`core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/core.py), [`client.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/client.py), [`login.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/login.py), [`field.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/field.py), [`exception.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/exception.py)) and inheriting from `AbstractCrawler` in [`base/base_crawler.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/base/base_crawler.py). You must also implement the store interface in `store/<new_platform>/_store_impl.py` and register the crawler in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py) to complete the integration.