# MediaCrawler Structure and Abstract Base Classes: A Deep Dive into the Modular Architecture

> Explore the MediaCrawler structure and abstract base classes. Understand its modular architecture, factory pattern, and how it enables rapid extension to new platforms.

- Repository: [程序员阿江-Relakkes/MediaCrawler](https://github.com/NanmiCoder/MediaCrawler)
- Tags: deep-dive
- Published: 2026-06-26

---

**The MediaCrawler project implements a layered architecture built on abstract base classes (`AbstractCrawler`, `AbstractLogin`, `AbstractStore`) that enforce consistent contracts across all platform implementations, enabling rapid extension to new social media platforms through a factory pattern and clean separation of concerns.**

The **NanmiCoder/MediaCrawler** repository is a modular web crawling framework targeting Chinese social media platforms including Weibo, Zhihu, TikTok (Douyin), and Xiaohongshu. Its architecture relies on a robust system of abstract base classes to standardize crawler behavior, storage mechanisms, and authentication flows across different platforms.


## Core Architecture Overview

The project organizes functionality into distinct layers that separate concerns between configuration, tooling, platform-specific logic, and data persistence.

| Layer | Purpose | Key Modules |
|------|----------|-------------|
| **Base** | Defines core contracts and interfaces | [`base/base_crawler.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/base/base_crawler.py) |
| **Config** | Centralized runtime settings (platform selection, proxy, headless mode) | [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py) |
| **Tools** | Utility helpers (Playwright launch, async file writers) | `tools/` (e.g., [`app_runner.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/app_runner.py)) |
| **Media Platform** | Concrete implementations per target platform | `media_platform/*/core.py` |
| **Factory** | Dynamic crawler selection based on input | [`main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/main.py) (`CrawlerFactory`) |
| **Store** | Persistence backends (CSV/JSON/Excel/SQL) | `store/` (e.g., [`excel_store_base.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/excel_store_base.py)) |

This structure ensures that every concrete crawler adheres to the same interface defined in [`base/base_crawler.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/base/base_crawler.py), allowing the [`main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/main.py) entry point to switch between platforms without changing its core orchestration logic.


## Abstract Base Classes in [`base/base_crawler.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/base/base_crawler.py)

The foundation of the MediaCrawler structure resides in [`base/base_crawler.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/base/base_crawler.py), which defines the contracts for crawlers, authentication handlers, storage backends, and API clients.

### `AbstractCrawler`

The `AbstractCrawler` class establishes the mandatory interface for all platform-specific crawlers. Every concrete implementation must override these methods:

```python
class AbstractCrawler(ABC):
    @abstractmethod
    async def start(self): ...

    @abstractmethod
    async def search(self): ...

    @abstractmethod
    async def launch_browser(self,
                             chromium: BrowserType,
                             playwright_proxy: Optional[Dict],
                             user_agent: Optional[str],
                             headless: bool = True) -> BrowserContext:
        ...

    async def launch_browser_with_cdp(self,
                                      playwright: Playwright,
                                      playwright_proxy: Optional[Dict],
                                      user_agent: Optional[str],
                                      headless: bool = True) -> BrowserContext:
        # Default fallback to launch_browser unless overridden

        return await self.launch_browser(playwright.chromium,
                                         playwright_proxy,
                                         user_agent,
                                         headless)

```

- **`start`** – Serves as the entry point for the crawler, handling initialization and navigation.
- **`search`** – Performs keyword-based crawling operations.
- **`launch_browser`** – Wraps Playwright browser initialization, allowing customization of proxy settings, user agents, and headless mode.
- **`launch_browser_with_cdp`** – Optional Chrome DevTools Protocol support that delegates to `launch_browser` by default.

### `AbstractLogin`

Authentication workflows are standardized through the `AbstractLogin` interface, ensuring that all platforms support multiple login methods:

```python
class AbstractLogin(ABC):
    @abstractmethod
    async def begin(self): ...

    @abstractmethod
    async def login_by_qrcode(self): ...

    @abstractmethod
    async def login_by_mobile(self): ...

    @abstractmethod
    async def login_by_cookies(self): ...

```

Platform-specific modules in `media_platform/*/login.py` implement these methods to support QR-code scanning, mobile number authentication, or cookie-based session restoration.

### `AbstractStore` and Media Variants

Data persistence is decoupled from crawling logic through the storage abstraction:

```python
class AbstractStore(ABC):
    @abstractmethod
    async def store_content(self, content_item: Dict): ...

    @abstractmethod
    async def store_comment(self, comment_item: Dict): ...

    @abstractmethod
    async def store_creator(self, creator: Dict): ...

class AbstractStoreImage(ABC):
    async def store_image(self, image_content_item: Dict): ...

class AbstractStoreVideo(ABC):
    async def store_video(self, video_content_item: Dict): ...

```

- **`AbstractStore`** – Mandatory for all crawlers; handles posts, comments, and creator metadata.
- **`AbstractStoreImage`** and **`AbstractStoreVideo`** – Optional extensions for platforms like Weibo that require media file handling.

### `AbstractApiClient`

HTTP interactions and cookie synchronization are abstracted through:

```python
class AbstractApiClient(ABC):
    @abstractmethod
    async def request(self, method, url, **kwargs): ...

    @abstractmethod
    async def update_cookies(self, browser_context: BrowserContext): ...

```

This interface supports both direct HTTP requests (via `httpx`) and cookie synchronization with Playwright browser contexts.


## Concrete Implementation Pattern

Each platform package contains a [`core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/core.py) file that subclasses `AbstractCrawler`. The Weibo implementation demonstrates this pattern:

```python

# media_platform/weibo/core.py

class WeiboCrawler(AbstractCrawler):
    async def start(self):
        # Initialize login, launch browser, then execute search

        ...

    async def search(self):
        # Platform-specific scraping logic using Playwright

        ...

    async def launch_browser(self, chromium, playwright_proxy, user_agent, headless=True):
        # Custom proxy handling for Weibo's anti-detection measures

        ...

```

All other platforms (`zhihu`, `xhs`, `douyin`, `kuaishou`, `bilibili`, `tieba`) follow this identical structure, ensuring that the public API (`start`, `search`, `launch_browser`) remains consistent across the entire suite.


## Factory Pattern for Dynamic Platform Selection

The entry point in [`main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/main.py) utilizes a factory class to map platform identifiers to concrete crawler implementations, decoupling the CLI from specific class names:

```python

# main.py

class CrawlerFactory:
    CRAWLERS = {
        "xhs": XiaoHongShuCrawler,
        "dy": DouYinCrawler,
        "ks": KuaishouCrawler,
        "bili": BilibiliCrawler,
        "wb": WeiboCrawler,
        "tieba": TieBaCrawler,
        "zhihu": ZhihuCrawler,
    }

    @staticmethod
    def create_crawler(platform: str) -> AbstractCrawler:
        crawler_class = CrawlerFactory.CRAWLERS.get(platform)
        if not crawler_class:
            raise ValueError(f"Unsupported platform: {platform}")
        return crawler_class()

```

This design allows the command-line interface in [`cmd_arg/arg.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/cmd_arg/arg.py) or the Web API in [`api/routers/crawler.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/api/routers/crawler.py) to instantiate crawlers using simple string identifiers like `"wb"` or `"zhihu"`.


## Practical Implementation Examples

### Running a Crawl from the Command Line

Execute a Weibo crawl using QR-code authentication:

```bash
uv run main.py --platform wb --lt qrcode --type search

```

Behind the scenes, the factory resolves the platform string:

```python

# Simplified flow from main.py

args = await cmd_arg.parse_cmd()
crawler = CrawlerFactory.create_crawler(config.PLATFORM)  # "wb" → WeiboCrawler

await crawler.start()                                     # Invokes WeiboCrawler.start()

```

### Adding a New Platform (e.g., "MySite")

1. Create the package structure `media_platform/mysite/` containing [`core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/core.py).
2. Implement the abstract methods:

```python

# media_platform/mysite/core.py

from base.base_crawler import AbstractCrawler

class MySiteCrawler(AbstractCrawler):
    async def start(self):
        await self.launch_browser(...)
        await self.search()

    async def search(self):
        # Implement MySite-specific scraping logic

        ...

    async def launch_browser(self, chromium, playwright_proxy, user_agent, headless=True):
        # Optional custom browser configuration

        return await super().launch_browser(chromium, playwright_proxy, user_agent, headless)

```

3. Register the crawler in [`main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/main.py):

```python
CrawlerFactory.CRAWLERS["mysite"] = MySiteCrawler

```

The new platform is immediately available via `uv run main.py --platform mysite ...`.

### Implementing a Custom Storage Backend

Subclass `AbstractStore` to persist data to Excel:

```python
from store.excel_store_base import ExcelStoreBase
from base.base_crawler import AbstractStore

class MyStore(AbstractStore):
    async def store_content(self, content_item: dict):
        await ExcelStoreBase.save_row(content_item)

# Usage within a crawler:

self.store = MyStore()
await self.store.store_content({"title": "...", "url": "..."})

```

The [`store/excel_store_base.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/excel_store_base.py) module provides the underlying Excel manipulation utilities.


## Key Source Files

| File | Role |
|------|------|
| [`base/base_crawler.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/base/base_crawler.py) | Defines all abstract base classes (`AbstractCrawler`, `AbstractLogin`, `AbstractStore`, `AbstractApiClient`). |
| [`main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/main.py) | Entry point containing `CrawlerFactory` and CLI orchestration. |
| [`media_platform/weibo/core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/weibo/core.py) | Reference implementation of `AbstractCrawler` for Weibo. |
| [`media_platform/zhihu/core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/zhihu/core.py) | Reference implementation for Zhihu. |
| [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py) | Global configuration (proxy, headless mode, data save options). |
| [`tools/app_runner.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/app_runner.py) | Async runtime helper with graceful shutdown handling. |
| [`store/excel_store_base.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/excel_store_base.py) | Example concrete storage backend. |
| [`cmd_arg/arg.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/cmd_arg/arg.py) | Command-line argument parser. |


## Summary

- **Abstract base classes** in [`base/base_crawler.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/base/base_crawler.py) define strict contracts for crawlers, login handlers, storage backends, and API clients.
- **Concrete implementations** in `media_platform/*/core.py` inherit from `AbstractCrawler` and implement platform-specific logic for `start`, `search`, and `launch_browser`.
- **Factory pattern** in [`main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/main.py) enables runtime platform selection via the `CrawlerFactory.CRAWLERS` mapping dictionary.
- **Pluggable storage** is achieved by implementing `AbstractStore` methods (`store_content`, `store_comment`, `store_creator`), with optional support for images and videos.
- **Consistent authentication** across platforms is enforced through the `AbstractLogin` interface requiring `login_by_qrcode`, `login_by_mobile`, and `login_by_cookies` implementations.


## Frequently Asked Questions

### What is the purpose of `AbstractCrawler` in the MediaCrawler structure?

`AbstractCrawler` defines the mandatory contract that every platform crawler must implement. According to the source code in [`base/base_crawler.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/base/base_crawler.py), it requires concrete classes to implement `start`, `search`, and `launch_browser` methods, ensuring consistent entry points across all platforms like Weibo, Zhihu, and Douyin.

### How does the factory pattern work in [`main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/main.py)?

The `CrawlerFactory` class maintains a static dictionary `CRAWLERS` that maps platform strings (e.g., `"wb"`, `"dy"`, `"zhihu"`) to concrete crawler classes. When `create_crawler(platform)` is called, it retrieves the corresponding class from the dictionary and instantiates it, returning an `AbstractCrawler` type that the main execution loop can use polymorphically.

### Can I implement custom storage without modifying the core framework files?

Yes. By subclassing `AbstractStore` (or `AbstractStoreImage`/`AbstractStoreVideo` for media) and implementing the required `store_content`, `store_comment`, and `store_creator` methods, you can create custom persistence logic. The crawler accepts any storage implementation that conforms to these interfaces, allowing you to place your custom class in a separate module without touching [`base/base_crawler.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/base/base_crawler.py).

### What methods must a new platform crawler implement to satisfy the abstract base class?

Any new crawler subclassing `AbstractCrawler` must implement three abstract methods: `start()` for initialization and workflow orchestration, `search()` for the actual data extraction logic, and `launch_browser()` for browser context creation. Additionally, if the platform requires authentication, you should create a corresponding login class that implements `AbstractLogin`'s `begin`, `login_by_qrcode`, `login_by_mobile`, and `login_by_cookies` methods.