MediaCrawler Structure and Abstract Base Classes: A Deep Dive into the Modular Architecture

The MediaCrawler project implements a layered architecture built on abstract base classes (AbstractCrawler, AbstractLogin, AbstractStore) that enforce consistent contracts across all platform implementations, enabling rapid extension to new social media platforms through a factory pattern and clean separation of concerns.

The NanmiCoder/MediaCrawler repository is a modular web crawling framework targeting Chinese social media platforms including Weibo, Zhihu, TikTok (Douyin), and Xiaohongshu. Its architecture relies on a robust system of abstract base classes to standardize crawler behavior, storage mechanisms, and authentication flows across different platforms.

Core Architecture Overview

The project organizes functionality into distinct layers that separate concerns between configuration, tooling, platform-specific logic, and data persistence.

Layer	Purpose	Key Modules
Base	Defines core contracts and interfaces	base/base_crawler.py
Config	Centralized runtime settings (platform selection, proxy, headless mode)	config/base_config.py
Tools	Utility helpers (Playwright launch, async file writers)	tools/ (e.g., app_runner.py)
Media Platform	Concrete implementations per target platform	media_platform/*/core.py
Factory	Dynamic crawler selection based on input	main.py (CrawlerFactory)
Store	Persistence backends (CSV/JSON/Excel/SQL)	store/ (e.g., excel_store_base.py)

This structure ensures that every concrete crawler adheres to the same interface defined in base/base_crawler.py, allowing the main.py entry point to switch between platforms without changing its core orchestration logic.

Abstract Base Classes in base/base_crawler.py

The foundation of the MediaCrawler structure resides in base/base_crawler.py, which defines the contracts for crawlers, authentication handlers, storage backends, and API clients.

AbstractCrawler

The AbstractCrawler class establishes the mandatory interface for all platform-specific crawlers. Every concrete implementation must override these methods:

class AbstractCrawler(ABC):
    @abstractmethod
    async def start(self): ...

    @abstractmethod
    async def search(self): ...

    @abstractmethod
    async def launch_browser(self,
                             chromium: BrowserType,
                             playwright_proxy: Optional[Dict],
                             user_agent: Optional[str],
                             headless: bool = True) -> BrowserContext:
        ...

    async def launch_browser_with_cdp(self,
                                      playwright: Playwright,
                                      playwright_proxy: Optional[Dict],
                                      user_agent: Optional[str],
                                      headless: bool = True) -> BrowserContext:
        # Default fallback to launch_browser unless overridden

        return await self.launch_browser(playwright.chromium,
                                         playwright_proxy,
                                         user_agent,
                                         headless)

• start – Serves as the entry point for the crawler, handling initialization and navigation.
• search – Performs keyword-based crawling operations.
• launch_browser – Wraps Playwright browser initialization, allowing customization of proxy settings, user agents, and headless mode.
• launch_browser_with_cdp – Optional Chrome DevTools Protocol support that delegates to launch_browser by default.

AbstractLogin

Authentication workflows are standardized through the AbstractLogin interface, ensuring that all platforms support multiple login methods:

class AbstractLogin(ABC):
    @abstractmethod
    async def begin(self): ...

    @abstractmethod
    async def login_by_qrcode(self): ...

    @abstractmethod
    async def login_by_mobile(self): ...

    @abstractmethod
    async def login_by_cookies(self): ...

Platform-specific modules in media_platform/*/login.py implement these methods to support QR-code scanning, mobile number authentication, or cookie-based session restoration.

AbstractStore and Media Variants

Data persistence is decoupled from crawling logic through the storage abstraction:

class AbstractStore(ABC):
    @abstractmethod
    async def store_content(self, content_item: Dict): ...

    @abstractmethod
    async def store_comment(self, comment_item: Dict): ...

    @abstractmethod
    async def store_creator(self, creator: Dict): ...

class AbstractStoreImage(ABC):
    async def store_image(self, image_content_item: Dict): ...

class AbstractStoreVideo(ABC):
    async def store_video(self, video_content_item: Dict): ...

• AbstractStore – Mandatory for all crawlers; handles posts, comments, and creator metadata.
• AbstractStoreImage and AbstractStoreVideo – Optional extensions for platforms like Weibo that require media file handling.

AbstractApiClient

HTTP interactions and cookie synchronization are abstracted through:

class AbstractApiClient(ABC):
    @abstractmethod
    async def request(self, method, url, **kwargs): ...

    @abstractmethod
    async def update_cookies(self, browser_context: BrowserContext): ...

This interface supports both direct HTTP requests (via httpx) and cookie synchronization with Playwright browser contexts.

Concrete Implementation Pattern

Each platform package contains a core.py file that subclasses AbstractCrawler. The Weibo implementation demonstrates this pattern:

# media_platform/weibo/core.py

class WeiboCrawler(AbstractCrawler):
    async def start(self):
        # Initialize login, launch browser, then execute search

        ...

    async def search(self):
        # Platform-specific scraping logic using Playwright

        ...

    async def launch_browser(self, chromium, playwright_proxy, user_agent, headless=True):
        # Custom proxy handling for Weibo's anti-detection measures

        ...

All other platforms (zhihu, xhs, douyin, kuaishou, bilibili, tieba) follow this identical structure, ensuring that the public API (start, search, launch_browser) remains consistent across the entire suite.

Factory Pattern for Dynamic Platform Selection

The entry point in main.py utilizes a factory class to map platform identifiers to concrete crawler implementations, decoupling the CLI from specific class names:

# main.py

class CrawlerFactory:
    CRAWLERS = {
        "xhs": XiaoHongShuCrawler,
        "dy": DouYinCrawler,
        "ks": KuaishouCrawler,
        "bili": BilibiliCrawler,
        "wb": WeiboCrawler,
        "tieba": TieBaCrawler,
        "zhihu": ZhihuCrawler,
    }

    @staticmethod
    def create_crawler(platform: str) -> AbstractCrawler:
        crawler_class = CrawlerFactory.CRAWLERS.get(platform)
        if not crawler_class:
            raise ValueError(f"Unsupported platform: {platform}")
        return crawler_class()

This design allows the command-line interface in cmd_arg/arg.py or the Web API in api/routers/crawler.py to instantiate crawlers using simple string identifiers like "wb" or "zhihu".

Practical Implementation Examples

Running a Crawl from the Command Line

Execute a Weibo crawl using QR-code authentication:

uv run main.py --platform wb --lt qrcode --type search

Behind the scenes, the factory resolves the platform string:

# Simplified flow from main.py

args = await cmd_arg.parse_cmd()
crawler = CrawlerFactory.create_crawler(config.PLATFORM)  # "wb" → WeiboCrawler

await crawler.start()                                     # Invokes WeiboCrawler.start()

Adding a New Platform (e.g., "MySite")

1. Create the package structure media_platform/mysite/ containing core.py.
2. Implement the abstract methods:

# media_platform/mysite/core.py

from base.base_crawler import AbstractCrawler

class MySiteCrawler(AbstractCrawler):
    async def start(self):
        await self.launch_browser(...)
        await self.search()

    async def search(self):
        # Implement MySite-specific scraping logic

        ...

    async def launch_browser(self, chromium, playwright_proxy, user_agent, headless=True):
        # Optional custom browser configuration

        return await super().launch_browser(chromium, playwright_proxy, user_agent, headless)

3. Register the crawler in main.py:

CrawlerFactory.CRAWLERS["mysite"] = MySiteCrawler

The new platform is immediately available via uv run main.py --platform mysite ....

Implementing a Custom Storage Backend

Subclass AbstractStore to persist data to Excel:

from store.excel_store_base import ExcelStoreBase
from base.base_crawler import AbstractStore

class MyStore(AbstractStore):
    async def store_content(self, content_item: dict):
        await ExcelStoreBase.save_row(content_item)

# Usage within a crawler:

self.store = MyStore()
await self.store.store_content({"title": "...", "url": "..."})

The store/excel_store_base.py module provides the underlying Excel manipulation utilities.

Key Source Files

File	Role
base/base_crawler.py	Defines all abstract base classes (AbstractCrawler, AbstractLogin, AbstractStore, AbstractApiClient).
main.py	Entry point containing CrawlerFactory and CLI orchestration.
media_platform/weibo/core.py	Reference implementation of AbstractCrawler for Weibo.
media_platform/zhihu/core.py	Reference implementation for Zhihu.
config/base_config.py	Global configuration (proxy, headless mode, data save options).
tools/app_runner.py	Async runtime helper with graceful shutdown handling.
store/excel_store_base.py	Example concrete storage backend.
cmd_arg/arg.py	Command-line argument parser.

Summary

• Abstract base classes in base/base_crawler.py define strict contracts for crawlers, login handlers, storage backends, and API clients.
• Concrete implementations in media_platform/*/core.py inherit from AbstractCrawler and implement platform-specific logic for start, search, and launch_browser.
• Factory pattern in main.py enables runtime platform selection via the CrawlerFactory.CRAWLERS mapping dictionary.
• Pluggable storage is achieved by implementing AbstractStore methods (store_content, store_comment, store_creator), with optional support for images and videos.
• Consistent authentication across platforms is enforced through the AbstractLogin interface requiring login_by_qrcode, login_by_mobile, and login_by_cookies implementations.

Frequently Asked Questions

What is the purpose of AbstractCrawler in the MediaCrawler structure?

AbstractCrawler defines the mandatory contract that every platform crawler must implement. According to the source code in base/base_crawler.py, it requires concrete classes to implement start, search, and launch_browser methods, ensuring consistent entry points across all platforms like Weibo, Zhihu, and Douyin.

How does the factory pattern work in main.py?

The CrawlerFactory class maintains a static dictionary CRAWLERS that maps platform strings (e.g., "wb", "dy", "zhihu") to concrete crawler classes. When create_crawler(platform) is called, it retrieves the corresponding class from the dictionary and instantiates it, returning an AbstractCrawler type that the main execution loop can use polymorphically.

Can I implement custom storage without modifying the core framework files?

Yes. By subclassing AbstractStore (or AbstractStoreImage/AbstractStoreVideo for media) and implementing the required store_content, store_comment, and store_creator methods, you can create custom persistence logic. The crawler accepts any storage implementation that conforms to these interfaces, allowing you to place your custom class in a separate module without touching base/base_crawler.py.

What methods must a new platform crawler implement to satisfy the abstract base class?

Any new crawler subclassing AbstractCrawler must implement three abstract methods: start() for initialization and workflow orchestration, search() for the actual data extraction logic, and launch_browser() for browser context creation. Additionally, if the platform requires authentication, you should create a corresponding login class that implements AbstractLogin's begin, login_by_qrcode, login_by_mobile, and login_by_cookies methods.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}