How to Select Platform-Specific Crawlers at Runtime Using Factory Pattern in MediaCrawler

The MediaCrawler repository uses a centralized CrawlerFactory in factory/crawler_factory.py to map platform identifiers like "xhs" or "zhihu" to concrete crawler classes, enabling runtime selection via the --platform CLI argument.

MediaCrawler isolates platform-specific crawling logic behind abstract interfaces defined in base/base_crawler.py. By implementing a factory pattern, the codebase allows a single entry point to instantiate different crawlers (Xiaohongshu, Zhihu, Weibo, Tieba) without hard-coding platform logic into the main execution flow.

How the Factory Pattern Works in MediaCrawler

The architecture separates concerns into three distinct layers: the abstract interface, the concrete implementations, and the factory registry that binds them at runtime.

Abstract Interface Design

All crawlers implement the AbstractCrawler base class defined in base/base_crawler.py. This interface standardizes the methods that the main execution loop expects, regardless of the underlying platform.

# base/base_crawler.py

from abc import ABC, abstractmethod

class AbstractCrawler(ABC):
    @abstractmethod
    async def start(self):
        """Initialize the crawler session."""
        pass
    
    @abstractmethod
    async def search(self):
        """Execute search queries."""
        pass
    
    @abstractmethod
    async def launch_browser(self, chromium, proxy, ua, headless=True):
        """Configure and launch browser instance."""
        pass

Platform Registration

The CrawlerFactory maintains a private _registry dictionary that maps string identifiers to crawler classes. New platforms are added by importing their concrete implementations and updating this registry.

# factory/crawler_factory.py

from base.base_crawler import AbstractCrawler
from media_platform.xhs.core import XhsCrawler
from media_platform.zhihu.core import ZhihuCrawler

class CrawlerFactory:
    """Factory that creates crawler instances based on platform name."""
    _registry = {
        "xhs": XhsCrawler,
        "zhihu": ZhihuCrawler,
        # Additional platforms register here

    }

    @classmethod
    def create(cls, platform: str) -> AbstractCrawler:
        """Instantiate the appropriate crawler for the given platform."""
        try:
            crawler_cls = cls._registry[platform]
        except KeyError as exc:
            raise ValueError(f"Unsupported platform: {platform}") from exc
        return crawler_cls()

Runtime Instantiation

The entry point in main.py parses command-line arguments and delegates instantiation to the factory. The rest of the codebase interacts only with the AbstractCrawler interface, not concrete implementations.

# main.py (simplified entry point)

from cmd_arg.arg import parse_args
from factory.crawler_factory import CrawlerFactory

async def run():
    args = parse_args()
    # args.platform contains the value from --platform CLI flag

    crawler = CrawlerFactory.create(args.platform)
    await crawler.start()
    await crawler.search()

if __name__ == "__main__":
    import asyncio
    asyncio.run(run())

Implementing a Concrete Crawler

Each platform resides in its own submodule under media_platform/. For example, the Xiaohongshu (XHS) implementation extends the base interface with platform-specific login flows and API handling.

# media_platform/xhs/core.py

from base.base_crawler import AbstractCrawler

class XhsCrawler(AbstractCrawler):
    async def start(self):
        """Platform-specific initialization."""
        # XHS login and session setup

    
    async def search(self):
        """Execute XHS search queries."""
        # API calls and HTML parsing logic

    
    async def launch_browser(self, chromium, proxy, ua, headless=True):
        """Configure Chromium for XHS anti-detection."""
        # Browser context configuration

Adding a New Platform to the Registry

Extending MediaCrawler to support a new platform requires three steps:

1. Create the concrete crawler class in media_platform/<platform>/core.py
2. Import the class in factory/crawler_factory.py
3. Register the mapping in the _registry dictionary

# Step 1: Create implementation (media_platform/bilibili/core.py)

from base.base_crawler import AbstractCrawler

class BiliCrawler(AbstractCrawler):
    async def start(self):
        pass
    
    async def search(self):
        pass
    
    async def launch_browser(self, chromium, proxy, ua, headless=True):
        pass

# Step 2 & 3: Register in factory/crawler_factory.py

from media_platform.bilibili.core import BiliCrawler

CrawlerFactory._registry["bilibili"] = BiliCrawler

After registration, the platform becomes available via CLI:

uv run main.py --platform bilibili --lt qrcode --type search

Testing with the Factory Pattern

The factory enables dependency injection for unit tests and allows mocking platform-specific behavior without modifying production code.

# tests/test_crawler_factory.py

from factory.crawler_factory import CrawlerFactory
from media_platform.xhs.core import XhsCrawler

def test_factory_returns_xhs_instance():
    """Verify factory creates correct type for 'xhs' platform."""
    crawler = CrawlerFactory.create("xhs")
    assert isinstance(crawler, XhsCrawler)

For integration tests, patch the registry to inject mock objects:

from unittest.mock import MagicMock, patch
from factory.crawler_factory import CrawlerFactory

@patch.object(CrawlerFactory, "_registry", {"mock": MagicMock})
def test_custom_crawler_execution():
    """Test workflow with mocked crawler."""
    crawler = CrawlerFactory.create("mock")
    crawler.start()
    crawler.start.assert_called_once()

Summary

• Abstract Interface: base/base_crawler.py defines the AbstractCrawler contract that all platforms must implement.
• Centralized Registry: CrawlerFactory._registry in factory/crawler_factory.py maps platform strings to concrete classes.
• Runtime Selection: The --platform CLI argument parsed in cmd_arg/arg.py drives which crawler class gets instantiated.
• Extensibility: Adding platforms requires only a new class and a single line of registration code.
• Testability: The registry pattern allows easy substitution of mock crawlers for testing.

Frequently Asked Questions

What is the benefit of using a factory pattern for crawlers?

The factory pattern decouples the main execution logic from platform-specific implementations. This allows the same binary to crawl Xiaohongshu, Zhihu, Weibo, or Tieba simply by changing the --platform flag, without conditionals scattered throughout the codebase. It also centralizes error handling for unsupported platforms in a single location.

How do I add support for a new platform in MediaCrawler?

Create a new class in media_platform/<name>/core.py that inherits from AbstractCrawler, implement the required start(), search(), and launch_browser() methods, then import and register the class in factory/crawler_factory.py's _registry dictionary with a unique string key.

Where is the platform argument parsed in the codebase?

The command-line argument --platform is parsed in cmd_arg/arg.py. The resulting value is passed to CrawlerFactory.create() in main.py, which looks up the corresponding crawler class in its registry and returns an instance ready for execution.

Can I use the factory pattern for other components like data stores?

Yes. The repository already implements a similar pattern for data persistence in store/xhs/_store_impl.py with XhsStoreFactory. You can extend this approach to any component that requires platform-specific variants, such as authentication handlers or proxy rotators, following the same registry-based structure used in CrawlerFactory.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}