# How to Select Platform-Specific Crawlers at Runtime Using Factory Pattern in MediaCrawler

> Learn how to select platform-specific crawlers at runtime using the Factory pattern in MediaCrawler. This guide details dynamic crawler selection via CLI arguments.

- Repository: [程序员阿江-Relakkes/MediaCrawler](https://github.com/NanmiCoder/MediaCrawler)
- Tags: how-to-guide
- Published: 2026-06-27

---

**The MediaCrawler repository uses a centralized `CrawlerFactory` in [`factory/crawler_factory.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/factory/crawler_factory.py) to map platform identifiers like `"xhs"` or `"zhihu"` to concrete crawler classes, enabling runtime selection via the `--platform` CLI argument.**

MediaCrawler isolates platform-specific crawling logic behind abstract interfaces defined in [`base/base_crawler.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/base/base_crawler.py). By implementing a factory pattern, the codebase allows a single entry point to instantiate different crawlers (Xiaohongshu, Zhihu, Weibo, Tieba) without hard-coding platform logic into the main execution flow.

## How the Factory Pattern Works in MediaCrawler

The architecture separates concerns into three distinct layers: the abstract interface, the concrete implementations, and the factory registry that binds them at runtime.

### Abstract Interface Design

All crawlers implement the `AbstractCrawler` base class defined in [`base/base_crawler.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/base/base_crawler.py). This interface standardizes the methods that the main execution loop expects, regardless of the underlying platform.

```python

# base/base_crawler.py

from abc import ABC, abstractmethod

class AbstractCrawler(ABC):
    @abstractmethod
    async def start(self):
        """Initialize the crawler session."""
        pass
    
    @abstractmethod
    async def search(self):
        """Execute search queries."""
        pass
    
    @abstractmethod
    async def launch_browser(self, chromium, proxy, ua, headless=True):
        """Configure and launch browser instance."""
        pass

```

### Platform Registration

The `CrawlerFactory` maintains a private `_registry` dictionary that maps string identifiers to crawler classes. New platforms are added by importing their concrete implementations and updating this registry.

```python

# factory/crawler_factory.py

from base.base_crawler import AbstractCrawler
from media_platform.xhs.core import XhsCrawler
from media_platform.zhihu.core import ZhihuCrawler

class CrawlerFactory:
    """Factory that creates crawler instances based on platform name."""
    _registry = {
        "xhs": XhsCrawler,
        "zhihu": ZhihuCrawler,
        # Additional platforms register here

    }

    @classmethod
    def create(cls, platform: str) -> AbstractCrawler:
        """Instantiate the appropriate crawler for the given platform."""
        try:
            crawler_cls = cls._registry[platform]
        except KeyError as exc:
            raise ValueError(f"Unsupported platform: {platform}") from exc
        return crawler_cls()

```

### Runtime Instantiation

The entry point in [`main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/main.py) parses command-line arguments and delegates instantiation to the factory. The rest of the codebase interacts only with the `AbstractCrawler` interface, not concrete implementations.

```python

# main.py (simplified entry point)

from cmd_arg.arg import parse_args
from factory.crawler_factory import CrawlerFactory

async def run():
    args = parse_args()
    # args.platform contains the value from --platform CLI flag

    crawler = CrawlerFactory.create(args.platform)
    await crawler.start()
    await crawler.search()

if __name__ == "__main__":
    import asyncio
    asyncio.run(run())

```

## Implementing a Concrete Crawler

Each platform resides in its own submodule under `media_platform/`. For example, the Xiaohongshu (XHS) implementation extends the base interface with platform-specific login flows and API handling.

```python

# media_platform/xhs/core.py

from base.base_crawler import AbstractCrawler

class XhsCrawler(AbstractCrawler):
    async def start(self):
        """Platform-specific initialization."""
        # XHS login and session setup

    
    async def search(self):
        """Execute XHS search queries."""
        # API calls and HTML parsing logic

    
    async def launch_browser(self, chromium, proxy, ua, headless=True):
        """Configure Chromium for XHS anti-detection."""
        # Browser context configuration

```

## Adding a New Platform to the Registry

Extending MediaCrawler to support a new platform requires three steps:

1. **Create the concrete crawler class** in `media_platform/<platform>/core.py`
2. **Import the class** in [`factory/crawler_factory.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/factory/crawler_factory.py)
3. **Register the mapping** in the `_registry` dictionary

```python

# Step 1: Create implementation (media_platform/bilibili/core.py)

from base.base_crawler import AbstractCrawler

class BiliCrawler(AbstractCrawler):
    async def start(self):
        pass
    
    async def search(self):
        pass
    
    async def launch_browser(self, chromium, proxy, ua, headless=True):
        pass

# Step 2 & 3: Register in factory/crawler_factory.py

from media_platform.bilibili.core import BiliCrawler

CrawlerFactory._registry["bilibili"] = BiliCrawler

```

After registration, the platform becomes available via CLI:

```bash
uv run main.py --platform bilibili --lt qrcode --type search

```

## Testing with the Factory Pattern

The factory enables dependency injection for unit tests and allows mocking platform-specific behavior without modifying production code.

```python

# tests/test_crawler_factory.py

from factory.crawler_factory import CrawlerFactory
from media_platform.xhs.core import XhsCrawler

def test_factory_returns_xhs_instance():
    """Verify factory creates correct type for 'xhs' platform."""
    crawler = CrawlerFactory.create("xhs")
    assert isinstance(crawler, XhsCrawler)

```

For integration tests, patch the registry to inject mock objects:

```python
from unittest.mock import MagicMock, patch
from factory.crawler_factory import CrawlerFactory

@patch.object(CrawlerFactory, "_registry", {"mock": MagicMock})
def test_custom_crawler_execution():
    """Test workflow with mocked crawler."""
    crawler = CrawlerFactory.create("mock")
    crawler.start()
    crawler.start.assert_called_once()

```

## Summary

- **Abstract Interface**: [`base/base_crawler.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/base/base_crawler.py) defines the `AbstractCrawler` contract that all platforms must implement.
- **Centralized Registry**: `CrawlerFactory._registry` in [`factory/crawler_factory.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/factory/crawler_factory.py) maps platform strings to concrete classes.
- **Runtime Selection**: The `--platform` CLI argument parsed in [`cmd_arg/arg.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/cmd_arg/arg.py) drives which crawler class gets instantiated.
- **Extensibility**: Adding platforms requires only a new class and a single line of registration code.
- **Testability**: The registry pattern allows easy substitution of mock crawlers for testing.

## Frequently Asked Questions

### What is the benefit of using a factory pattern for crawlers?

The factory pattern decouples the main execution logic from platform-specific implementations. This allows the same binary to crawl Xiaohongshu, Zhihu, Weibo, or Tieba simply by changing the `--platform` flag, without conditionals scattered throughout the codebase. It also centralizes error handling for unsupported platforms in a single location.

### How do I add support for a new platform in MediaCrawler?

Create a new class in `media_platform/<name>/core.py` that inherits from `AbstractCrawler`, implement the required `start()`, `search()`, and `launch_browser()` methods, then import and register the class in [`factory/crawler_factory.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/factory/crawler_factory.py)'s `_registry` dictionary with a unique string key.

### Where is the platform argument parsed in the codebase?

The command-line argument `--platform` is parsed in [`cmd_arg/arg.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/cmd_arg/arg.py). The resulting value is passed to `CrawlerFactory.create()` in [`main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/main.py), which looks up the corresponding crawler class in its registry and returns an instance ready for execution.

### Can I use the factory pattern for other components like data stores?

Yes. The repository already implements a similar pattern for data persistence in [`store/xhs/_store_impl.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/xhs/_store_impl.py) with `XhsStoreFactory`. You can extend this approach to any component that requires platform-specific variants, such as authentication handlers or proxy rotators, following the same registry-based structure used in `CrawlerFactory`.