# How to Implement a Custom Crawler in MediaCrawler: A Complete Developer Guide

> Implement a custom crawler in MediaCrawler by subclassing AbstractCrawler. Follow this developer guide to easily add your own crawlers and extend MediaCrawler's capabilities.

- Repository: [程序员阿江-Relakkes/MediaCrawler](https://github.com/NanmiCoder/MediaCrawler)
- Tags: how-to-guide
- Published: 2026-06-26

---

**You can implement a custom crawler in MediaCrawler by subclassing `AbstractCrawler` from [`base/base_crawler.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/base/base_crawler.py), implementing the `start()` and `search()` methods, and registering your class in `CrawlerFactory.CRAWLERS` within [`main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/main.py).**

MediaCrawler is a modular, open-source scraping framework designed for social media platforms. To implement a custom crawler in MediaCrawler, you extend the base abstraction layer and leverage existing utilities for browser automation, proxy handling, and data storage. This guide walks through the exact file structure, method signatures, and registration steps required to add support for any new platform.

## Understanding the Core Architecture

MediaCrawler follows a clean plugin architecture that separates browser handling, data extraction, and storage concerns.

### The AbstractCrawler Interface

The `AbstractCrawler` class in [`base/base_crawler.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/base/base_crawler.py) defines the contract every crawler must implement. Key methods include `start()` for initialization, `search()` for data discovery, and `launch_browser()` for browser instantiation. When you implement a custom crawler, you inherit from this base class and override the abstract methods with platform-specific logic.

### CrawlerFactory Registration

The `CrawlerFactory` class in [`main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/main.py) maintains a mapping dictionary called `CRAWLERS` that associates platform names (like `zhihu` or `xhs`) with concrete crawler classes. To make your crawler accessible from the command line via `--platform your_platform`, you must add your class to this registry.

### Browser and Proxy Utilities

Common helper functions live in [`tools/crawler_util.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/crawler_util.py), including `format_proxy_info()` for proxy configuration and `get_user_agent()` for rotation. The `AbstractCrawler` base class provides `launch_browser()` and `launch_browser_with_cdp()` methods that handle Playwright initialization, allowing your custom crawler to support both standard and Chrome DevTools Protocol (CDP) modes without rewriting browser logic.

## Step-by-Step Implementation Guide

Follow these steps to add a new platform crawler to the MediaCrawler codebase.

### Step 1: Create the Platform Package Structure

Create a new directory under `media_platform/<your_platform>/` containing at minimum:

- [`core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/core.py): Your crawler class implementation
- [`client.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/client.py): API wrapper for the target site
- [`login.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/login.py): (Optional) Authentication handler
- [`__init__.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/__init__.py): Package initialization

Reference the Zhihu implementation at [`media_platform/zhihu/core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/zhihu/core.py) for a complete working example.

### Step 2: Subclass AbstractCrawler

In `media_platform/<your_platform>/core.py`, import `AbstractCrawler` from `base.base_crawler` and create your subclass:

```python
from base.base_crawler import AbstractCrawler

class MyPlatformCrawler(AbstractCrawler):
    def __init__(self) -> None:
        self.base_url = "https://example.com"
        self.user_agent = get_user_agent()
        self.browser_context = None
        self.context_page = None
        self.client = None

```

Implement `__init__()` to initialize platform-specific attributes like base URLs, cookie lists, and API clients.

### Step 3: Implement Browser Initialization

Override the `start()` method to handle browser launch and proxy setup:

```python
async def start(self) -> None:
    # Configure proxy if enabled

    playwright_proxy, httpx_proxy = None, None
    if config.ENABLE_IP_PROXY:
        from proxy.proxy_ip_pool import create_ip_pool
        self.ip_proxy_pool = await create_ip_pool(3, enable_validate_ip=True)
        pinfo = await self.ip_proxy_pool.get_proxy()
        playwright_proxy, httpx_proxy = format_proxy_info(pinfo)
    
    # Launch browser using base class helpers

    async with async_playwright() as pw:
        if config.ENABLE_CDP_MODE:
            self.browser_context = await self.launch_browser_with_cdp(
                pw, playwright_proxy, self.user_agent, headless=config.HEADLESS
            )
        else:
            self.browser_context = await self.launch_browser(
                pw.chromium, playwright_proxy, self.user_agent, headless=config.HEADLESS
            )
        
        self.context_page = await self.browser_context.new_page()
        await self.context_page.goto(self.base_url)

```

### Step 4: Build the API Client

Create a client class in [`client.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/client.py) that handles HTTP requests and HTML parsing. In `start()`, instantiate this client after browser initialization:

```python
async def start(self) -> None:
    # ... browser setup ...

    self.client = await self._create_client(httpx_proxy)
    
    # Handle login if needed

    if not await self.client.check_session():
        await self._login()
    
    # Route to appropriate crawl method

    if config.CRAWLER_TYPE == "search":
        await self.search()
    elif config.CRAWLER_TYPE == "detail":
        await self.get_specified_items()

```

### Step 5: Register in CrawlerFactory

Open [`main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/main.py) and locate the `CrawlerFactory` class. Import your crawler and add it to the `CRAWLERS` dictionary:

```python
from media_platform.myplatform.core import MyPlatformCrawler

class CrawlerFactory:
    CRAWLERS = {
        "xhs": XiaoHongShuCrawler,
        "zhihu": ZhiHuCrawler,
        "myplatform": MyPlatformCrawler,  # Add your crawler here

    }

```

Your crawler is now accessible via `python main.py --platform myplatform`.

## Complete Code Example: Custom Platform Skeleton

Here is a complete, runnable skeleton for [`media_platform/myplatform/core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/myplatform/core.py):

```python
import asyncio
from base.base_crawler import AbstractCrawler
from tools import utils
from tools.crawler_util import get_user_agent, format_proxy_info
from proxy.proxy_ip_pool import create_ip_pool, IpInfoModel
from config import CRAWLER_MAX_SLEEP_SEC, ENABLE_IP_PROXY, HEADLESS
from playwright.async_api import BrowserContext, async_playwright

class MyPlatformCrawler(AbstractCrawler):
    def __init__(self) -> None:
        self.base_url = "https://example.com"
        self.cookie_urls = [self.base_url]
        self.user_agent = get_user_agent()
        self.ip_proxy_pool = None
        self.browser_context: BrowserContext
        self.context_page = None
        self.client = None

    async def start(self) -> None:
        playwright_proxy, httpx_proxy = None, None
        
        if ENABLE_IP_PROXY:
            self.ip_proxy_pool = await create_ip_pool(3, enable_validate_ip=True)
            pinfo: IpInfoModel = await self.ip_proxy_pool.get_proxy()
            playwright_proxy, httpx_proxy = format_proxy_info(pinfo)

        async with async_playwright() as pw:
            if utils.config.ENABLE_CDP_MODE:
                self.browser_context = await self.launch_browser_with_cdp(
                    pw, playwright_proxy, self.user_agent, headless=HEADLESS
                )
            else:
                self.browser_context = await self.launch_browser(
                    pw.chromium, playwright_proxy, self.user_agent, headless=HEADLESS
                )

            self.context_page = await self.browser_context.new_page()
            await self.context_page.goto(self.base_url, wait_until="domcontentloaded")
            
            self.client = await self._create_client(httpx_proxy)
            
            cookies_ok = await self.client.check_session()
            if not cookies_ok:
                await self._login_via_qrcode()

            if utils.config.CRAWLER_TYPE == "search":
                await self.search()
            elif utils.config.CRAWLER_TYPE == "detail":
                await self.get_specified_items()

    async def search(self) -> None:
        for kw in utils.config.KEYWORDS.split(","):
            page = 1
            while page <= utils.config.MAX_PAGE:
                items = await self.client.search_items(keyword=kw, page=page)
                if not items:
                    break
                for it in items:
                    await utils.store.update_myplatform_item(it)
                await asyncio.sleep(CRAWLER_MAX_SLEEP_SEC)
                page += 1

    async def _create_client(self, httpx_proxy: str | None):
        cookie_str, cookie_dict = await utils.convert_browser_context_cookies(
            self.browser_context, urls=self.cookie_urls
        )
        from .client import MyPlatformClient
        return MyPlatformClient(
            proxy=httpx_proxy,
            headers={"user-agent": self.user_agent, "cookie": cookie_str},
            playwright_page=self.context_page,
            cookie_dict=cookie_dict,
            proxy_ip_pool=self.ip_proxy_pool,
        )

    async def _login_via_qrcode(self):
        from .login import MyPlatformLogin
        login = MyPlatformLogin(
            browser_context=self.browser_context,
            context_page=self.context_page,
            cookie_str=utils.config.COOKIES,
        )
        await login.begin()
        await self.client.update_cookies(self.browser_context, urls=self.cookie_urls)

```

## Key Files and Reference Implementations

When implementing a custom crawler in MediaCrawler, reference these critical files:

- **[`base/base_crawler.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/base/base_crawler.py)**: Defines `AbstractCrawler` with required methods `start()`, `search()`, and browser launchers
- **[`main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/main.py)**: Contains `CrawlerFactory` where you register new crawlers in the `CRAWLERS` dictionary
- **[`tools/crawler_util.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/crawler_util.py)**: Provides `format_proxy_info()` and `get_user_agent()` for network configuration
- **[`tools/utils.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/utils.py)**: Contains `convert_browser_context_cookies()` for extracting cookies from Playwright contexts
- **[`media_platform/zhihu/core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/zhihu/core.py)**: Complete reference implementation showing login handling, CDP mode support, and comment collection

## Summary

To implement a custom crawler in MediaCrawler:

- **Subclass `AbstractCrawler`** from [`base/base_crawler.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/base/base_crawler.py) and implement `start()` and `search()` methods
- **Reuse browser utilities** like `launch_browser()` and `launch_browser_with_cdp()` from the base class to handle Playwright initialization
- **Use helper functions** from [`tools/crawler_util.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/crawler_util.py) for proxy formatting and user-agent rotation
- **Create a client wrapper** in [`client.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/client.py) to handle platform-specific HTTP requests and authentication
- **Register in `CrawlerFactory`** by adding your class to the `CRAWLERS` dictionary in [`main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/main.py)
- **Follow the Zhihu pattern** in [`media_platform/zhihu/core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/zhihu/core.py) for handling sessions, pagination, and data storage

## Frequently Asked Questions

### What methods must I implement in AbstractCrawler?

You must implement `start()` to handle initialization and browser setup. You should also implement `search()` if using search-based crawling, or `get_specified_items()` if crawling specific URLs. The `AbstractCrawler` base class in [`base/base_crawler.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/base/base_crawler.py) provides default implementations for browser launching that you can call via `super()`.

### How do I handle authentication in my custom crawler?

Implement a login method in your crawler class (like `_login_via_qrcode()` shown above) that uses a dedicated `Login` class from your platform's [`login.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/login.py). Check for valid sessions using your client’s `check_session()` method, and if cookies are missing, trigger the login flow before proceeding to data extraction. Use `utils.convert_browser_context_cookies()` to sync cookies between Playwright and your HTTP client.

### Can I use CDP mode with my custom crawler?

Yes. The `AbstractCrawler` base class provides `launch_browser_with_cdp()` which you can call when `config.ENABLE_CDP_MODE` is True. This method handles Chrome DevTools Protocol connection setup, allowing you to bypass certain detection mechanisms. Check [`media_platform/zhihu/core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/zhihu/core.py) for a working implementation that falls back to standard Playwright when CDP is disabled.

### Where should I store scraped data?

Your crawler should call storage functions from the `store/` module (e.g., `store.update_myplatform_item()`) rather than handling file I/O directly. Create a new storage module in `store/` following the pattern of existing platform stores like `store/zhihu`. This keeps your crawler focused on extraction while the storage layer handles JSON, Excel, or database output formats.