How to Implement a Custom Crawler in MediaCrawler: A Complete Developer Guide

You can implement a custom crawler in MediaCrawler by subclassing AbstractCrawler from base/base_crawler.py, implementing the start() and search() methods, and registering your class in CrawlerFactory.CRAWLERS within main.py.

MediaCrawler is a modular, open-source scraping framework designed for social media platforms. To implement a custom crawler in MediaCrawler, you extend the base abstraction layer and leverage existing utilities for browser automation, proxy handling, and data storage. This guide walks through the exact file structure, method signatures, and registration steps required to add support for any new platform.

Understanding the Core Architecture

MediaCrawler follows a clean plugin architecture that separates browser handling, data extraction, and storage concerns.

The AbstractCrawler Interface

The AbstractCrawler class in base/base_crawler.py defines the contract every crawler must implement. Key methods include start() for initialization, search() for data discovery, and launch_browser() for browser instantiation. When you implement a custom crawler, you inherit from this base class and override the abstract methods with platform-specific logic.

CrawlerFactory Registration

The CrawlerFactory class in main.py maintains a mapping dictionary called CRAWLERS that associates platform names (like zhihu or xhs) with concrete crawler classes. To make your crawler accessible from the command line via --platform your_platform, you must add your class to this registry.

Browser and Proxy Utilities

Common helper functions live in tools/crawler_util.py, including format_proxy_info() for proxy configuration and get_user_agent() for rotation. The AbstractCrawler base class provides launch_browser() and launch_browser_with_cdp() methods that handle Playwright initialization, allowing your custom crawler to support both standard and Chrome DevTools Protocol (CDP) modes without rewriting browser logic.

Step-by-Step Implementation Guide

Follow these steps to add a new platform crawler to the MediaCrawler codebase.

Step 1: Create the Platform Package Structure

Create a new directory under media_platform/<your_platform>/ containing at minimum:

• core.py: Your crawler class implementation
• client.py: API wrapper for the target site
• login.py: (Optional) Authentication handler
• __init__.py: Package initialization

Reference the Zhihu implementation at media_platform/zhihu/core.py for a complete working example.

Step 2: Subclass AbstractCrawler

In media_platform/<your_platform>/core.py, import AbstractCrawler from base.base_crawler and create your subclass:

from base.base_crawler import AbstractCrawler

class MyPlatformCrawler(AbstractCrawler):
    def __init__(self) -> None:
        self.base_url = "https://example.com"
        self.user_agent = get_user_agent()
        self.browser_context = None
        self.context_page = None
        self.client = None

Implement __init__() to initialize platform-specific attributes like base URLs, cookie lists, and API clients.

Step 3: Implement Browser Initialization

Override the start() method to handle browser launch and proxy setup:

async def start(self) -> None:
    # Configure proxy if enabled

    playwright_proxy, httpx_proxy = None, None
    if config.ENABLE_IP_PROXY:
        from proxy.proxy_ip_pool import create_ip_pool
        self.ip_proxy_pool = await create_ip_pool(3, enable_validate_ip=True)
        pinfo = await self.ip_proxy_pool.get_proxy()
        playwright_proxy, httpx_proxy = format_proxy_info(pinfo)
    
    # Launch browser using base class helpers

    async with async_playwright() as pw:
        if config.ENABLE_CDP_MODE:
            self.browser_context = await self.launch_browser_with_cdp(
                pw, playwright_proxy, self.user_agent, headless=config.HEADLESS
            )
        else:
            self.browser_context = await self.launch_browser(
                pw.chromium, playwright_proxy, self.user_agent, headless=config.HEADLESS
            )
        
        self.context_page = await self.browser_context.new_page()
        await self.context_page.goto(self.base_url)

Step 4: Build the API Client

Create a client class in client.py that handles HTTP requests and HTML parsing. In start(), instantiate this client after browser initialization:

async def start(self) -> None:
    # ... browser setup ...

    self.client = await self._create_client(httpx_proxy)
    
    # Handle login if needed

    if not await self.client.check_session():
        await self._login()
    
    # Route to appropriate crawl method

    if config.CRAWLER_TYPE == "search":
        await self.search()
    elif config.CRAWLER_TYPE == "detail":
        await self.get_specified_items()

Step 5: Register in CrawlerFactory

Open main.py and locate the CrawlerFactory class. Import your crawler and add it to the CRAWLERS dictionary:

from media_platform.myplatform.core import MyPlatformCrawler

class CrawlerFactory:
    CRAWLERS = {
        "xhs": XiaoHongShuCrawler,
        "zhihu": ZhiHuCrawler,
        "myplatform": MyPlatformCrawler,  # Add your crawler here

    }

Your crawler is now accessible via python main.py --platform myplatform.

Complete Code Example: Custom Platform Skeleton

Here is a complete, runnable skeleton for media_platform/myplatform/core.py:

import asyncio
from base.base_crawler import AbstractCrawler
from tools import utils
from tools.crawler_util import get_user_agent, format_proxy_info
from proxy.proxy_ip_pool import create_ip_pool, IpInfoModel
from config import CRAWLER_MAX_SLEEP_SEC, ENABLE_IP_PROXY, HEADLESS
from playwright.async_api import BrowserContext, async_playwright

class MyPlatformCrawler(AbstractCrawler):
    def __init__(self) -> None:
        self.base_url = "https://example.com"
        self.cookie_urls = [self.base_url]
        self.user_agent = get_user_agent()
        self.ip_proxy_pool = None
        self.browser_context: BrowserContext
        self.context_page = None
        self.client = None

    async def start(self) -> None:
        playwright_proxy, httpx_proxy = None, None
        
        if ENABLE_IP_PROXY:
            self.ip_proxy_pool = await create_ip_pool(3, enable_validate_ip=True)
            pinfo: IpInfoModel = await self.ip_proxy_pool.get_proxy()
            playwright_proxy, httpx_proxy = format_proxy_info(pinfo)

        async with async_playwright() as pw:
            if utils.config.ENABLE_CDP_MODE:
                self.browser_context = await self.launch_browser_with_cdp(
                    pw, playwright_proxy, self.user_agent, headless=HEADLESS
                )
            else:
                self.browser_context = await self.launch_browser(
                    pw.chromium, playwright_proxy, self.user_agent, headless=HEADLESS
                )

            self.context_page = await self.browser_context.new_page()
            await self.context_page.goto(self.base_url, wait_until="domcontentloaded")
            
            self.client = await self._create_client(httpx_proxy)
            
            cookies_ok = await self.client.check_session()
            if not cookies_ok:
                await self._login_via_qrcode()

            if utils.config.CRAWLER_TYPE == "search":
                await self.search()
            elif utils.config.CRAWLER_TYPE == "detail":
                await self.get_specified_items()

    async def search(self) -> None:
        for kw in utils.config.KEYWORDS.split(","):
            page = 1
            while page <= utils.config.MAX_PAGE:
                items = await self.client.search_items(keyword=kw, page=page)
                if not items:
                    break
                for it in items:
                    await utils.store.update_myplatform_item(it)
                await asyncio.sleep(CRAWLER_MAX_SLEEP_SEC)
                page += 1

    async def _create_client(self, httpx_proxy: str | None):
        cookie_str, cookie_dict = await utils.convert_browser_context_cookies(
            self.browser_context, urls=self.cookie_urls
        )
        from .client import MyPlatformClient
        return MyPlatformClient(
            proxy=httpx_proxy,
            headers={"user-agent": self.user_agent, "cookie": cookie_str},
            playwright_page=self.context_page,
            cookie_dict=cookie_dict,
            proxy_ip_pool=self.ip_proxy_pool,
        )

    async def _login_via_qrcode(self):
        from .login import MyPlatformLogin
        login = MyPlatformLogin(
            browser_context=self.browser_context,
            context_page=self.context_page,
            cookie_str=utils.config.COOKIES,
        )
        await login.begin()
        await self.client.update_cookies(self.browser_context, urls=self.cookie_urls)

Key Files and Reference Implementations

When implementing a custom crawler in MediaCrawler, reference these critical files:

• base/base_crawler.py: Defines AbstractCrawler with required methods start(), search(), and browser launchers
• main.py: Contains CrawlerFactory where you register new crawlers in the CRAWLERS dictionary
• tools/crawler_util.py: Provides format_proxy_info() and get_user_agent() for network configuration
• tools/utils.py: Contains convert_browser_context_cookies() for extracting cookies from Playwright contexts
• media_platform/zhihu/core.py: Complete reference implementation showing login handling, CDP mode support, and comment collection

Summary

To implement a custom crawler in MediaCrawler:

• Subclass AbstractCrawler from base/base_crawler.py and implement start() and search() methods
• Reuse browser utilities like launch_browser() and launch_browser_with_cdp() from the base class to handle Playwright initialization
• Use helper functions from tools/crawler_util.py for proxy formatting and user-agent rotation
• Create a client wrapper in client.py to handle platform-specific HTTP requests and authentication
• Register in CrawlerFactory by adding your class to the CRAWLERS dictionary in main.py
• Follow the Zhihu pattern in media_platform/zhihu/core.py for handling sessions, pagination, and data storage

Frequently Asked Questions

What methods must I implement in AbstractCrawler?

You must implement start() to handle initialization and browser setup. You should also implement search() if using search-based crawling, or get_specified_items() if crawling specific URLs. The AbstractCrawler base class in base/base_crawler.py provides default implementations for browser launching that you can call via super().

How do I handle authentication in my custom crawler?

Implement a login method in your crawler class (like _login_via_qrcode() shown above) that uses a dedicated Login class from your platform's login.py. Check for valid sessions using your client’s check_session() method, and if cookies are missing, trigger the login flow before proceeding to data extraction. Use utils.convert_browser_context_cookies() to sync cookies between Playwright and your HTTP client.

Can I use CDP mode with my custom crawler?

Yes. The AbstractCrawler base class provides launch_browser_with_cdp() which you can call when config.ENABLE_CDP_MODE is True. This method handles Chrome DevTools Protocol connection setup, allowing you to bypass certain detection mechanisms. Check media_platform/zhihu/core.py for a working implementation that falls back to standard Playwright when CDP is disabled.

Where should I store scraped data?

Your crawler should call storage functions from the store/ module (e.g., store.update_myplatform_item()) rather than handling file I/O directly. Create a new storage module in store/ following the pattern of existing platform stores like store/zhihu. This keeps your crawler focused on extraction while the storage layer handles JSON, Excel, or database output formats.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}