MediaCrawler Playwright Automation Setup: Complete CDP Browser Management Guide

MediaCrawler orchestrates Chromium browsers via Playwright's Chrome DevTools Protocol (CDP) integration through the CDPBrowserManager class, enabling automated authentication and data extraction on platforms like Zhihu and Xiaohongshu without manual browser intervention.

The MediaCrawler repository is a Python framework designed for crawling media-rich social platforms including Zhihu, Xiaohongshu, Weibo, and Kuaishou. The MediaCrawler Playwright automation setup centers on asynchronous Playwright integration with Chrome DevTools Protocol (CDP) to manage browser lifecycles, handle authentication flows, and persist session states across crawling operations.

Core Architecture Components

The automation layer consists of three primary modules that abstract complex browser operations into manageable interfaces.

CDPBrowserManager

Located in tools/cdp_browser.py, the CDPBrowserManager class encapsulates the entire browser lifecycle from launch to cleanup. It handles both new browser instances and connections to existing Chromium processes via CDP WebSocket endpoints, providing a unified interface for platform clients in media_platform/zhihu/core.py and similar modules.

BrowserLauncher

The BrowserLauncher module in tools/browser_launcher.py manages OS-specific browser detection and process initialization. It locates Chrome or Edge binaries across Windows, macOS, and Linux systems, assigns free TCP ports for remote debugging, and spawns processes with the --remote-debugging-port flag.

Crawler Utilities

The tools/crawler_util.py file provides stateless helper functions for common automation tasks including QR code extraction, cookie format conversion, and proxy configuration formatting. These utilities support platform-specific login implementations without coupling to browser management logic.

CDPBrowserManager Implementation Details

The manager class implements a complete state machine for browser control, supporting both persistent connections and ephemeral instances.

Browser Initialization and Connection

The launch_and_connect method determines whether to attach to an existing browser instance based on the config.CDP_CONNECT_EXISTING setting. When creating new instances, it calls _get_browser_path to resolve the Chromium binary, checking config.CUSTOM_BROWSER_PATH first before falling back to auto-detection via BrowserLauncher.detect_browser_paths().

The _launch_browser method starts the process with the selected debug port, optional headless mode, and user-data directory when config.SAVE_LOGIN_STATE is enabled. The _connect_via_cdp method retrieves the WebSocket URL from /json/version or constructs ws://localhost:<port>/devtools/browser, then establishes the connection via playwright.chromium.connect_over_cdp.

Context Creation and Configuration

The _create_browser_context method returns a Playwright BrowserContext configured with a 1920×1080 viewport, download permissions, and optional user-agent strings. This context persists across operations when login state preservation is enabled, storing authentication data in the specified user-data directory.

Stealth and Cleanup Operations

The manager injects anti-detection scripts via add_stealth_script (loading libs/stealth.min.js) and manages cookies through add_cookies and get_cookies methods. The cleanup method handles graceful shutdown, disconnecting CDP clients and terminating browser processes based on config.AUTO_CLOSE_BROWSER. The class also registers atexit and signal handlers to ensure cleanup occurs during script interruption.

BrowserLauncher and Process Management

The BrowserLauncher class handles low-level system interactions required before Playwright can connect. The detect_browser_paths method walks standard installation directories to locate chrome.exe, chrome, or Edge binaries across operating systems.

The find_available_port utility selects TCP ports for the remote debugging endpoint, while wait_for_browser_ready polls the debug port until the CDP service responds with a valid WebSocket URL. When launching, the browser spawns with --remote-debugging-port=<port> and optional --headless and --user-data-dir flags to support persistent logins and background operation.

Utility Functions for Authentication and Networking

Platform clients rely on helper functions in tools/crawler_util.py to handle authentication challenges without implementing protocol details repeatedly.

QR Code Extraction

The find_login_qrcode function locates QR code elements using CSS selectors (such as .LoginModal-qrCode img on Zhihu), downloads the image data via httpx, and returns base64-encoded strings compatible with the show_qrcode display utility. This enables headless automation of mobile-app-based login flows.

Cookie and Proxy Handling

convert_cookies and convert_browser_context_cookies transform Playwright cookie objects into header-compatible formats for HTTP clients. The format_proxy_info function generates Playwright-compatible proxy dictionaries and httpx URL strings, handling authentication credentials cleanly across both browser and direct HTTP contexts.

Complete Implementation Example

The following example demonstrates launching a browser, navigating to Zhihu, and utilizing platform-specific helpers:

import asyncio
from playwright.async_api import async_playwright
from tools.cdp_browser import CDPBrowserManager
from media_platform.zhihu.core import ZhihuCore

async def main():
    async with async_playwright() as playwright:
        cdp_manager = CDPBrowserManager()
        context = await cdp_manager.launch_and_connect(
            playwright,
            playwright_proxy=None,
            user_agent=None,
            headless=False,
        )

        page = await context.new_page()
        await page.goto("https://www.zhihu.com")

        zhihu = ZhihuCore(page)
        await zhihu.login()

        await cdp_manager.cleanup()

asyncio.run(main())

For QR code-based authentication, combine the browser manager with utility functions:

from tools.crawler_util import find_login_qrcode, show_qrcode

async def login_via_qr(page):
    qr_base64 = await find_login_qrcode(page, ".LoginModal-qrCode img")
    if qr_base64:
        show_qrcode(qr_base64)

Summary

• CDPBrowserManager in tools/cdp_browser.py orchestrates the full browser lifecycle via CDP connections, supporting both new launches and existing browser attachment
• BrowserLauncher in tools/browser_launcher.py handles cross-platform browser detection and remote debugging port assignment
• Utility functions in tools/crawler_util.py manage QR code extraction, cookie conversion, and proxy formatting for authentication workflows
• Configuration options including CDP_CONNECT_EXISTING, SAVE_LOGIN_STATE, and AUTO_CLOSE_BROWSER control connection behavior and resource persistence
• Anti-detection measures are implemented via libs/stealth.min.js injection through the browser manager's add_stealth_script method

Frequently Asked Questions

What is CDP and why does MediaCrawler use it?

The Chrome DevTools Protocol (CDP) allows MediaCrawler to communicate directly with Chromium browsers via WebSocket connections. According to the source code in tools/cdp_browser.py, CDP enables the framework to attach to existing browser instances or launch new ones with fine-grained control over debugging ports, contexts, and page interactions, facilitating seamless integration between Playwright and manually configured browser environments.

How does MediaCrawler handle browser cleanup and resource management?

The CDPBrowserManager.cleanup() method implements graceful shutdown sequences that terminate browser contexts, disconnect CDP clients, and optionally kill the browser process based on the config.AUTO_CLOSE_BROWSER setting. The manager also registers atexit and signal handlers to ensure cleanup occurs even during script interruption, preventing orphaned Chromium processes.

Where does MediaCrawler store login sessions and browser data?

When config.SAVE_LOGIN_STATE is enabled, the browser launches with a persistent --user-data-dir specified in the configuration. The _create_browser_context method maintains this state across sessions, while get_cookies and add_cookies methods in the manager enable programmatic session management without relying solely on disk storage, allowing crawlers to resume authenticated sessions across restarts.

Can MediaCrawler run in headless mode for server deployments?

Yes. The launch_and_connect method accepts a headless parameter that passes the --headless flag to the browser process. However, some platforms detect headless automation, so the framework supports add_stealth_script to inject evasion scripts from libs/stealth.min.js when running in visible or headless modes, making it suitable for both development workstations and containerized production environments.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}