How to Configure CDP Mode for Stealth Browsing in MediaCrawler

MediaCrawler supports stealth browsing via Chrome DevTools Protocol (CDP) by connecting to a remote debugging endpoint and injecting anti-fingerprinting scripts from libs/stealth.min.js into every page context.

MediaCrawler is an open-source scraping framework that leverages Playwright to automate browser interactions. When you configure CDP mode for stealth browsing in MediaCrawler, the tool connects directly to Chrome or Edge via the Chrome DevTools Protocol, bypassing standard automation detection while maintaining full control over the browser instance.

Enable CDP Mode in Configuration

All CDP behavior is controlled through the configuration module defined in config/base_config.py. The following flags determine how the browser launches and maintains its connection.

• CDP_CONNECT_EXISTING: Set to True to attach to an already-running Chrome instance with --remote-debugging-port enabled, or False to let MediaCrawler launch a fresh browser process.
• CDP_DEBUG_PORT: The TCP port (default 9222) where the browser’s CDP server listens.
• CUSTOM_BROWSER_PATH: Absolute path to a Chrome/Edge binary; if empty, the launcher auto-detects installed browsers.
• SAVE_LOGIN_STATE: When True, creates a persistent user-data directory so cookies and local storage survive restarts.
• BROWSER_LAUNCH_TIMEOUT: Seconds to wait for the browser to become reachable (default 30).
• AUTO_CLOSE_BROWSER: Whether to terminate the browser process on script exit (True) or leave it running (False).

These values are read in tools/cdp_browser.py, where the decision to attach versus launch is made on lines 108-110.

Launch or Attach to a CDP Browser Instance

The CDPBrowserManager class in tools/cdp_browser.py handles the lifecycle of the CDP connection. Use the launch_and_connect() method to establish the session according to your configuration.

Launching a Fresh Browser Instance

When CDP_CONNECT_EXISTING is False, MediaCrawler spawns a new browser process and waits for the CDP endpoint.

from tools.cdp_browser import CDPBrowserManager
from playwright.async_api import async_playwright

async def start_fresh():
    async with async_playwright() as pw:
        manager = CDPBrowserManager()
        context = await manager.launch_and_connect(
            playwright=pw,
            playwright_proxy=None,
            user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36...",
            headless=False,  # Set True in production

        )
        # Connection established (see lines [97-104](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/cdp_browser.py#L97-L104))

        return manager, context

Attaching to an Existing Chrome Process

Set CDP_CONNECT_EXISTING to True and start Chrome manually with the remote debugging flag:

chrome --remote-debugging-port=9222

Then connect via Python:

async def attach_existing():
    async with async_playwright() as pw:
        manager = CDPBrowserManager()
        # In practice, set this via config or before calling launch_and_connect

        manager.manager_config = {"CDP_CONNECT_EXISTING": True}
        context = await manager.launch_and_connect(playwright=pw, headless=False)
        # Manager waits for endpoint (see lines [166-176](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/cdp_browser.py#L166-L176))

        return manager, context

Inject the Stealth Script

After establishing the context, inject the anti-detection logic. MediaCrawler ships with a pre-minified stealth script located at libs/stealth.min.js.

Call add_stealth_script() to register the script as an init script for every new page:

await manager.add_stealth_script()  # Uses default libs/stealth.min.js

# Or specify a custom path:

# await manager.add_stealth_script("custom_stealth.js")

This method verifies the file exists and registers it via page.addInitScript so it executes before any page scripts run (see lines 400-407).

Complete Working Example

Combine the configuration, launch, and stealth injection into a single workflow:

import asyncio
from tools.cdp_browser import CDPBrowserManager
from playwright.async_api import async_playwright

async def run_stealth_crawl():
    async with async_playwright() as pw:
        mgr = CDPBrowserManager()
        
        # Launch CDP browser (respects settings in base_config.py)

        ctx = await mgr.launch_and_connect(
            playwright=pw,
            headless=False,
        )
        
        # Enable stealth browsing

        await mgr.add_stealth_script()
        
        # Begin crawling

        page = await ctx.new_page()
        await page.goto("https://www.zhihu.com")
        
        # Cleanup

        await mgr.cleanup()

asyncio.run(run_stealth_crawl())

Persist Browser State Across Sessions

When SAVE_LOGIN_STATE is enabled, the manager creates a directory at browser_data/cdp_<platform> (see lines 55-63). This folder stores cookies, local storage, and extension data, allowing you to maintain login sessions across crawler restarts.

# Ensure in config/base_config.py:

# SAVE_LOGIN_STATE = True

# The manager automatically uses this path when creating the context

ctx = await mgr.launch_and_connect(playwright=pw)

Summary

• Configuration: Set CDP_CONNECT_EXISTING, CDP_DEBUG_PORT, and related flags in config/base_config.py to control launch behavior.
• Connection: Use CDPBrowserManager.launch_and_connect() in tools/cdp_browser.py to attach to or spawn a CDP-enabled browser.
• Stealth: Call add_stealth_script() to inject libs/stealth.min.js and mask automation fingerprints.
• Persistence: Enable SAVE_LOGIN_STATE to retain cookies in browser_data/cdp_<platform> for long-running crawls.

Frequently Asked Questions

What is CDP mode in MediaCrawler?

CDP mode drives Chrome or Edge via the Chrome DevTools Protocol instead of standard Playwright browser management. This allows MediaCrawler to attach to existing browser instances, use custom user-data directories, and inject stealth scripts at the CDP level to evade bot detection.

How does the stealth script prevent detection?

The script in libs/stealth.min.js overrides JavaScript properties that automation frameworks typically expose—such as navigator.webdriver, window.chrome, and plugin inconsistencies. By executing as an init script (lines 400-407), it sanitizes the browser environment before any page scripts run.

Can I use a custom stealth script instead of the default?

Yes. Pass the absolute path to your custom script file when calling await manager.add_stealth_script("path/to/custom.js"). The method validates the file existence and registers it as the init script for all subsequent pages.

Why should I use CDP mode instead of standard Playwright?

CDP mode provides lower-level control over the browser process, enables connection to already-running instances (useful for debugging), and supports persistent user-data directories that standard Playwright contexts may handle differently. It also facilitates stealth configurations that are harder to achieve through high-level browser APIs alone.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}