How to Configure CDP Mode and Connect to an Existing Chrome Browser in MediaCrawler for Reduced Detection

To enable CDP mode in MediaCrawler and connect to an existing Chrome browser, set ENABLE_CDP_MODE = True and CDP_CONNECT_EXISTING = True in config/base_config.py, then launch Chrome with --remote-debugging-port=9222 before running the crawler.

MediaCrawler supports CDP (Chrome DevTools Protocol) mode, which allows the crawler to drive a real Chrome or Edge instance via the DevTools protocol rather than spawning a fresh headless browser. This approach reuses your existing browser profile—including extensions, cookies, and login state—making the resulting traffic significantly harder for target sites to detect as automated. In this guide, you'll learn how to configure and use CDP mode with an existing browser instance in the NanmiCoder/MediaCrawler repository.

Enable CDP Mode in Configuration File

All CDP-related settings reside in config/base_config.py. To activate CDP mode and connect to an existing browser, modify these four key variables:

# config/base_config.py

ENABLE_CDP_MODE = True          # Activates CDP mode throughout the crawler

CDP_CONNECT_EXISTING = True     # Attaches to already-running Chrome instead of launching new

CDP_DEBUG_PORT = 9222           # Port where Chrome's remote debugging listens

CUSTOM_BROWSER_PATH = ""        # Optional: full path to Chrome/Edge executable

AUTO_CLOSE_BROWSER = False      # Optional: keep Chrome open after script finishes

The ENABLE_CDP_MODE flag is the master switch. When enabled, platform-specific crawlers (such as ZhihuCrawler in media_platform/zhihu/core.py) automatically route browser management through CDPBrowserManager instead of the standard Playwright launcher.

Prepare Chrome with Remote Debugging Enabled

Before running MediaCrawler, you must manually start Chrome with remote debugging activated. This is the only step requiring action outside the Python codebase.

Windows

"C:\Program Files\Google\Chrome\Application\chrome.exe" --remote-debugging-port=9222

macOS

open -a "Google Chrome" --args --remote-debugging-port=9222

Linux

google-chrome --remote-debugging-port=9222

Alternatively, navigate to chrome://inspect/#remote-debugging in an existing Chrome window and enable "Discover network targets," then configure the port through Chrome's settings.

How CDPBrowserManager Connects to Your Browser

The connection logic is implemented in tools/cdp_browser.py. When CDP_CONNECT_EXISTING = True, CDPBrowserManager executes this workflow internally:

1. Read configured debug port from config.CDP_DEBUG_PORT
2. Wait for port availability—loops testing socket connectivity up to BROWSER_LAUNCH_TIMEOUT seconds
3. Attempt direct CDP connection via playwright.chromium.connect_over_cdp(ws_url); falls back to /json/version discovery on failure
4. Reuse existing browser context if present, otherwise create fresh context
5. Inject anti-detection script (libs/stealth.min.js) and apply cookies

This process eliminates the need for additional anti-detection scripts or proxy configurations in many cases, since your authentic browser profile already contains legitimate cookies and extensions.

Launching the Crawler with CDP Mode Active

Once configuration is complete and Chrome is running with remote debugging, start your target platform crawler normally. The CDP integration happens automatically:

from media_platform.zhihu.core import ZhihuCrawler
import asyncio

async def run():
    crawler = ZhihuCrawler()
    await crawler.start()  # Internally triggers CDPBrowserManager when ENABLE_CDP_MODE is True

asyncio.run(run())

The start() method in platform cores checks config.ENABLE_CDP_MODE and conditionally invokes CDPBrowserManager.launch_and_connect() rather than standard browser initialization.

Alternative: Launch Fresh Browser in CDP Mode

If you prefer CDP protocol benefits without managing an external Chrome process, disable existing connection mode:

# config/base_config.py

CDP_CONNECT_EXISTING = False   # Manager will spawn new Chrome process

In this configuration, CDPBrowserManager executes BrowserLauncher.detect_browser_paths() from tools/browser_launcher.py to locate Chrome, then launches it with:

• --remote-debugging-port=<available_port>
• --no-sandbox
• Temporary user-data directory

It then connects via CDP and creates a fresh browser context. This provides CDP's debugging capabilities while maintaining isolation between runs.

Key Source Files Reference

File	Purpose	Direct Link
config/base_config.py	Central configuration for all CDP mode flags	[config/base_config.py](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py)
tools/cdp_browser.py	Implements CDPBrowserManager class handling connection, context management, and cleanup	[tools/cdp_browser.py](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/cdp_browser.py)
tools/browser_launcher.py	Low-level Chrome launching with required flags	[tools/browser_launcher.py](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/browser_launcher.py)
media_platform/zhihu/core.py	Representative platform implementation showing CDP mode selection logic	[media_platform/zhihu/core.py](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/zhihu/core.py)

Summary

• Enable CDP mode by setting ENABLE_CDP_MODE = True in config/base_config.py
• Connect to existing Chrome with CDP_CONNECT_EXISTING = True and matching CDP_DEBUG_PORT
• Start Chrome manually with --remote-debugging-port=9222 before running the crawler
• Let platform crawlers handle the rest—CDP connection occurs automatically in start() methods
• Preserve browser session across runs by setting AUTO_CLOSE_BROWSER = False
• Reduce detection risk by leveraging real user profiles, cookies, and extensions already present in your Chrome instance

Frequently Asked Questions

What is the main advantage of using CDP mode over standard headless browsing?

CDP mode drives a genuine Chrome instance with your actual user profile, so traffic carries authentic cookies, extensions, and browsing history that headless browsers cannot replicate. This makes detection by sophisticated anti-bot systems significantly more difficult.

Can I use CDP mode with browsers other than Chrome?

Yes. Edge and other Chromium-based browsers supporting the Chrome DevTools Protocol work with MediaCrawler's CDP implementation. specify the executable path in CUSTOM_BROWSER_PATH if auto-detection fails.

Why does my connection fail with timeout errors?

The most common cause is Chrome not running with remote debugging enabled, or running on a different port than specified in CDP_DEBUG_PORT. Verify Chrome launched with --remote-debugging-port=9222 and that no firewall rules block local connections to that port.

Is anti-detection still necessary when using CDP mode with an existing browser?

The libs/stealth.min.js script is automatically injected even in CDP mode, but its importance diminishes when connecting to an existing browser with legitimate user data. The authentic profile already provides strong anti-detection characteristics, though stealth scripts add defense-in-depth for advanced fingerprinting techniques.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}