ENABLE_CDP_MODE vs Playwright Headless Mode in MediaCrawler: Key Differences Explained

ENABLE_CDP_MODE determines whether MediaCrawler connects to an existing Chrome browser via the Chrome DevTools Protocol (CDP) or launches a fresh Playwright-managed instance, while the headless flags only control whether the browser window is visible during execution.

MediaCrawler is an open-source social media scraping framework that provides flexible browser automation strategies to balance stealth and performance. Understanding the distinction between connection mode and visibility settings is essential for configuring effective anti-detection measures and managing system resources efficiently.

What ENABLE_CDP_MODE Controls

ENABLE_CDP_MODE is a boolean configuration option defined in config/base_config.py (line 59) that fundamentally changes how the crawler initiates browser sessions.

When set to True, MediaCrawler creates a CDPBrowserManager instance from tools/cdp_browser.py and invokes its launch_and_connect method to attach to a real Chrome or Edge browser via the Chrome DevTools Protocol. This approach preserves the user's existing browser profile, including cookies, extensions, and login states, significantly improving anti-detection capabilities by making automated traffic appear indistinguishable from regular browsing.

When set to False, the crawler follows the standard Playwright path, launching a fresh, temporary Chromium instance that is isolated from the user's normal browsing environment.

What Headless Mode Controls

The headless flags (HEADLESS and CDP_HEADLESS) exclusively manage UI visibility and do not affect connection strategy.

• HEADLESS (defined in config/base_config.py, line 50): Applied when ENABLE_CDP_MODE is False, passed directly to chromium.launch(headless=config.HEADLESS, ...)
• CDP_HEADLESS (defined in config/base_config.py, line 73): Applied when ENABLE_CDP_MODE is True, forwarded to CDPBrowserManager.launch_and_connect(..., headless=config.CDP_HEADLESS)

These flags determine whether the browser renders a visible window. Running headless (True) consumes fewer resources but increases detection risk, as many sites flag headless browsers as suspicious automation.

Implementation in Platform Core Modules

Each platform implementation branches on ENABLE_CDP_MODE to select the appropriate launch strategy. In media_platform/zhihu/core.py (lines 58-90), the logic appears as:

if config.ENABLE_CDP_MODE:
    # CDP path - connects to existing Chrome

    browser_context = await self.launch_browser_with_cdp(
        playwright, playwright_proxy, self.user_agent,
        headless=config.CDP_HEADLESS,
    )
else:
    # Standard Playwright launch - creates new Chromium

    browser_context = await self.launch_browser(
        chromium, playwright_proxy, self.user_agent,
        headless=config.HEADLESS,
    )

The launch_browser_with_cdp method initializes the CDP connection, while launch_browser handles standard Playwright instantiation. Similar branching logic exists across all platform cores, including media_platform/xhs/core.py and others.

Configuration Patterns and Use Cases

Maximum Stealth (Recommended for Production)

# config/base_config.py

ENABLE_CDP_MODE = True      # Connect to real Chrome

CDP_HEADLESS = False        # Show the browser window

CDP_CONNECT_EXISTING = True # Reuse existing Chrome process

This configuration leverages the user's genuine browser profile and visible UI, minimizing detection probability. The crawler connects to an already-running Chrome instance rather than spawning a new process.

Lightweight Testing (Development Environment)

# config/base_config.py

ENABLE_CDP_MODE = False     # Use Playwright's Chromium

HEADLESS = True             # Run without visible window

AUTO_CLOSE_BROWSER = False  # Keep browser open for debugging

This pattern launches a fresh, headless browser suitable for rapid development iterations where anti-detection is not critical.

Headless CDP (Server Deployment)

# config/base_config.py

ENABLE_CDP_MODE = True
CDP_HEADLESS = True

Even when connecting via CDP, you can run Chrome in headless mode by starting the remote browser with the --headless flag. The CDPBrowserManager passes this configuration to playwright.chromium.connect_over_cdp as implemented in tools/cdp_browser.py (lines 13-31).

Anti-Detection and Resource Implications

Profile Persistence: CDP mode maintains persistent sessions across crawler runs because it operates within the user's actual browser profile. Standard Playwright mode creates temporary profiles that are deleted when AUTO_CLOSE_BROWSER triggers (unless explicitly disabled).

Process Management: When ENABLE_CDP_MODE is True and CDP_CONNECT_EXISTING is enabled, MediaCrawler attaches to an existing browser process rather than spawning new ones. This reduces memory overhead and avoids the process churn associated with launching fresh Chromium instances.

Detection Risk: Headless detection remains a concern regardless of connection mode. Even with CDP enabled, setting CDP_HEADLESS=True may trigger anti-bot measures on sophisticated platforms. The command-line argument parser in cmd_arg/arg.py exposes these flags independently, allowing runtime overrides without modifying configuration files.

Summary

• ENABLE_CDP_MODE switches between connecting to an existing Chrome via CDP (True) and launching a fresh Playwright Chromium instance (False)
• Headless flags only control UI visibility and are independent of connection mode
• CDP mode preserves user profiles, cookies, and extensions for superior anti-detection
• Standard Playwright mode creates isolated, temporary browser sessions suitable for contained scraping
• Configuration options are defined in config/base_config.py and applied in platform-specific core modules under media_platform/*/

Frequently Asked Questions

Can I use headless mode when ENABLE_CDP_MODE is enabled?

Yes. Set ENABLE_CDP_MODE = True and CDP_HEADLESS = True in config/base_config.py. The crawler will connect to your Chrome instance via CDP, but the remote Chrome must be started with headless flags (e.g., --headless=new). The CDP_HEADLESS parameter is passed to connect_over_cdp in tools/cdp_browser.py to ensure consistency.

Which configuration is best for avoiding bot detection?

Use ENABLE_CDP_MODE = True with CDP_HEADLESS = False and CDP_CONNECT_EXISTING = True. This connects to a visible Chrome instance with your real user profile, cookies, and extensions intact. Headless browsers are easily flagged by sophisticated anti-bot systems, while CDP-connected real browsers appear as legitimate user traffic.

Where do I configure these settings in MediaCrawler?

All browser configuration flags reside in config/base_config.py. Lines 50-73 contain the relevant settings: HEADLESS (line 50) for standard mode, ENABLE_CDP_MODE (line 59) for CDP toggle, and CDP_HEADLESS (line 73) for CDP visibility. You can also override these via command-line arguments defined in cmd_arg/arg.py.

What happens if I set HEADLESS=True but ENABLE_CDP_MODE=True?

The HEADLESS variable is ignored when ENABLE_CDP_MODE is active. In this state, only CDP_HEADLESS affects visibility. The platform core modules explicitly check config.ENABLE_CDP_MODE first and use the corresponding headless flag (CDP_HEADLESS vs HEADLESS) for that specific launch path.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}