How to Use MediaCrawler CDP Mode with an Existing Chrome Browser

MediaCrawler connects to an existing Chrome instance via the Chrome DevTools Protocol (CDP) to reuse logged-in sessions, cookies, and browser extensions, eliminating the overhead of launching fresh browser instances and significantly reducing detection risks.

MediaCrawler is an open-source crawling framework that supports CDP mode with existing Chrome to streamline data extraction from platforms like Xiaohongshu, Douyin, and others. By attaching to a browser you already have running with remote debugging enabled, you preserve authentication states and avoid repetitive login flows. This approach leverages the CDPBrowserManager class in tools/cdp_browser.py to manage the connection lifecycle, port detection, and context creation.

How CDP Mode Works in MediaCrawler

The CDP implementation centers on the CDPBrowserManager class located in tools/cdp_browser.py. This orchestrator handles two distinct modes: launching a fresh Playwright-managed browser or attaching to an existing Chrome instance. When configured for existing browser attachment, the manager bypasses the standard launch sequence and instead establishes a WebSocket connection to Chrome's remote debugging endpoint.

The workflow depends on two critical configuration flags in config/base_config.py:

• CDP_CONNECT_EXISTING: Boolean flag that switches between launching new browser (False) or connecting to existing (True)
• CDP_DEBUG_PORT: Integer specifying the port for the CDP connection (default: 9222)

Prerequisites: Enabling Remote Debugging in Chrome

Before configuring MediaCrawler, you must enable remote debugging on your Chrome instance. This exposes the Chrome DevTools Protocol on a local port, allowing external tools to control the browser.

1. Open Chrome and navigate to chrome://inspect/#remote-debugging
2. Enable "Allow remote debugging for this browser instance"
3. Verify the server is running at 127.0.0.1:9222 (or your custom port)

Verify the port is accessible using a simple socket probe:

import socket

def test_cdp_port(port=9222):
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.settimeout(5)
    result = s.connect_ex(('localhost', port))
    print('Port open - CDP ready' if result == 0 else 'Port closed')
    return result == 0

test_cdp_port()

Configuring MediaCrawler for Existing Chrome Connection

Edit config/base_config.py to enable CDP mode with your existing browser:

# config/base_config.py

CDP_CONNECT_EXISTING = True
CDP_DEBUG_PORT = 9222  # Change if using custom port

SAVE_LOGIN_STATE = True  # Optional: preserve cookies between sessions

Alternatively, set environment variables before running your crawler:

export CDP_CONNECT_EXISTING=true
export CDP_DEBUG_PORT=9222

The Connection Workflow Explained

When CDP_CONNECT_EXISTING is True, the launch_and_connect() method initiates a specific sequence defined in tools/cdp_browser.py:

Port Detection and Waiting

The _test_cdp_connection() method performs repeated socket probes against the configured debug port. If the connection fails initially, the manager logs progress and prompts you to verify remote debugging is enabled via chrome://inspect/#remote-debugging.

WebSocket URL Resolution

If direct CDP connection fails, the system falls back to _get_browser_websocket_url(), which queries the /json/version HTTP endpoint on the debug port to retrieve the proper WebSocket URL for Chrome DevTools Protocol communication.

Browser Context Creation

Once connected, _create_browser_context() either reuses the first existing context (if the browser already has tabs open) or creates a fresh context with these defaults:

• Viewport: 1920×1080
• Downloads enabled
• Accepts downloads from any origin

Managing Sessions, Cookies, and Stealth

The CDPBrowserManager provides helper methods to manage session state and evade detection:

• add_stealth_script(): Injects anti-detection scripts into the browser context
• add_cookies(): Programmatically inserts cookies into the existing Chrome session
• get_cookies(): Extracts current cookies for persistence

These methods operate on the browser context obtained from the existing Chrome instance, allowing you to synchronize login states between manual browsing and automated crawling.

Running the Crawler with Existing Chrome

Execute your crawler using the standard CLI interface while the configuration points to your existing browser:

uv run main.py --platform xhs --lt qrcode --type search

The CDPBrowserManager instance created in main.py automatically detects the CDP_CONNECT_EXISTING setting and attaches to your running Chrome rather than spawning a new process.

Cleanup and Resource Management

The cleanup() method in CDPBrowserManager handles resource termination intelligently. When attached to an external browser (CDP_CONNECT_EXISTING = True), it skips process termination and only closes the Playwright contexts it created, leaving your Chrome instance running for future use.

from tools.cdp_browser import CDPBrowserManager
import asyncio

async def safe_cleanup():
    manager = CDPBrowserManager()
    await manager.cleanup()  # Closes contexts, keeps Chrome running

asyncio.run(safe_cleanup())

Summary

• Enable remote debugging in Chrome via chrome://inspect/#remote-debugging before starting MediaCrawler
• Set CDP_CONNECT_EXISTING = True in config/base_config.py to switch from launch mode to attach mode
• Use CDP_DEBUG_PORT to specify the remote debugging port (default 9222)
• Leverage CDPBrowserManager in tools/cdp_browser.py to handle connection logic, context creation, and stealth injection
• Preserve browser state by connecting to existing Chrome profiles with saved logins and extensions
• Cleanup safely using the cleanup() method, which respects external browser processes

Frequently Asked Questions

How do I check if Chrome is ready for CDP connection?

Verify the debug port is accessible by testing socket connectivity to localhost:9222 (or your custom port). The CDPBrowserManager includes internal probing via _test_cdp_connection(), but you can manually check using the Python socket example provided above or by visiting http://localhost:9222/json/version in your browser.

Can I use Chrome profiles or extensions with CDP mode?

Yes. Connecting to an existing Chrome instance preserves all profile data, extensions, and cookies because MediaCrawler attaches to the running process rather than launching an isolated browser. This is the primary advantage of using CDP_CONNECT_EXISTING over standard headless mode.

What happens if the debug port is already in use?

If port 9222 is occupied, Chrome will fail to start remote debugging on that port. You must either free the port or configure MediaCrawler to use an alternative port by changing CDP_DEBUG_PORT in config/base_config.py and restarting Chrome with the new debug port (e.g., --remote-debugging-port=9223).

Does CDP mode work with Chromium or Edge browsers?

Yes. Any browser supporting the Chrome DevTools Protocol—including Chromium, Microsoft Edge, and Brave—can be used with MediaCrawler's CDP mode. Ensure the browser is launched with the --remote-debugging-port flag and that CDP_DEBUG_PORT matches your configuration.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}