MediaCrawler Headless Browser Options: Playwright and CDP Configuration Guide

MediaCrawler supports two distinct headless browser options—Playwright headless mode and Chrome DevTools Protocol (CDP) headless mode—controlled via config.HEADLESS, config.CDP_HEADLESS, or the unified --headless CLI flag.

MediaCrawler is an open-source social media scraping framework that provides flexible headless browser options for running web automation without a graphical display. Understanding these configuration settings allows you to deploy crawlers in server environments, CI pipelines, or Docker containers where no GUI is available. This guide examines the specific implementation details found in the NanmiCoder/MediaCrawler repository.

Available Headless Browser Modes

Playwright Headless Mode

The primary headless browser option leverages Playwright's native headless=True argument when launching Chromium. In base/base_crawler.py, the launch_browser method checks the config.HEADLESS flag before initializing the browser context. When enabled, Playwright runs the browser in the background without rendering a visible window, making it ideal for automated scraping tasks across platforms like Zhihu, Xiaohongshu, and others.

Chrome DevTools Protocol (CDP) Headless Mode

For scenarios requiring direct Chrome control, MediaCrawler offers a CDP-based headless mode. The launch_browser_with_cdp method in base/base_crawler.py respects the config.CDP_HEADLESS setting, passing the headless flag through the Chrome DevTools Protocol connection. This mode provides lower-level access to browser internals while maintaining background execution capabilities.

Configuration Flags and Command-Line Arguments

MediaCrawler exposes headless browser options through two configuration constants and a unified CLI argument:

• config.HEADLESS: Controls Playwright headless mode in config/base_config.py
• config.CDP_HEADLESS: Controls CDP headless mode in config/base_config.py
• --headless: A command-line flag defined in cmd_arg/arg.py that synchronizes both settings simultaneously

When you pass --headless true via CLI, both boolean flags are set to True, ensuring consistent behavior across Playwright and CDP execution paths.

BrowserLauncher Implementation Details

The BrowserLauncher class in tools/browser_launcher.py constructs the actual Chrome command-line arguments. When headless=True is passed to launch_browser(), it appends the modern --headless=new switch to the Chrome executable invocation. This approach ensures compatibility with recent Chrome versions while avoiding the deprecated default headless behavior.

Code Examples

Enable Headless Mode via Command Line

python -m MediaCrawler.main --headless true

Configure Playwright Headless Programmatically

from playwright.async_api import async_playwright

async def run():
    async with async_playwright() as pw:
        chromium = pw.chromium
        # headless=True activates Playwright's headless mode

        browser = await chromium.launch(headless=True)
        context = await browser.new_context()
        # Use context for crawling

Launch CDP Headless Browser

from playwright.async_api import async_playwright

async def run_cdp():
    async with async_playwright() as pw:
        chromium = pw.chromium
        # Pass headless flag to CDP launch routine

        browser = await chromium.launch(headless=True, channel="chrome")
        context = await browser.new_context()
        # Use context for crawling

Direct BrowserLauncher Usage

from tools.browser_launcher import BrowserLauncher

launcher = BrowserLauncher()

# Starts Chrome with the modern --headless=new flag

launcher.launch_browser(
    browser_path="/usr/bin/google-chrome",
    debug_port=9222,
    headless=True,
)

Summary

• MediaCrawler provides two headless browser options: Playwright mode (config.HEADLESS) and CDP mode (config.CDP_HEADLESS)
• The BrowserLauncher class in tools/browser_launcher.py implements the --headless=new Chrome flag for modern compatibility
• Use the --headless CLI argument to enable both modes simultaneously across all platforms
• Both modes are implemented in base/base_crawler.py via the launch_browser and launch_browser_with_cdp methods

Frequently Asked Questions

What is the difference between Playwright headless and CDP headless in MediaCrawler?

Playwright headless mode uses Playwright's high-level API with headless=True passed to chromium.launch(), while CDP headless mode connects directly to Chrome via the Chrome DevTools Protocol. The CDP approach in base/base_crawler.py offers more granular control over browser internals but requires config.CDP_HEADLESS to be set separately in config/base_config.py.

How do I enable headless mode for all MediaCrawler platforms at once?

Use the --headless true command-line argument when running python -m MediaCrawler.main. This flag, parsed in cmd_arg/arg.py, automatically sets both config.HEADLESS and config.CDP_HEADLESS to True, ensuring consistent headless behavior across all supported media platforms without modifying configuration files.

Which Chrome flag does MediaCrawler use for headless execution?

According to tools/browser_launcher.py, MediaCrawler appends --headless=new to the Chrome command line when headless=True is specified. This modern flag replaces the older default headless mode and provides better compatibility with extensions and modern web features compared to the legacy headless implementation.

Can I run MediaCrawler headless in Docker or CI environments?

Yes, both headless browser options are designed for environments without displays. Configure config.HEADLESS=True and config.CDP_HEADLESS=True in your config/base_config.py or use the --headless CLI flag to run entirely in the background without requiring X11, xvfb, or display servers.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}