# MediaCrawler Headless Browser Options: Playwright and CDP Configuration Guide

> Explore MediaCrawler's headless browser options: Playwright and CDP. Configure headless mode using config settings or CLI flags for efficient web scraping.

- Repository: [程序员阿江-Relakkes/MediaCrawler](https://github.com/NanmiCoder/MediaCrawler)
- Tags: how-to-guide
- Published: 2026-06-27

---

**MediaCrawler supports two distinct headless browser options—Playwright headless mode and Chrome DevTools Protocol (CDP) headless mode—controlled via `config.HEADLESS`, `config.CDP_HEADLESS`, or the unified `--headless` CLI flag.**

MediaCrawler is an open-source social media scraping framework that provides flexible **headless browser options** for running web automation without a graphical display. Understanding these configuration settings allows you to deploy crawlers in server environments, CI pipelines, or Docker containers where no GUI is available. This guide examines the specific implementation details found in the NanmiCoder/MediaCrawler repository.

## Available Headless Browser Modes

### Playwright Headless Mode

The primary headless browser option leverages Playwright's native `headless=True` argument when launching Chromium. In [`base/base_crawler.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/base/base_crawler.py), the `launch_browser` method checks the `config.HEADLESS` flag before initializing the browser context. When enabled, Playwright runs the browser in the background without rendering a visible window, making it ideal for automated scraping tasks across platforms like Zhihu, Xiaohongshu, and others.

### Chrome DevTools Protocol (CDP) Headless Mode

For scenarios requiring direct Chrome control, MediaCrawler offers a CDP-based headless mode. The `launch_browser_with_cdp` method in [`base/base_crawler.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/base/base_crawler.py) respects the `config.CDP_HEADLESS` setting, passing the headless flag through the Chrome DevTools Protocol connection. This mode provides lower-level access to browser internals while maintaining background execution capabilities.

## Configuration Flags and Command-Line Arguments

MediaCrawler exposes headless browser options through two configuration constants and a unified CLI argument:

- **`config.HEADLESS`**: Controls Playwright headless mode in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py)
- **`config.CDP_HEADLESS`**: Controls CDP headless mode in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py)  
- **`--headless`**: A command-line flag defined in [`cmd_arg/arg.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/cmd_arg/arg.py) that synchronizes both settings simultaneously

When you pass `--headless true` via CLI, both boolean flags are set to `True`, ensuring consistent behavior across Playwright and CDP execution paths.

## BrowserLauncher Implementation Details

The `BrowserLauncher` class in [`tools/browser_launcher.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/browser_launcher.py) constructs the actual Chrome command-line arguments. When `headless=True` is passed to `launch_browser()`, it appends the modern `--headless=new` switch to the Chrome executable invocation. This approach ensures compatibility with recent Chrome versions while avoiding the deprecated default headless behavior.

## Code Examples

### Enable Headless Mode via Command Line

```bash
python -m MediaCrawler.main --headless true

```

### Configure Playwright Headless Programmatically

```python
from playwright.async_api import async_playwright

async def run():
    async with async_playwright() as pw:
        chromium = pw.chromium
        # headless=True activates Playwright's headless mode

        browser = await chromium.launch(headless=True)
        context = await browser.new_context()
        # Use context for crawling

```

### Launch CDP Headless Browser

```python
from playwright.async_api import async_playwright

async def run_cdp():
    async with async_playwright() as pw:
        chromium = pw.chromium
        # Pass headless flag to CDP launch routine

        browser = await chromium.launch(headless=True, channel="chrome")
        context = await browser.new_context()
        # Use context for crawling

```

### Direct BrowserLauncher Usage

```python
from tools.browser_launcher import BrowserLauncher

launcher = BrowserLauncher()

# Starts Chrome with the modern --headless=new flag

launcher.launch_browser(
    browser_path="/usr/bin/google-chrome",
    debug_port=9222,
    headless=True,
)

```

## Summary

- MediaCrawler provides two **headless browser options**: Playwright mode (`config.HEADLESS`) and CDP mode (`config.CDP_HEADLESS`)
- The `BrowserLauncher` class in [`tools/browser_launcher.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/browser_launcher.py) implements the `--headless=new` Chrome flag for modern compatibility
- Use the `--headless` CLI argument to enable both modes simultaneously across all platforms
- Both modes are implemented in [`base/base_crawler.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/base/base_crawler.py) via the `launch_browser` and `launch_browser_with_cdp` methods

## Frequently Asked Questions

### What is the difference between Playwright headless and CDP headless in MediaCrawler?

Playwright headless mode uses Playwright's high-level API with `headless=True` passed to `chromium.launch()`, while CDP headless mode connects directly to Chrome via the Chrome DevTools Protocol. The CDP approach in [`base/base_crawler.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/base/base_crawler.py) offers more granular control over browser internals but requires `config.CDP_HEADLESS` to be set separately in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py).

### How do I enable headless mode for all MediaCrawler platforms at once?

Use the `--headless true` command-line argument when running `python -m MediaCrawler.main`. This flag, parsed in [`cmd_arg/arg.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/cmd_arg/arg.py), automatically sets both `config.HEADLESS` and `config.CDP_HEADLESS` to `True`, ensuring consistent headless behavior across all supported media platforms without modifying configuration files.

### Which Chrome flag does MediaCrawler use for headless execution?

According to [`tools/browser_launcher.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/browser_launcher.py), MediaCrawler appends `--headless=new` to the Chrome command line when `headless=True` is specified. This modern flag replaces the older default headless mode and provides better compatibility with extensions and modern web features compared to the legacy headless implementation.

### Can I run MediaCrawler headless in Docker or CI environments?

Yes, both headless browser options are designed for environments without displays. Configure `config.HEADLESS=True` and `config.CDP_HEADLESS=True` in your [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py) or use the `--headless` CLI flag to run entirely in the background without requiring X11, xvfb, or display servers.