How to Switch Between Headless and Non-Headless Browser Modes in MediaCrawler

MediaCrawler controls headless browser operation through two configuration flags (HEADLESS for standard mode, CDP_HEADLESS for CDP mode) that can be set via CLI argument, config file edits, or programmatic override.

The NanmiCoder/MediaCrawler repository provides flexible control over whether Chromium runs with a visible UI or in the background. Understanding how to toggle these modes is essential for debugging automation scripts versus running production crawlers at scale. This guide covers all three methods to switch between headless and non-headless browser modes in MediaCrawler, with direct references to the source implementation.

Understanding the Dual Flag Architecture

MediaCrawler maintains separate configuration flags for its two browser launch pathways:

Flag	Purpose	Default
config.HEADLESS	Controls standard browser launches (new Playwright instance)	False
config.CDP_HEADLESS	Controls CDP mode (attaches to existing Chrome/Edge via Chrome DevTools Protocol)	False

Both flags reside in config/base_config.py and are consumed by platform-specific core classes. In media_platform/zhihu/core.py and similar platform cores, the launch call passes the flag directly:

# Standard mode launch

self.browser_context = await self.launch_browser(
    chromium, None, self.user_agent, headless=config.HEADLESS
)

# CDP mode launch

self.browser_context = await self.launch_browser_with_cdp(
    playwright, proxy_fmt, self.user_agent, headless=config.CDP_HEADLESS
)

The actual browser behavior is implemented in tools/browser_launcher.py. When headless=True, the launcher appends --headless=new to the Chromium command line and disables GPU acceleration. When headless=False, it adds --start-maximized to open a visible browser window.

Method 1: Switch Browser Modes via CLI Argument

The fastest way to toggle headless mode is through MediaCrawler's command-line interface. The --headless argument updates both flags simultaneously, ensuring consistent behavior across standard and CDP launches.

# Enable headless mode for any platform

python main.py --platform zhihu --headless true

# Explicitly disable headless (matches default behavior)

python main.py --platform douyin --headless false

The argument parser in cmd_arg/arg.py handles this logic:

config.HEADLESS = enable_headless
config.CDP_HEADLESS = enable_headless

This method is ideal for CI/CD pipelines and scheduled jobs where you want headless operation, or for quick debugging sessions where you need to watch the browser interact with pages.

Method 2: Switch Browser Modes via Config File Edit

For persistent default behavior, modify config/base_config.py directly:

# config/base_config.py

# Standard mode: True = headless, False = visible window

HEADLESS = True

# CDP mode: True = headless, False = visible window

CDP_HEADLESS = True

After saving changes, run MediaCrawler without the --headless flag:

python main.py --platform xhs

This approach suits development environments where you consistently want one mode or the other. Note that CLI arguments override these defaults when both are specified.

Method 3: Programmatic Override in Python Scripts

For dynamic control within custom automation workflows, import and modify base_config before initializing any crawler:

from config import base_config as config
from media_platform.bilibili.core import BilibiliCrawler

# Force headless execution for this specific run

config.HEADLESS = True
config.CDP_HEADLESS = True

crawler = BilibiliCrawler()
await crawler.run()

This pattern enables runtime conditional logic—for example, running headless in production but visible during local development based on environment detection:

import os
from config import base_config as config

# Auto-detect mode from environment variable

is_production = os.getenv("ENV") == "production"
config.HEADLESS = is_production
config.CDP_HEADLESS = is_production

Key Source Files Reference

File	Function
config/base_config.py	Defines HEADLESS and CDP_HEADLESS default values
cmd_arg/arg.py	Parses --headless CLI flag and syncs both config values
tools/browser_launcher.py	Constructs launch command with --headless=new or --start-maximized
tools/cdp_browser.py	Forwards headless parameter to BrowserLauncher for CDP connections
media_platform/*/core.py	Platform cores that read config flags and invoke launchers

Summary

• MediaCrawler uses two independent flags (HEADLESS for standard mode, CDP_HEADLESS for CDP mode) to control browser visibility
• The --headless CLI argument in cmd_arg/arg.py updates both flags simultaneously for convenience
• Direct edits to config/base_config.py provide persistent defaults across runs
• Programmatic import and modification of base_config enables dynamic mode switching at runtime
• The actual Chromium launch command construction happens in tools/browser_launcher.py, which translates boolean flags to --headless=new or --start-maximized switches

Frequently Asked Questions

What is the difference between HEADLESS and CDP_HEADLESS?

HEADLESS controls standard Playwright browser launches where MediaCrawler spins up a fresh Chromium instance. CDP_HEADLESS controls CDP mode where MediaCrawler attaches to an already-running Chrome or Edge browser via the Chrome DevTools Protocol. Most users should set both to the same value, which the --headless CLI flag handles automatically.

Can I run headless for some platforms and visible for others?

Yes, but not through the CLI alone. Set config.HEADLESS and config.CDP_HEADLESS programmatically before instantiating each platform's crawler class. The flags are global, so you must change them between crawler initializations if you need mixed modes in the same script.

Does headless mode affect detection by anti-bot systems?

Headless detection varies by target site. Some platforms employ headless-specific fingerprinting that --headless=new may trigger. MediaCrawler's tools/browser_launcher.py mitigates common detection vectors by disabling GPU and other headless-indicative features, but visible mode (HEADLESS=False) generally presents a more authentic browser fingerprint when facing sophisticated bot detection.

What Chromium version does MediaCrawler use in headless mode?

MediaCrawler uses Playwright's bundled Chromium by default in standard mode, or your system Chrome/Edge in CDP mode. The --headless=new flag (as implemented in tools/browser_launcher.py) invokes Chromium's modern headless implementation rather than the deprecated legacy headless mode.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}