How to Configure a Custom User Agent and Browser Fingerprint in MediaCrawler

Use the user_agent parameter in CDPBrowserManager.launch_and_connect() to set a custom user agent, then extend context_options in _create_browser_context() to modify additional fingerprint attributes like viewport, locale, and timezone.

MediaCrawler leverages Playwright's Chromium DevTools Protocol (CDP) interface for browser automation. Customizing the browser fingerprint—including the user agent, viewport dimensions, and locale settings—is essential for avoiding detection on sites that analyze client characteristics. This guide walks you through the exact methods and configuration points in the NanmiCoder/MediaCrawler source code.

Where User Agent and Fingerprint Configuration Lives

The core browser management logic resides in tools/cdp_browser.py. Understanding this file's structure is key to effective fingerprint customization.

The launch_and_connect() Entry Point

The launch_and_connect() method accepts a user_agent argument directly. This is the primary injection point for custom user agent strings.

# Source: tools/cdp_browser.py, lines 101-104

async def launch_and_connect(
    self,
    playwright: Playwright,
    user_agent: str = None,  # <-- Custom user agent parameter

    ...
):

When provided, this value propagates through to context creation.

The _create_browser_context() Implementation

Inside _create_browser_context(), the user_agent value is merged into the browser context options dictionary:

# Source: tools/cdp_browser.py, lines 84-86

context_options = {
    "user_agent": user_agent,  # Applied if not None

    ...
}

The default viewport is hardcoded to 1920 × 1080 at lines 79-81:

viewport={
    "width": 1920,
    "height": 1080
}

Complete Configuration Example

Below is a fully functional script that demonstrates custom user agent and extended fingerprint configuration when using MediaCrawler's CDPBrowserManager.

import asyncio
from playwright.async_api import async_playwright
from tools.cdp_browser import CDPBrowserManager

async def run():
    # Define the custom fingerprint you want

    custom_user_agent = (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/124.0.0.0 Safari/537.36"
    )
    
    # Additional fingerprint fields (optional)

    extra_context_options = {
        "locale": "en-US",
        "timezone_id": "America/New_York",
        "device_scale_factor": 2,
    }

    async with async_playwright() as playwright:
        cdp_manager = CDPBrowserManager()
        
        # Launch the browser and pass the custom user-agent

        browser_context = await cdp_manager.launch_and_connect(
            playwright,
            user_agent=custom_user_agent,
            headless=False,          # set to True for headless mode

        )

        # Dynamically adjust viewport after context creation

        await browser_context.set_viewport_size(
            {"width": 1366, "height": 768}
        )
        await browser_context.grant_permissions(["geolocation"])

        page = await browser_context.new_page()
        await page.goto(
            "https://www.whatismybrowser.com/detect/what-is-my-user-agent"
        )
        await asyncio.sleep(5)
        await cdp_manager.cleanup()

asyncio.run(run())

Extending Browser Fingerprint Beyond User Agent

For more comprehensive fingerprint customization, you can modify the context_options dictionary in _create_browser_context() before the context is instantiated. Playwright supports numerous fingerprint-related options:

Option	Purpose	Example Value
locale	Browser language	"en-US", "zh-CN"
timezone_id	System timezone	"America/New_York", "Asia/Shanghai"
device_scale_factor	HiDPI simulation	1, 2, 3
color_scheme	Light/dark mode	"light", "dark"
geolocation	Location spoofing	{"latitude": 40.7, "longitude": -74.0}

To apply these permanently, create a subclass or patch the _create_browser_context() method to merge your custom options with the defaults.

Externalizing Configuration

To keep fingerprint settings out of your main codebase, implement a configuration loader that feeds values into launch_and_connect():

import json
from pathlib import Path

def load_fingerprint_config(path: str = "config/fingerprint.json"):
    """Load fingerprint settings from external JSON file."""
    config_file = Path(path)
    if config_file.exists():
        return json.loads(config_file.read_text())
    return {}

# Usage in your crawler script

config = load_fingerprint_config()
context = await cdp_manager.launch_and_connect(
    playwright,
    user_agent=config.get("user_agent"),
    headless=config.get("headless", False),
)

Key Files Reference

Understanding where configuration flows through MediaCrawler helps with maintenance and debugging:

File	Role	Key Sections
tools/cdp_browser.py	Core CDP browser manager	launch_and_connect() signature at lines 101-104; _create_browser_context() user-agent handling at lines 84-86; default viewport at lines 79-81
config/base_config.py	Default configuration flags	Contains CDP_CONNECT_EXISTING and AUTO_CLOSE_BROWSER settings affecting browser startup behavior
Your entry script (e.g., main.py)	Instantiation point	Where you construct CDPBrowserManager and pass custom parameters

Summary

• Primary method: Pass user_agent to CDPBrowserManager.launch_and_connect() in tools/cdp_browser.py.
• Extended fingerprint: Modify context_options in _create_browser_context() to set viewport, locale, timezone, and other Playwright-supported attributes.
• Default viewport: 1920 × 1080 as defined in lines 79-81 of tools/cdp_browser.py.
• External config: Wrap the manager in a loader that reads from JSON, YAML, or environment variables to avoid hardcoding values.

Frequently Asked Questions

How does MediaCrawler apply the custom user agent to the browser?

MediaCrawler passes the user_agent parameter through launch_and_connect() into _create_browser_context(), where it becomes part of the Playwright browser context options dictionary. Playwright then applies this string to all requests originating from that context.

Can I change the browser fingerprint after the context is created?

Certain attributes like viewport size can be modified dynamically using browser_context.set_viewport_size(). However, core fingerprint elements including user agent, locale, and timezone must be set at context creation time. To change these, create a new context with updated options.

What is the default browser fingerprint in MediaCrawler?

The default configuration uses a viewport of 1920 × 1080 and relies on Playwright's default Chromium user agent unless overridden. No explicit locale, timezone, or device scale factor is set by default, causing the browser to use system-derived values.

Does MediaCrawler support mobile browser fingerprint emulation?

Yes—through Playwright's standard capabilities. Pass a mobile user agent string and set device_scale_factor appropriately in context_options. For full device emulation, use Playwright's built-in device descriptors via playwright.devices and merge the profile into your context options before calling launch_and_connect().

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}