How to Configure Playwright Proxy Settings with Different Proxy Providers in MediaCrawler

To configure Playwright proxy settings in MediaCrawler, enable ENABLE_IP_PROXY in config/base_config.py, set IP_PROXY_PROVIDER_NAME to your provider (kuaidaili, wandouhttp, or static), and let tools/crawler_util.py format the proxy dict for Playwright's proxy parameter.

MediaCrawler's browser automation relies on Playwright (and Chrome DevTools Protocol) to scrape content from platforms like Xiaohongshu and Douyin. When you need to route traffic through residential or data-center proxies, understanding how the codebase passes proxy credentials to Playwright is essential for reliable crawling.

This guide walks through the complete proxy configuration pipeline: from global settings in config/base_config.py to the provider-agnostic formatting logic in tools/crawler_util.py and final injection into Playwright's launch context.

Global Proxy Configuration in base_config.py

All proxy behavior starts in config/base_config.py. Three variables control everything:

# config/base_config.py (lines 33-42)

ENABLE_IP_PROXY = True                     # Master switch for proxy usage

IP_PROXY_PROVIDER_NAME = "kuaidaili"       # Provider: kuaidaili | wandouhttp | static

STATIC_PROXY_URL = "http://user:pwd@host:port"  # Only used when provider = "static"

Setting ENABLE_IP_PROXY = True activates the proxy subsystem. The IP_PROXY_PROVIDER_NAME string determines which acquisition strategy runs:

• kuaidaili — Fetches rotating IPs from KuaiDaili's REST API
• wandouhttp — Pulls from WandouHTTP's proxy pool
• static — Uses the hardcoded STATIC_PROXY_URL without external calls

For static configurations, include credentials directly in the URL: http://username:password@proxy.example.com:8080.

Converting Provider Data to Playwright Format with format_proxy_info()

MediaCrawler decouples proxy acquisition from Playwright's expected interface through format_proxy_info() in tools/crawler_util.py. This function accepts an IpInfoModel (regardless of source) and outputs two formats:

# tools/crawler_util.py (lines 89-102)

server = f"{ip_proxy_info.ip}:{ip_proxy_info.port}"
playwright_proxy = {"server": server}
if ip_proxy_info.user and ip_proxy_info.password:
    playwright_proxy["username"] = ip_proxy_info.user
    playwright_proxy["password"] = ip_proxy_info.password

The Playwright proxy dict follows Playwright's specification: {"server": "host:port", "username": "...", "password": "..."}. The function also returns an httpx-compatible URL string for raw HTTP requests outside browser automation.

This design means you can swap providers without touching browser launch code—the same IpInfoModel abstraction normalizes KuaiDaili, WandouHTTP, and static inputs identically.

Injecting Proxies into Playwright Browser Contexts

The formatted proxy dict reaches Playwright through tools/cdp_browser.py. When launching persistent or ephemeral browser contexts, the proxy parameter attaches directly:

# Simplified from tools/cdp_browser.py

async def launch_browser(self, playwright: Playwright):
    # Obtain proxy dict if proxy is enabled globally

    playwright_proxy, _ = format_proxy_info(proxy_ip_info) if ENABLE_IP_PROXY else (None, None)
    
    context = await playwright.chromium.launch_persistent_context(
        user_data_dir=USER_DATA_DIR,
        headless=HEADLESS,
        proxy=playwright_proxy,  # ← Proxy injected here

        # ... other options

    )

The same proxy configuration works for CDP connections. If you're using the default CDP mode, tools/cdp_browser.py handles proxy propagation automatically—you never modify launch code manually.

Provider-Specific Configuration Patterns

Each proxy provider follows a distinct acquisition path before converging at format_proxy_info():

Provider	Data Source	Implementation Notes
kuaidaili	KuaiDaili REST API	Dynamically imported from proxy/kuaidaili_provider.py; handles free/paid tiers with automatic authentication
wandouhttp	WandouHTTP proxy pool	Imported from proxy/wandouhttp_provider.py; optimized for high-volume HTTP/SOCKS5 rotation
static	STATIC_PROXY_URL string	Parsed directly in tools/crawler_util.py when IP_PROXY_PROVIDER_NAME == "static"; zero network overhead

For static proxies, simply populate STATIC_PROXY_URL and set IP_PROXY_PROVIDER_NAME = "static". No API keys, no external dependencies—ideal for dedicated proxy servers or local debugging.

Complete Configuration Example

Here's a working configuration for three common scenarios:

1. KuaiDaili Dynamic Rotation

# config/base_config.py

ENABLE_IP_PROXY = True
IP_PROXY_PROVIDER_NAME = "kuaidaili"
STATIC_PROXY_URL = ""  # Ignored

# Ensure your KuaiDaili API credentials are available in environment

# or the provider module's configuration

2. WandouHTTP Pool

# config/base_config.py

ENABLE_IP_PROXY = True
IP_PROXY_PROVIDER_NAME = "wandouhttp"
STATIC_PROXY_URL = ""

3. Static Proxy with Authentication

# config/base_config.py

ENABLE_IP_PROXY = True
IP_PROXY_PROVIDER_NAME = "static"
STATIC_PROXY_URL = "http://proxyuser:proxypass@123.45.67.89:3128"

Verifying Proxy Functionality

Test your configuration with this standalone script:

from tools.crawler_util import format_proxy_info
from proxy.proxy_ip_pool import IpInfoModel
import asyncio
from playwright.async_api import async_playwright

async def verify_proxy():
    # Simulate provider output (normally fetched automatically)

    proxy_info = IpInfoModel(
        ip="123.45.67.89",
        port=3128,
        user="proxyuser",
        password="proxypass"
    )
    
    playwright_proxy, _ = format_proxy_info(proxy_info)
    print(f"Playwright proxy dict: {playwright_proxy}")
    # Output: {'server': '123.45.67.89:3128', 'username': 'proxyuser', 'password': 'proxypass'}

    
    async with async_playwright() as p:
        browser = await p.chromium.launch(
            headless=False,
            proxy=playwright_proxy
        )
        page = await browser.new_page()
        await page.goto("https://httpbin.org/ip")
        await page.screenshot(path="proxy_verification.png")
        await browser.close()

asyncio.run(verify_proxy())

Troubleshooting Common Proxy Issues

Authentication failures usually indicate missing credentials in IpInfoModel. Verify that user and password fields propagate from your provider response—format_proxy_info() only includes auth keys when both are non-empty.

Connection timeouts when using rotating proxies often mean the acquired IP expired between fetching and browser launch. The KuaiDaili and WandouHTTP modules typically implement fresh-IP acquisition per session; check provider-specific retry logic if you see intermittent failures.

Static proxy not applying almost always traces to IP_PROXY_PROVIDER_NAME not being set to "static". The codebase ignores STATIC_PROXY_URL for dynamic providers even when the variable contains a valid URL.

Summary

• Enable globally with ENABLE_IP_PROXY = True in config/base_config.py
• Choose provider via IP_PROXY_PROVIDER_NAME: kuaidaili, wandouhttp, or static
• Normalize formats through tools/crawler_util.py's format_proxy_info() function
• Inject automatically into Playwright contexts by tools/cdp_browser.py without manual launch code changes
• Static proxies require only STATIC_PROXY_URL populated; no external API calls occur

Frequently Asked Questions

How do I switch from KuaiDaili to a static proxy in MediaCrawler?

Change IP_PROXY_PROVIDER_NAME from "kuaidaili" to "static" in config/base_config.py, then set STATIC_PROXY_URL to your full proxy URL including credentials. No other file modifications are needed—the same format_proxy_info() path handles both providers.

Does MediaCrawler support SOCKS5 proxies with Playwright?

The format_proxy_info() function accepts any ip:port combination. For SOCKS5, ensure your Playwright installation has SOCKS support enabled and prefix the server with socks5:// in the server field if your provider returns it. The static URL parser preserves protocol prefixes when present.

Where is the proxy actually attached to the browser instance?

In tools/cdp_browser.py, the launch helpers pass the playwright_proxy dict to playwright.chromium.launch_persistent_context() or equivalent launch methods via the proxy= keyword argument. This occurs after format_proxy_info() processes the provider-specific IP information.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}