# How to Configure Playwright Proxy Settings with Different Proxy Providers in MediaCrawler

> Learn to configure Playwright proxy settings in MediaCrawler. Easily integrate with providers like kuaidaili, wandouhttp, or static IPs by enabling proxy settings and specifying your provider name.

- Repository: [程序员阿江-Relakkes/MediaCrawler](https://github.com/NanmiCoder/MediaCrawler)
- Tags: how-to-guide
- Published: 2026-08-14

---

**To configure Playwright proxy settings in MediaCrawler, enable `ENABLE_IP_PROXY` in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py), set `IP_PROXY_PROVIDER_NAME` to your provider (kuaidaili, wandouhttp, or static), and let [`tools/crawler_util.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/crawler_util.py) format the proxy dict for Playwright's `proxy` parameter.**

MediaCrawler's browser automation relies on Playwright (and Chrome DevTools Protocol) to scrape content from platforms like Xiaohongshu and Douyin. When you need to route traffic through residential or data-center proxies, understanding how the codebase passes proxy credentials to Playwright is essential for reliable crawling.

This guide walks through the complete proxy configuration pipeline: from global settings in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py) to the provider-agnostic formatting logic in [`tools/crawler_util.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/crawler_util.py) and final injection into Playwright's launch context.

## Global Proxy Configuration in base_config.py

All proxy behavior starts in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py). Three variables control everything:

```python

# config/base_config.py (lines 33-42)

ENABLE_IP_PROXY = True                     # Master switch for proxy usage

IP_PROXY_PROVIDER_NAME = "kuaidaili"       # Provider: kuaidaili | wandouhttp | static

STATIC_PROXY_URL = "http://user:pwd@host:port"  # Only used when provider = "static"

```

Setting `ENABLE_IP_PROXY = True` activates the proxy subsystem. The `IP_PROXY_PROVIDER_NAME` string determines which acquisition strategy runs:

- **kuaidaili** — Fetches rotating IPs from KuaiDaili's REST API
- **wandouhttp** — Pulls from WandouHTTP's proxy pool
- **static** — Uses the hardcoded `STATIC_PROXY_URL` without external calls

For static configurations, include credentials directly in the URL: `http://username:password@proxy.example.com:8080`.

## Converting Provider Data to Playwright Format with format_proxy_info()

MediaCrawler decouples proxy acquisition from Playwright's expected interface through `format_proxy_info()` in [`tools/crawler_util.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/crawler_util.py). This function accepts an `IpInfoModel` (regardless of source) and outputs two formats:

```python

# tools/crawler_util.py (lines 89-102)

server = f"{ip_proxy_info.ip}:{ip_proxy_info.port}"
playwright_proxy = {"server": server}
if ip_proxy_info.user and ip_proxy_info.password:
    playwright_proxy["username"] = ip_proxy_info.user
    playwright_proxy["password"] = ip_proxy_info.password

```

The **Playwright proxy dict** follows Playwright's specification: `{"server": "host:port", "username": "...", "password": "..."}`. The function also returns an **httpx-compatible URL string** for raw HTTP requests outside browser automation.

This design means you can swap providers without touching browser launch code—the same `IpInfoModel` abstraction normalizes KuaiDaili, WandouHTTP, and static inputs identically.

## Injecting Proxies into Playwright Browser Contexts

The formatted proxy dict reaches Playwright through [`tools/cdp_browser.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/cdp_browser.py). When launching persistent or ephemeral browser contexts, the proxy parameter attaches directly:

```python

# Simplified from tools/cdp_browser.py

async def launch_browser(self, playwright: Playwright):
    # Obtain proxy dict if proxy is enabled globally

    playwright_proxy, _ = format_proxy_info(proxy_ip_info) if ENABLE_IP_PROXY else (None, None)
    
    context = await playwright.chromium.launch_persistent_context(
        user_data_dir=USER_DATA_DIR,
        headless=HEADLESS,
        proxy=playwright_proxy,  # ← Proxy injected here

        # ... other options

    )

```

The same proxy configuration works for CDP connections. If you're using the default CDP mode, [`tools/cdp_browser.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/cdp_browser.py) handles proxy propagation automatically—you never modify launch code manually.

## Provider-Specific Configuration Patterns

Each proxy provider follows a distinct acquisition path before converging at `format_proxy_info()`:

| Provider | Data Source | Implementation Notes |
|----------|-------------|----------------------|
| **kuaidaili** | KuaiDaili REST API | Dynamically imported from [`proxy/kuaidaili_provider.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/proxy/kuaidaili_provider.py); handles free/paid tiers with automatic authentication |
| **wandouhttp** | WandouHTTP proxy pool | Imported from [`proxy/wandouhttp_provider.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/proxy/wandouhttp_provider.py); optimized for high-volume HTTP/SOCKS5 rotation |
| **static** | `STATIC_PROXY_URL` string | Parsed directly in [`tools/crawler_util.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/crawler_util.py) when `IP_PROXY_PROVIDER_NAME == "static"`; zero network overhead |

For **static proxies**, simply populate `STATIC_PROXY_URL` and set `IP_PROXY_PROVIDER_NAME = "static"`. No API keys, no external dependencies—ideal for dedicated proxy servers or local debugging.

## Complete Configuration Example

Here's a working configuration for three common scenarios:

### 1. KuaiDaili Dynamic Rotation

```python

# config/base_config.py

ENABLE_IP_PROXY = True
IP_PROXY_PROVIDER_NAME = "kuaidaili"
STATIC_PROXY_URL = ""  # Ignored

# Ensure your KuaiDaili API credentials are available in environment

# or the provider module's configuration

```

### 2. WandouHTTP Pool

```python

# config/base_config.py

ENABLE_IP_PROXY = True
IP_PROXY_PROVIDER_NAME = "wandouhttp"
STATIC_PROXY_URL = ""

```

### 3. Static Proxy with Authentication

```python

# config/base_config.py

ENABLE_IP_PROXY = True
IP_PROXY_PROVIDER_NAME = "static"
STATIC_PROXY_URL = "http://proxyuser:proxypass@123.45.67.89:3128"

```

### Verifying Proxy Functionality

Test your configuration with this standalone script:

```python
from tools.crawler_util import format_proxy_info
from proxy.proxy_ip_pool import IpInfoModel
import asyncio
from playwright.async_api import async_playwright

async def verify_proxy():
    # Simulate provider output (normally fetched automatically)

    proxy_info = IpInfoModel(
        ip="123.45.67.89",
        port=3128,
        user="proxyuser",
        password="proxypass"
    )
    
    playwright_proxy, _ = format_proxy_info(proxy_info)
    print(f"Playwright proxy dict: {playwright_proxy}")
    # Output: {'server': '123.45.67.89:3128', 'username': 'proxyuser', 'password': 'proxypass'}

    
    async with async_playwright() as p:
        browser = await p.chromium.launch(
            headless=False,
            proxy=playwright_proxy
        )
        page = await browser.new_page()
        await page.goto("https://httpbin.org/ip")
        await page.screenshot(path="proxy_verification.png")
        await browser.close()

asyncio.run(verify_proxy())

```

## Troubleshooting Common Proxy Issues

**Authentication failures** usually indicate missing credentials in `IpInfoModel`. Verify that `user` and `password` fields propagate from your provider response—`format_proxy_info()` only includes auth keys when both are non-empty.

**Connection timeouts** when using rotating proxies often mean the acquired IP expired between fetching and browser launch. The KuaiDaili and WandouHTTP modules typically implement fresh-IP acquisition per session; check provider-specific retry logic if you see intermittent failures.

**Static proxy not applying** almost always traces to `IP_PROXY_PROVIDER_NAME` not being set to `"static"`. The codebase ignores `STATIC_PROXY_URL` for dynamic providers even when the variable contains a valid URL.

## Summary

- **Enable globally** with `ENABLE_IP_PROXY = True` in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py)
- **Choose provider** via `IP_PROXY_PROVIDER_NAME`: kuaidaili, wandouhttp, or static
- **Normalize formats** through [`tools/crawler_util.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/crawler_util.py)'s `format_proxy_info()` function
- **Inject automatically** into Playwright contexts by [`tools/cdp_browser.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/cdp_browser.py) without manual launch code changes
- **Static proxies** require only `STATIC_PROXY_URL` populated; no external API calls occur

## Frequently Asked Questions

### How do I switch from KuaiDaili to a static proxy in MediaCrawler?

Change `IP_PROXY_PROVIDER_NAME` from `"kuaidaili"` to `"static"` in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py), then set `STATIC_PROXY_URL` to your full proxy URL including credentials. No other file modifications are needed—the same `format_proxy_info()` path handles both providers.

### Does MediaCrawler support SOCKS5 proxies with Playwright?

The `format_proxy_info()` function accepts any `ip:port` combination. For SOCKS5, ensure your Playwright installation has SOCKS support enabled and prefix the server with `socks5://` in the `server` field if your provider returns it. The static URL parser preserves protocol prefixes when present.

### Where is the proxy actually attached to the browser instance?

In [`tools/cdp_browser.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/cdp_browser.py), the launch helpers pass the `playwright_proxy` dict to `playwright.chromium.launch_persistent_context()` or equivalent launch methods via the `proxy=` keyword argument. This occurs after `format_proxy_info()` processes the provider-specific IP information.