# How to Set Up Proxy Servers for MediaCrawler: A Complete Configuration Guide

> Learn how to set up proxy servers for MediaCrawler with this complete configuration guide. Easily integrate KuaiDaili, Wandou HTTP, or static proxies for enhanced performance.

- Repository: [程序员阿江-Relakkes/MediaCrawler](https://github.com/NanmiCoder/MediaCrawler)
- Tags: how-to-guide
- Published: 2026-06-28

---

**MediaCrawler supports three proxy providers—KuaiDaili, Wandou HTTP, and static proxies—configured through [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py) and managed via the `ProxyIpPool` class in [`proxy/proxy_ip_pool.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/proxy/proxy_ip_pool.py).**

Setting up proxy servers for MediaCrawler enables distributed crawling and IP rotation to avoid rate limits. The repository implements a flexible provider architecture that handles proxy lifecycle management, automatic refreshing, and compatibility with both **Playwright** browser automation and **httpx** async HTTP requests.

## Understanding MediaCrawler's Proxy Architecture

MediaCrawler delegates proxy management to the **`ProxyIpPool`** class, which maintains a pool of `IpInfoModel` objects and validates their availability. The system supports dynamic providers that fetch fresh IPs via HTTP APIs and static configurations for fixed corporate proxies.

### Supported Proxy Providers

The framework includes three distinct provider implementations:

- **KuaiDaili** ([`proxy/providers/kuaidl_proxy.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/proxy/providers/kuaidl_proxy.py)): Fetches dynamic residential proxies via kuaidaili.com's API; requires authentication credentials and suits high-volume crawling.
- **Wandou HTTP** ([`proxy/providers/wandou_http_proxy.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/proxy/providers/wandou_http_proxy.py)): Alternative dynamic proxy source with similar HTTP API integration.
- **Static** ([`proxy/proxy_ip_pool.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/proxy/proxy_ip_pool.py) → `StaticProxyProvider`): Uses a single fixed proxy URL (e.g., `http://user:pass@host:port`) for simple setups or corporate gateways.

## Configuring Proxy Settings in base_config.py

All proxy configuration resides in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py). Set these variables to enable and customize proxy behavior:

```python

# Enable proxy usage globally

ENABLE_IP_PROXY = True

# Maintain 2 proxies in the rotation pool

IP_PROXY_POOL_COUNT = 2

# Select provider: "kuaidaili", "wandouhttp", or "static"

IP_PROXY_PROVIDER_NAME = "kuaidaili"

# For static provider only: full proxy URL with optional auth

STATIC_PROXY_URL = "http://user:password@proxy.example.com:8080"

```

The `ProxyIpPool` reads these settings at runtime to instantiate the appropriate provider and populate the IP pool.

## Setting Up Dynamic Proxies with KuaiDaili

To use KuaiDaili's dynamic proxy API, export the required credentials as environment variables before running the crawler. The `KuaiDailiProxy` class ([`proxy/providers/kuaidl_proxy.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/proxy/providers/kuaidl_proxy.py)) reads these via `os.getenv()`:

```bash
export KDL_SECERT_ID="your_secret_id"
export KDL_SIGNATURE="your_signature"
export KDL_USER_NAME="your_username"
export KDL_USER_PWD="your_password"

```

After configuring credentials and setting `IP_PROXY_PROVIDER_NAME = "kuaidaili"` in [`base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/base_config.py), initialize the pool:

```python
import config
from proxy.proxy_ip_pool import create_ip_pool

async def init_proxy_pool():
    if config.ENABLE_IP_PROXY:
        pool = await create_ip_pool(
            ip_pool_count=config.IP_PROXY_POOL_COUNT,
            enable_validate_ip=True,  # Optional: verify proxies with test requests

        )
        return pool
    return None

```

## Configuring Static Proxies for Simple Setups

For a single fixed proxy, change the provider name and supply the complete URL:

```python

# config/base_config.py

ENABLE_IP_PROXY = True
IP_PROXY_PROVIDER_NAME = "static"
STATIC_PROXY_URL = "http://user:passwd@corporate-proxy.company.com:3128"

```

The `StaticProxyProvider` class parses this URL into an `IpInfoModel` and treats the proxy as non-expiring. This approach works well for development environments or corporate networks with gateway proxies.

## Initializing and Using the Proxy Pool

The `create_ip_pool` factory function returns a `ProxyIpPool` instance that manages proxy lifecycle. Retrieve a valid proxy using `get_or_refresh_proxy()`, which automatically refreshes expired entries:

```python
from proxy.proxy_ip_pool import create_ip_pool
from tools.crawler_util import format_proxy_info

# Initialize once at startup

proxy_pool = await create_ip_pool(
    ip_pool_count=config.IP_PROXY_POOL_COUNT,
    enable_validate_ip=False,
)

# Get proxy for each request or browser instance

proxy = await proxy_pool.get_or_refresh_proxy()
playwright_proxy, httpx_proxy = format_proxy_info(proxy)

```

The `format_proxy_info()` function in [`tools/crawler_util.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/crawler_util.py) converts `IpInfoModel` into dictionaries compatible with both engines:

**Playwright browser launch:**

```python
browser = await playwright.chromium.launch(
    headless=config.HEADLESS,
    proxy=playwright_proxy,  # Dict: {"server": "host:port", "username": "...", "password": "..."}

)

```

**httpx async client:**

```python
from tools.httpx_util import make_async_client

async with make_async_client(proxy=httpx_proxy) as client:
    response = await client.get("https://api.target.com/data")

```

Both formats automatically include authentication credentials when present in the proxy URL.

## Handling SSL Verification for Corporate Proxies

When routing traffic through corporate MITM proxies (e.g., Burp Suite or mitmproxy), disable SSL certificate verification to prevent connection errors:

```python

# config/base_config.py

DISABLE_SSL_VERIFY = True

```

The `make_async_client` function in [`tools/httpx_util.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/httpx_util.py) respects this flag, and Playwright launch configurations similarly bypass verification when this setting is enabled.

## Summary

- **Three providers available**: KuaiDaili (dynamic API), Wandou HTTP (dynamic API), and Static (single fixed proxy).
- **Core configuration** happens in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py) via `ENABLE_IP_PROXY`, `IP_PROXY_PROVIDER_NAME`, and `IP_PROXY_POOL_COUNT`.
- **KuaiDaili authentication** requires four environment variables: `KDL_SECERT_ID`, `KDL_SIGNATURE`, `KDL_USER_NAME`, and `KDL_USER_PWD`.
- **Pool initialization** uses `create_ip_pool()` from [`proxy/proxy_ip_pool.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/proxy/proxy_ip_pool.py) to preload and validate proxies.
- **Runtime usage** calls `proxy_pool.get_or_refresh_proxy()` followed by `format_proxy_info()` to generate Playwright and httpx compatible configurations.
- **Static proxies** use the `STATIC_PROXY_URL` format `http://user:pass@host:port` with provider name set to `"static"`.
- **SSL verification** can be disabled via `DISABLE_SSL_VERIFY` for corporate proxy environments.

## Frequently Asked Questions

### How do I switch between different proxy providers in MediaCrawler?

Change the `IP_PROXY_PROVIDER_NAME` value in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py) to `"kuaidaili"`, `"wandouhttp"`, or `"static"`. The `create_ip_pool` factory automatically instantiates the corresponding provider class based on this string. Ensure you provide the required credentials for dynamic providers or the `STATIC_PROXY_URL` for static configurations.

### Why does my proxy pool return empty or invalid proxies?

If `enable_validate_ip` is set to `True` during `create_ip_pool()` initialization, MediaCrawler tests each proxy with an HTTP request and discards failures. Check that your KuaiDaili credentials are correctly exported as environment variables and that your account has available proxy quotas. For static proxies, verify that `STATIC_PROXY_URL` includes the correct scheme, host, port, and authentication details.

### Can I use MediaCrawler with a proxy that requires IP whitelisting instead of username/password?

The current implementation in [`proxy/proxy_ip_pool.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/proxy/proxy_ip_pool.py) and [`tools/crawler_util.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/crawler_util.py) primarily supports credential-based authentication via URL parsing (`user:pass@host`). For IP-whitelisted proxies, omit the credentials from `STATIC_PROXY_URL` (e.g., `http://proxy.company.com:8080`). The `format_proxy_info()` function will generate configuration dicts without authentication fields, allowing the proxy to recognize your crawler's egress IP.

### How does MediaCrawler handle proxy rotation during long-running crawls?

The `ProxyIpPool` class manages a fixed-size pool (`IP_PROXY_POOL_COUNT`) and automatically refreshes individual proxies when they expire or fail validation. Each call to `get_or_refresh_proxy()` returns the current valid proxy from the pool, and the system asynchronously fetches replacements for expired entries without blocking the crawling process.