How to Set Up Proxy Servers for MediaCrawler: A Complete Configuration Guide

MediaCrawler supports three proxy providers—KuaiDaili, Wandou HTTP, and static proxies—configured through config/base_config.py and managed via the ProxyIpPool class in proxy/proxy_ip_pool.py.

Setting up proxy servers for MediaCrawler enables distributed crawling and IP rotation to avoid rate limits. The repository implements a flexible provider architecture that handles proxy lifecycle management, automatic refreshing, and compatibility with both Playwright browser automation and httpx async HTTP requests.

Understanding MediaCrawler's Proxy Architecture

MediaCrawler delegates proxy management to the ProxyIpPool class, which maintains a pool of IpInfoModel objects and validates their availability. The system supports dynamic providers that fetch fresh IPs via HTTP APIs and static configurations for fixed corporate proxies.

Supported Proxy Providers

The framework includes three distinct provider implementations:

• KuaiDaili (proxy/providers/kuaidl_proxy.py): Fetches dynamic residential proxies via kuaidaili.com's API; requires authentication credentials and suits high-volume crawling.
• Wandou HTTP (proxy/providers/wandou_http_proxy.py): Alternative dynamic proxy source with similar HTTP API integration.
• Static (proxy/proxy_ip_pool.py → StaticProxyProvider): Uses a single fixed proxy URL (e.g., http://user:pass@host:port) for simple setups or corporate gateways.

Configuring Proxy Settings in base_config.py

All proxy configuration resides in config/base_config.py. Set these variables to enable and customize proxy behavior:

# Enable proxy usage globally

ENABLE_IP_PROXY = True

# Maintain 2 proxies in the rotation pool

IP_PROXY_POOL_COUNT = 2

# Select provider: "kuaidaili", "wandouhttp", or "static"

IP_PROXY_PROVIDER_NAME = "kuaidaili"

# For static provider only: full proxy URL with optional auth

STATIC_PROXY_URL = "http://user:password@proxy.example.com:8080"

The ProxyIpPool reads these settings at runtime to instantiate the appropriate provider and populate the IP pool.

Setting Up Dynamic Proxies with KuaiDaili

To use KuaiDaili's dynamic proxy API, export the required credentials as environment variables before running the crawler. The KuaiDailiProxy class (proxy/providers/kuaidl_proxy.py) reads these via os.getenv():

export KDL_SECERT_ID="your_secret_id"
export KDL_SIGNATURE="your_signature"
export KDL_USER_NAME="your_username"
export KDL_USER_PWD="your_password"

After configuring credentials and setting IP_PROXY_PROVIDER_NAME = "kuaidaili" in base_config.py, initialize the pool:

import config
from proxy.proxy_ip_pool import create_ip_pool

async def init_proxy_pool():
    if config.ENABLE_IP_PROXY:
        pool = await create_ip_pool(
            ip_pool_count=config.IP_PROXY_POOL_COUNT,
            enable_validate_ip=True,  # Optional: verify proxies with test requests

        )
        return pool
    return None

Configuring Static Proxies for Simple Setups

For a single fixed proxy, change the provider name and supply the complete URL:

# config/base_config.py

ENABLE_IP_PROXY = True
IP_PROXY_PROVIDER_NAME = "static"
STATIC_PROXY_URL = "http://user:passwd@corporate-proxy.company.com:3128"

The StaticProxyProvider class parses this URL into an IpInfoModel and treats the proxy as non-expiring. This approach works well for development environments or corporate networks with gateway proxies.

Initializing and Using the Proxy Pool

The create_ip_pool factory function returns a ProxyIpPool instance that manages proxy lifecycle. Retrieve a valid proxy using get_or_refresh_proxy(), which automatically refreshes expired entries:

from proxy.proxy_ip_pool import create_ip_pool
from tools.crawler_util import format_proxy_info

# Initialize once at startup

proxy_pool = await create_ip_pool(
    ip_pool_count=config.IP_PROXY_POOL_COUNT,
    enable_validate_ip=False,
)

# Get proxy for each request or browser instance

proxy = await proxy_pool.get_or_refresh_proxy()
playwright_proxy, httpx_proxy = format_proxy_info(proxy)

The format_proxy_info() function in tools/crawler_util.py converts IpInfoModel into dictionaries compatible with both engines:

Playwright browser launch:

browser = await playwright.chromium.launch(
    headless=config.HEADLESS,
    proxy=playwright_proxy,  # Dict: {"server": "host:port", "username": "...", "password": "..."}

)

httpx async client:

from tools.httpx_util import make_async_client

async with make_async_client(proxy=httpx_proxy) as client:
    response = await client.get("https://api.target.com/data")

Both formats automatically include authentication credentials when present in the proxy URL.

Handling SSL Verification for Corporate Proxies

When routing traffic through corporate MITM proxies (e.g., Burp Suite or mitmproxy), disable SSL certificate verification to prevent connection errors:

# config/base_config.py

DISABLE_SSL_VERIFY = True

The make_async_client function in tools/httpx_util.py respects this flag, and Playwright launch configurations similarly bypass verification when this setting is enabled.

Summary

• Three providers available: KuaiDaili (dynamic API), Wandou HTTP (dynamic API), and Static (single fixed proxy).
• Core configuration happens in config/base_config.py via ENABLE_IP_PROXY, IP_PROXY_PROVIDER_NAME, and IP_PROXY_POOL_COUNT.
• KuaiDaili authentication requires four environment variables: KDL_SECERT_ID, KDL_SIGNATURE, KDL_USER_NAME, and KDL_USER_PWD.
• Pool initialization uses create_ip_pool() from proxy/proxy_ip_pool.py to preload and validate proxies.
• Runtime usage calls proxy_pool.get_or_refresh_proxy() followed by format_proxy_info() to generate Playwright and httpx compatible configurations.
• Static proxies use the STATIC_PROXY_URL format http://user:pass@host:port with provider name set to "static".
• SSL verification can be disabled via DISABLE_SSL_VERIFY for corporate proxy environments.

Frequently Asked Questions

How do I switch between different proxy providers in MediaCrawler?

Change the IP_PROXY_PROVIDER_NAME value in config/base_config.py to "kuaidaili", "wandouhttp", or "static". The create_ip_pool factory automatically instantiates the corresponding provider class based on this string. Ensure you provide the required credentials for dynamic providers or the STATIC_PROXY_URL for static configurations.

Why does my proxy pool return empty or invalid proxies?

If enable_validate_ip is set to True during create_ip_pool() initialization, MediaCrawler tests each proxy with an HTTP request and discards failures. Check that your KuaiDaili credentials are correctly exported as environment variables and that your account has available proxy quotas. For static proxies, verify that STATIC_PROXY_URL includes the correct scheme, host, port, and authentication details.

Can I use MediaCrawler with a proxy that requires IP whitelisting instead of username/password?

The current implementation in proxy/proxy_ip_pool.py and tools/crawler_util.py primarily supports credential-based authentication via URL parsing (user:pass@host). For IP-whitelisted proxies, omit the credentials from STATIC_PROXY_URL (e.g., http://proxy.company.com:8080). The format_proxy_info() function will generate configuration dicts without authentication fields, allowing the proxy to recognize your crawler's egress IP.

How does MediaCrawler handle proxy rotation during long-running crawls?

The ProxyIpPool class manages a fixed-size pool (IP_PROXY_POOL_COUNT) and automatically refreshes individual proxies when they expire or fail validation. Each call to get_or_refresh_proxy() returns the current valid proxy from the pool, and the system asynchronously fetches replacements for expired entries without blocking the crawling process.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}