MediaCrawler IP Proxy Configuration: KuaiDaili, WanDouHTTP, and Static Setup

MediaCrawler implements a provider-based proxy architecture that supports dynamic IP rotation through KuaiDaili and WanDouHTTP APIs, alongside a static proxy fallback, all managed through an async pool with automatic expiration handling.

MediaCrawler offers a flexible MediaCrawler IP Proxy Configuration system designed to prevent IP bans during large-scale scraping. The architecture abstracts proxy acquisition behind a unified provider interface, allowing seamless switching between commercial proxy services and self-hosted static proxies via environment variables and configuration files.

Architecture Overview

The proxy subsystem follows a three-layer provider pattern. At the base, an abstract ProxyProvider class defines the async get_proxy(num) contract. Concrete implementations in proxy/providers/ handle vendor-specific API integrations, while ProxyIpPool in proxy/proxy_ip_pool.py orchestrates caching, validation, and expiration management. All providers return standardized IpInfoModel objects defined in proxy/types.py, ensuring consistent data structures across the application.

Supported Proxy Providers

KuaiDaili HTTP Provider

Located in [proxy/providers/kuaidl_proxy.py](https://github.com/NanmiCoder/MediaCrawler/blob/main/proxy/providers/kuaidl_proxy.py), the KuaiDaiLiProxy class integrates with the KuaiDaili dynamic proxy service. The get_proxy method constructs HTTP requests to https://dps.kdlapi.com/api/getdps/, parses the comma-separated "ip:port,expire" response format via parse_kuaidaili_proxy, and converts relative expiration seconds into absolute Unix timestamps. Retrieved proxies are stored in IpCache using composite keys formatted as "{PROVIDER}_{IP}_{PORT}" to prevent duplicates and enable efficient expiration tracking.

WanDouHTTP Provider

The [proxy/providers/wandou_http_proxy.py](https://github.com/NanmiCoder/MediaCrawler/blob/main/proxy/providers/wandou_http_proxy.py) file implements WanDouHttpProxy, which communicates with the WanDou proxy API. This provider constructs request URLs containing the app_key and requested proxy count (num), processes the JSON response, and parses the expire_time string into a Unix timestamp for storage in the cache.

Static Proxy Provider

For users with dedicated proxy infrastructure, [proxy/proxy_ip_pool.py](https://github.com/NanmiCoder/MediaCrawler/blob/main/proxy/proxy_ip_pool.py) contains the StaticProxyProvider class (lines 55-86). It reads config.STATIC_PROXY_URL and returns a single, non-expiring IpInfoModel instance. Unlike dynamic providers, this implementation bypasses external API calls and network latency, making it ideal for development environments or corporate proxy gateways.

Core Components

Data Models and Provider Types

[proxy/types.py](https://github.com/NanmiCoder/MediaCrawler/blob/main/proxy/types.py) defines the IpInfoModel dataclass that standardizes proxy representation across all implementations. Key fields include ip, port, user, password, and expire_ts. The file also exports the IpProxyProvider enumeration, which maps string identifiers ("kuaidaili", "wandouhttp", "static") to their respective factory functions (new_kuai_daili_proxy(), new_wandou_http_proxy(), and StaticProxyProvider).

Proxy Pool Management

The [proxy/proxy_ip_pool.py](https://github.com/NanmiCoder/MediaCrawler/blob/main/proxy/proxy_ip_pool.py) module implements the ProxyIpPool class. The create_ip_pool(ip_pool_count, enable_validate_ip) factory function selects the appropriate provider based on config.IP_PROXY_PROVIDER_NAME, instantiates the provider, and pre-loads the requested number of proxies. The pool's get_or_refresh_proxy() method checks expiration via IpInfoModel.is_expired(buffer_seconds) and optionally validates connectivity by issuing test requests to https://echo.apifox.cn/ before returning a proxy to the crawler.

Configuration Setup

Proxy behavior is controlled through [config/base_config.py](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py). The key variables include:

• IP_PROXY_PROVIDER_NAME – Defaults to "kuaidaili"; accepts "wandouhttp" or "static"
• STATIC_PROXY_URL – Required when using static mode (e.g., http://user:pass@host:port)
• KDL_APP_KEY and KDL_APP_SECRET – Authentication credentials for KuaiDaili
• ENABLE_IP_PROXY – Boolean flag to enable or disable proxy usage globally

Environment variables can override these defaults at runtime.

Implementation Examples

Dynamic KuaiDaili Pool

import config
from proxy.proxy_ip_pool import create_ip_pool

# Select KuaiDaili provider

config.IP_PROXY_PROVIDER_NAME = "kuaidaili"

# Initialize pool with 10 proxies and IP validation enabled

pool = await create_ip_pool(ip_pool_count=10, enable_validate_ip=True)
proxy = await pool.get_or_refresh_proxy()
print(f"Assigned proxy: {proxy.ip}:{proxy.port}")

Static Proxy Configuration

import config
from proxy.proxy_ip_pool import create_ip_pool

# Configure static proxy

config.STATIC_PROXY_URL = "http://myuser:mypass@123.45.67.89:3128"
config.IP_PROXY_PROVIDER_NAME = "static"

# Create pool (validation disabled for static proxies)

pool = await create_ip_pool(ip_pool_count=1, enable_validate_ip=False)
proxy = await pool.get_or_refresh_proxy()

# proxy.expire_ts is None (never expires)

Direct Provider Instantiation

from proxy.providers.kuaidl_proxy import new_kuai_daili_proxy

# Initialize provider directly (automatically reads KDL_* environment variables)

provider = new_kuai_daili_proxy()
proxies = await provider.get_proxy(num=5)
for proxy in proxies:
    print(proxy.model_dump_json())

Summary

• MediaCrawler uses a provider pattern to abstract proxy acquisition across KuaiDaili, WanDouHTTP, and static configurations
• The ProxyIpPool class manages caching, expiration checking via IpInfoModel.is_expired(), and optional validation against https://echo.apifox.cn/
• Provider selection is controlled by IP_PROXY_PROVIDER_NAME in config/base_config.py
• Each provider implements get_proxy(num) to return standardized IpInfoModel instances
• Static proxies bypass API calls and expiration logic, while dynamic providers handle automatic rotation and timestamp conversion

Frequently Asked Questions

How do I switch from KuaiDaili to a static proxy in MediaCrawler?

Set config.IP_PROXY_PROVIDER_NAME = "static" and define config.STATIC_PROXY_URL with your proxy URL including credentials if required. The StaticProxyProvider will return your fixed proxy for every request without contacting external APIs.

Where does MediaCrawler store fetched proxy IPs before using them?

Proxies are cached in the IpCache class using composite keys formatted as "{PROVIDER}_{IP}_{PORT}". This storage mechanism prevents duplicate entries and enables quick expiration checks before the crawler assigns a proxy to an HTTP request.

Can I validate proxies before adding them to the pool?

Yes. Pass enable_validate_ip=True to create_ip_pool(). The ProxyIpPool will then perform test requests to https://echo.apifox.cn/ for each new proxy to verify connectivity before marking it as available for scraping operations.

What file contains the data structure for proxy information?

The IpInfoModel dataclass is defined in [proxy/types.py](https://github.com/NanmiCoder/MediaCrawler/blob/main/proxy/types.py). It standardizes the schema for proxy data including IP address, port, authentication credentials, and expiration timestamps across all provider implementations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}