# MediaCrawler IP Proxy Configuration: KuaiDaili, WanDouHTTP, and Static Setup

> Configure MediaCrawler IP proxies with KuaiDaili, WanDouHTTP, or static setup. Learn dynamic IP rotation and async pool management for efficient crawling.

- Repository: [程序员阿江-Relakkes/MediaCrawler](https://github.com/NanmiCoder/MediaCrawler)
- Tags: how-to-guide
- Published: 2026-07-31

---

**MediaCrawler implements a provider-based proxy architecture that supports dynamic IP rotation through KuaiDaili and WanDouHTTP APIs, alongside a static proxy fallback, all managed through an async pool with automatic expiration handling.**

MediaCrawler offers a flexible **MediaCrawler IP Proxy Configuration** system designed to prevent IP bans during large-scale scraping. The architecture abstracts proxy acquisition behind a unified provider interface, allowing seamless switching between commercial proxy services and self-hosted static proxies via environment variables and configuration files.

## Architecture Overview

The proxy subsystem follows a three-layer provider pattern. At the base, an abstract `ProxyProvider` class defines the async `get_proxy(num)` contract. Concrete implementations in `proxy/providers/` handle vendor-specific API integrations, while `ProxyIpPool` in [`proxy/proxy_ip_pool.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/proxy/proxy_ip_pool.py) orchestrates caching, validation, and expiration management. All providers return standardized `IpInfoModel` objects defined in [`proxy/types.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/proxy/types.py), ensuring consistent data structures across the application.

## Supported Proxy Providers

### KuaiDaili HTTP Provider

Located in [[`proxy/providers/kuaidl_proxy.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/proxy/providers/kuaidl_proxy.py)](https://github.com/NanmiCoder/MediaCrawler/blob/main/proxy/providers/kuaidl_proxy.py), the `KuaiDaiLiProxy` class integrates with the KuaiDaili dynamic proxy service. The `get_proxy` method constructs HTTP requests to `https://dps.kdlapi.com/api/getdps/`, parses the comma-separated `"ip:port,expire"` response format via `parse_kuaidaili_proxy`, and converts relative expiration seconds into absolute Unix timestamps. Retrieved proxies are stored in `IpCache` using composite keys formatted as `"{PROVIDER}_{IP}_{PORT}"` to prevent duplicates and enable efficient expiration tracking.

### WanDouHTTP Provider

The [[`proxy/providers/wandou_http_proxy.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/proxy/providers/wandou_http_proxy.py)](https://github.com/NanmiCoder/MediaCrawler/blob/main/proxy/providers/wandou_http_proxy.py) file implements `WanDouHttpProxy`, which communicates with the WanDou proxy API. This provider constructs request URLs containing the `app_key` and requested proxy count (`num`), processes the JSON response, and parses the `expire_time` string into a Unix timestamp for storage in the cache.

### Static Proxy Provider

For users with dedicated proxy infrastructure, [[`proxy/proxy_ip_pool.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/proxy/proxy_ip_pool.py)](https://github.com/NanmiCoder/MediaCrawler/blob/main/proxy/proxy_ip_pool.py) contains the `StaticProxyProvider` class (lines 55-86). It reads `config.STATIC_PROXY_URL` and returns a single, non-expiring `IpInfoModel` instance. Unlike dynamic providers, this implementation bypasses external API calls and network latency, making it ideal for development environments or corporate proxy gateways.

## Core Components

### Data Models and Provider Types

[[`proxy/types.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/proxy/types.py)](https://github.com/NanmiCoder/MediaCrawler/blob/main/proxy/types.py) defines the `IpInfoModel` dataclass that standardizes proxy representation across all implementations. Key fields include `ip`, `port`, `user`, `password`, and `expire_ts`. The file also exports the `IpProxyProvider` enumeration, which maps string identifiers (`"kuaidaili"`, `"wandouhttp"`, `"static"`) to their respective factory functions (`new_kuai_daili_proxy()`, `new_wandou_http_proxy()`, and `StaticProxyProvider`).

### Proxy Pool Management

The [[`proxy/proxy_ip_pool.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/proxy/proxy_ip_pool.py)](https://github.com/NanmiCoder/MediaCrawler/blob/main/proxy/proxy_ip_pool.py) module implements the `ProxyIpPool` class. The `create_ip_pool(ip_pool_count, enable_validate_ip)` factory function selects the appropriate provider based on `config.IP_PROXY_PROVIDER_NAME`, instantiates the provider, and pre-loads the requested number of proxies. The pool's `get_or_refresh_proxy()` method checks expiration via `IpInfoModel.is_expired(buffer_seconds)` and optionally validates connectivity by issuing test requests to `https://echo.apifox.cn/` before returning a proxy to the crawler.

## Configuration Setup

Proxy behavior is controlled through [[`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py)](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py). The key variables include:

- `IP_PROXY_PROVIDER_NAME` – Defaults to `"kuaidaili"`; accepts `"wandouhttp"` or `"static"`
- `STATIC_PROXY_URL` – Required when using static mode (e.g., `http://user:pass@host:port`)
- `KDL_APP_KEY` and `KDL_APP_SECRET` – Authentication credentials for KuaiDaili
- `ENABLE_IP_PROXY` – Boolean flag to enable or disable proxy usage globally

Environment variables can override these defaults at runtime.

## Implementation Examples

### Dynamic KuaiDaili Pool

```python
import config
from proxy.proxy_ip_pool import create_ip_pool

# Select KuaiDaili provider

config.IP_PROXY_PROVIDER_NAME = "kuaidaili"

# Initialize pool with 10 proxies and IP validation enabled

pool = await create_ip_pool(ip_pool_count=10, enable_validate_ip=True)
proxy = await pool.get_or_refresh_proxy()
print(f"Assigned proxy: {proxy.ip}:{proxy.port}")

```

### Static Proxy Configuration

```python
import config
from proxy.proxy_ip_pool import create_ip_pool

# Configure static proxy

config.STATIC_PROXY_URL = "http://myuser:mypass@123.45.67.89:3128"
config.IP_PROXY_PROVIDER_NAME = "static"

# Create pool (validation disabled for static proxies)

pool = await create_ip_pool(ip_pool_count=1, enable_validate_ip=False)
proxy = await pool.get_or_refresh_proxy()

# proxy.expire_ts is None (never expires)

```

### Direct Provider Instantiation

```python
from proxy.providers.kuaidl_proxy import new_kuai_daili_proxy

# Initialize provider directly (automatically reads KDL_* environment variables)

provider = new_kuai_daili_proxy()
proxies = await provider.get_proxy(num=5)
for proxy in proxies:
    print(proxy.model_dump_json())

```

## Summary

- MediaCrawler uses a provider pattern to abstract proxy acquisition across KuaiDaili, WanDouHTTP, and static configurations
- The `ProxyIpPool` class manages caching, expiration checking via `IpInfoModel.is_expired()`, and optional validation against `https://echo.apifox.cn/`
- Provider selection is controlled by `IP_PROXY_PROVIDER_NAME` in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py)
- Each provider implements `get_proxy(num)` to return standardized `IpInfoModel` instances
- Static proxies bypass API calls and expiration logic, while dynamic providers handle automatic rotation and timestamp conversion

## Frequently Asked Questions

### How do I switch from KuaiDaili to a static proxy in MediaCrawler?

Set `config.IP_PROXY_PROVIDER_NAME = "static"` and define `config.STATIC_PROXY_URL` with your proxy URL including credentials if required. The `StaticProxyProvider` will return your fixed proxy for every request without contacting external APIs.

### Where does MediaCrawler store fetched proxy IPs before using them?

Proxies are cached in the `IpCache` class using composite keys formatted as `"{PROVIDER}_{IP}_{PORT}"`. This storage mechanism prevents duplicate entries and enables quick expiration checks before the crawler assigns a proxy to an HTTP request.

### Can I validate proxies before adding them to the pool?

Yes. Pass `enable_validate_ip=True` to `create_ip_pool()`. The `ProxyIpPool` will then perform test requests to `https://echo.apifox.cn/` for each new proxy to verify connectivity before marking it as available for scraping operations.

### What file contains the data structure for proxy information?

The `IpInfoModel` dataclass is defined in [[`proxy/types.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/proxy/types.py)](https://github.com/NanmiCoder/MediaCrawler/blob/main/proxy/types.py). It standardizes the schema for proxy data including IP address, port, authentication credentials, and expiration timestamps across all provider implementations.