# How to Set Up Proxy Rotation with KuaiDaili in MediaCrawler

> Easily set up proxy rotation with KuaiDaili in MediaCrawler. Enable IP proxy, add credentials to .env, and let ProxyIpPool manage proxies automatically for seamless scraping.

- Repository: [程序员阿江-Relakkes/MediaCrawler](https://github.com/NanmiCoder/MediaCrawler)
- Tags: how-to-guide
- Published: 2026-06-27

---

**Configure MediaCrawler to rotate KuaiDaili proxies by setting `ENABLE_IP_PROXY = True` in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py), providing `KDL_SECERT_ID` and other credentials in a `.env` file, and letting the `ProxyIpPool` class automatically manage proxy lifecycle and expiration.**

MediaCrawler provides a production-ready proxy rotation system that integrates with KuaiDaili (快代理) to prevent IP bans during large-scale crawling. The architecture separates concerns into provider-specific implementations, a caching pool manager, and format converters for different HTTP clients.

## How KuaiDaili Proxy Rotation Works

The system relies on three core components working together:

1. **Provider Implementation** – The `KuaiDaiLiProxy` class in [`proxy/providers/kuaidl_proxy.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/proxy/providers/kuaidl_proxy.py) handles authentication with the KuaiDaili API and parses the returned proxy strings.
2. **Proxy Pool** – `ProxyIpPool` in [`proxy/proxy_ip_pool.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/proxy/proxy_ip_pool.py) maintains a cache of valid proxies, validates them if enabled, and rotates them based on expiration timestamps.
3. **Format Conversion** – The `format_proxy_info` function in [`tools/crawler_util.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/crawler_util.py) converts the unified `IpInfoModel` into formats compatible with **httpx** and **Playwright**.

When `ENABLE_IP_PROXY` is `True`, [`main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/main.py) initializes the pool via `create_ip_pool()`. Each request then calls `await ip_pool.get_or_refresh_proxy()` to receive a fresh proxy, which is automatically formatted and injected into the HTTP client.

## Step-by-Step Configuration

### Enable Proxy Support

Open [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py) and set the proxy flag:

```python

# config/base_config.py

ENABLE_IP_PROXY = True
IP_PROXY_PROVIDER_NAME = "kuaidaili"  # Default provider

```

### Configure KuaiDaili Credentials

Create a `.env` file in the repository root with your KuaiDaili account details. The `new_kuai_daili_proxy()` constructor in [`proxy/providers/kuaidl_proxy.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/proxy/providers/kuaidl_proxy.py) reads these environment variables:

```bash

# .env

KDL_SECERT_ID=your_kuaidaili_secret_id
KDL_SIGNATURE=your_kuaidaili_signature
KDL_USER_NAME=your_kuaidaili_username
KDL_USER_PWD=your_kuaidaili_password

```

### Adjust Pool Settings (Optional)

Control the number of proxies fetched per API call and enable validation in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py):

```python

# Number of proxies to maintain in the pool

IP_PROXY_POOL_COUNT = 5

# Validate proxies before use (pings a test URL)

ENABLE_VALIDATE_IP = True

```

## The Proxy Rotation Mechanism

The rotation logic implemented in [`proxy/proxy_ip_pool.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/proxy/proxy_ip_pool.py) follows this workflow:

1. **Initialization** – `ProxyIpPool.__init__` receives the provider instance and calls `load_proxies()`.
2. **Fetching** – `KuaiDaiLiProxy.get_proxy(num)` contacts the KuaiDaili API endpoint `/api/getdps/` and parses each `"ip:port,expire"` line using `parse_kuaidaili_proxy`.
3. **Caching** – Results are converted to `IpInfoModel` objects with absolute expiration timestamps (`expire_ts + now - DELTA_EXPIRED_SECOND`) and stored in `IpCache`.
4. **Rotation** – `get_or_refresh_proxy()` checks `is_current_proxy_expired`. If the proxy expires within 30 seconds or is missing, it draws a new one from the cache.
5. **Formatting** – Before each request, `format_proxy_info` converts `IpInfoModel` into a Playwright dictionary `{"server": "host:port", "username": "...", "password": "..."}` and an httpx URL string `http://user:pwd@host:port`.

Because expiration timestamps are computed on receipt, the pool automatically avoids using proxies that are about to expire, ensuring seamless rotation without failed requests.

## Code Examples

### Environment Variables Setup

Create a `.env` file in the project root:

```text
KDL_SECERT_ID=your_kuaidaili_secret_id
KDL_SIGNATURE=your_kuaidaili_signature
KDL_USER_NAME=your_kuaidaili_username
KDL_USER_PWD=your_kuaidaili_password

```

### Manual Proxy Pool Usage

Use this pattern to manually fetch and rotate proxies in custom scripts:

```python
import asyncio
from proxy.proxy_ip_pool import create_ip_pool, ProxyIpPool
from tools.crawler_util import format_proxy_info
import httpx

async def fetch_with_rotation():
    # Initialize pool with 5 proxies and validation enabled

    ip_pool: ProxyIpPool = await create_ip_pool(
        ip_pool_count=5, 
        enable_validate_ip=True
    )
    
    # Get a fresh proxy (auto-rotates if expired)

    proxy = await ip_pool.get_or_refresh_proxy()
    print(f"Using proxy: {proxy.ip}:{proxy.port}")
    
    # Convert to httpx format

    _, httpx_proxy = format_proxy_info(proxy)
    
    async with httpx.AsyncClient(proxy=httpx_proxy) as client:
        resp = await client.get("https://httpbin.org/ip")
        print(resp.json())

asyncio.run(fetch_with_rotation())

```

### Integration in Main Crawler

The standard MediaCrawler entry point in [`main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/main.py) handles initialization automatically:

```python

# main.py (simplified)

from proxy.proxy_ip_pool import create_ip_pool
from config.base_config import ENABLE_IP_PROXY, IP_PROXY_POOL_COUNT, ENABLE_VALIDATE_IP

async def main():
    if ENABLE_IP_PROXY:
        ip_pool = await create_ip_pool(
            ip_pool_count=IP_PROXY_POOL_COUNT,
            enable_validate_ip=ENABLE_VALIDATE_IP,
        )
        # Pass ip_pool to platform clients; they call 

        # await ip_pool.get_or_refresh_proxy() before each request

```

## Summary

- **Enable proxies** by setting `ENABLE_IP_PROXY = True` in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py).
- **Authenticate** with KuaiDaili using the four `KDL_*` environment variables read by [`proxy/providers/kuaidl_proxy.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/proxy/providers/kuaidl_proxy.py).
- **Control pool size** via `IP_PROXY_POOL_COUNT` and optionally validate proxies with `ENABLE_VALIDATE_IP`.
- **Automatic rotation** is handled by `ProxyIpPool.get_or_refresh_proxy()`, which checks expiration timestamps and pulls from the `IpCache` when needed.
- **Format conversion** is performed by `format_proxy_info` in [`tools/crawler_util.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/crawler_util.py) to support both httpx and Playwright clients.

## Frequently Asked Questions

### How does MediaCrawler handle proxy expiration?

MediaCrawler calculates absolute expiration timestamps when proxies are received from KuaiDaili. The `ProxyIpPool` class checks these timestamps via `is_current_proxy_expired` before each use, automatically fetching a new proxy from the pool if the current one expires within 30 seconds. This prevents request failures due to expired IPs.

### What is the difference between `IP_PROXY_POOL_COUNT` and the actual number of requests?

`IP_PROXY_POOL_COUNT` defines how many proxy IPs are fetched and cached per API call to KuaiDaili. It does not limit the number of requests you can make. Once the pool is initialized, `get_or_refresh_proxy()` cycles through cached proxies, and only contacts the KuaiDaili API again when the local pool runs low or expires.

### Can I use KuaiDaili proxy rotation without validation?

Yes. Set `ENABLE_VALIDATE_IP = False` in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py) to skip the validation step. In this mode, `ProxyIpPool` assumes all received proxies are valid and immediately adds them to the `IpCache`. Validation is recommended for production crawls to ensure connectivity, but disabling it reduces initialization time.

### Where does MediaCrawler store the proxy credentials?

The KuaiDaili credentials are read from environment variables (`KDL_SECERT_ID`, `KDL_SIGNATURE`, `KDL_USER_NAME`, `KDL_USER_PWD`) by the `new_kuai_daili_proxy()` function in [`proxy/providers/kuaidl_proxy.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/proxy/providers/kuaidl_proxy.py). MediaCrawler does not hardcode these values; it expects them in a `.env` file or exported in the shell environment before runtime.