How to Set Up Proxy Rotation with KuaiDaili in MediaCrawler

Configure MediaCrawler to rotate KuaiDaili proxies by setting ENABLE_IP_PROXY = True in config/base_config.py, providing KDL_SECERT_ID and other credentials in a .env file, and letting the ProxyIpPool class automatically manage proxy lifecycle and expiration.

MediaCrawler provides a production-ready proxy rotation system that integrates with KuaiDaili (快代理) to prevent IP bans during large-scale crawling. The architecture separates concerns into provider-specific implementations, a caching pool manager, and format converters for different HTTP clients.

How KuaiDaili Proxy Rotation Works

The system relies on three core components working together:

1. Provider Implementation – The KuaiDaiLiProxy class in proxy/providers/kuaidl_proxy.py handles authentication with the KuaiDaili API and parses the returned proxy strings.
2. Proxy Pool – ProxyIpPool in proxy/proxy_ip_pool.py maintains a cache of valid proxies, validates them if enabled, and rotates them based on expiration timestamps.
3. Format Conversion – The format_proxy_info function in tools/crawler_util.py converts the unified IpInfoModel into formats compatible with httpx and Playwright.

When ENABLE_IP_PROXY is True, main.py initializes the pool via create_ip_pool(). Each request then calls await ip_pool.get_or_refresh_proxy() to receive a fresh proxy, which is automatically formatted and injected into the HTTP client.

Step-by-Step Configuration

Enable Proxy Support

Open config/base_config.py and set the proxy flag:

# config/base_config.py

ENABLE_IP_PROXY = True
IP_PROXY_PROVIDER_NAME = "kuaidaili"  # Default provider

Configure KuaiDaili Credentials

Create a .env file in the repository root with your KuaiDaili account details. The new_kuai_daili_proxy() constructor in proxy/providers/kuaidl_proxy.py reads these environment variables:

# .env

KDL_SECERT_ID=your_kuaidaili_secret_id
KDL_SIGNATURE=your_kuaidaili_signature
KDL_USER_NAME=your_kuaidaili_username
KDL_USER_PWD=your_kuaidaili_password

Adjust Pool Settings (Optional)

Control the number of proxies fetched per API call and enable validation in config/base_config.py:

# Number of proxies to maintain in the pool

IP_PROXY_POOL_COUNT = 5

# Validate proxies before use (pings a test URL)

ENABLE_VALIDATE_IP = True

The Proxy Rotation Mechanism

The rotation logic implemented in proxy/proxy_ip_pool.py follows this workflow:

1. Initialization – ProxyIpPool.__init__ receives the provider instance and calls load_proxies().
2. Fetching – KuaiDaiLiProxy.get_proxy(num) contacts the KuaiDaili API endpoint /api/getdps/ and parses each "ip:port,expire" line using parse_kuaidaili_proxy.
3. Caching – Results are converted to IpInfoModel objects with absolute expiration timestamps (expire_ts + now - DELTA_EXPIRED_SECOND) and stored in IpCache.
4. Rotation – get_or_refresh_proxy() checks is_current_proxy_expired. If the proxy expires within 30 seconds or is missing, it draws a new one from the cache.
5. Formatting – Before each request, format_proxy_info converts IpInfoModel into a Playwright dictionary {"server": "host:port", "username": "...", "password": "..."} and an httpx URL string http://user:pwd@host:port.

Because expiration timestamps are computed on receipt, the pool automatically avoids using proxies that are about to expire, ensuring seamless rotation without failed requests.

Code Examples

Environment Variables Setup

Create a .env file in the project root:

KDL_SECERT_ID=your_kuaidaili_secret_id
KDL_SIGNATURE=your_kuaidaili_signature
KDL_USER_NAME=your_kuaidaili_username
KDL_USER_PWD=your_kuaidaili_password

Manual Proxy Pool Usage

Use this pattern to manually fetch and rotate proxies in custom scripts:

import asyncio
from proxy.proxy_ip_pool import create_ip_pool, ProxyIpPool
from tools.crawler_util import format_proxy_info
import httpx

async def fetch_with_rotation():
    # Initialize pool with 5 proxies and validation enabled

    ip_pool: ProxyIpPool = await create_ip_pool(
        ip_pool_count=5, 
        enable_validate_ip=True
    )
    
    # Get a fresh proxy (auto-rotates if expired)

    proxy = await ip_pool.get_or_refresh_proxy()
    print(f"Using proxy: {proxy.ip}:{proxy.port}")
    
    # Convert to httpx format

    _, httpx_proxy = format_proxy_info(proxy)
    
    async with httpx.AsyncClient(proxy=httpx_proxy) as client:
        resp = await client.get("https://httpbin.org/ip")
        print(resp.json())

asyncio.run(fetch_with_rotation())

Integration in Main Crawler

The standard MediaCrawler entry point in main.py handles initialization automatically:

# main.py (simplified)

from proxy.proxy_ip_pool import create_ip_pool
from config.base_config import ENABLE_IP_PROXY, IP_PROXY_POOL_COUNT, ENABLE_VALIDATE_IP

async def main():
    if ENABLE_IP_PROXY:
        ip_pool = await create_ip_pool(
            ip_pool_count=IP_PROXY_POOL_COUNT,
            enable_validate_ip=ENABLE_VALIDATE_IP,
        )
        # Pass ip_pool to platform clients; they call 

        # await ip_pool.get_or_refresh_proxy() before each request

Summary

• Enable proxies by setting ENABLE_IP_PROXY = True in config/base_config.py.
• Authenticate with KuaiDaili using the four KDL_* environment variables read by proxy/providers/kuaidl_proxy.py.
• Control pool size via IP_PROXY_POOL_COUNT and optionally validate proxies with ENABLE_VALIDATE_IP.
• Automatic rotation is handled by ProxyIpPool.get_or_refresh_proxy(), which checks expiration timestamps and pulls from the IpCache when needed.
• Format conversion is performed by format_proxy_info in tools/crawler_util.py to support both httpx and Playwright clients.

Frequently Asked Questions

How does MediaCrawler handle proxy expiration?

MediaCrawler calculates absolute expiration timestamps when proxies are received from KuaiDaili. The ProxyIpPool class checks these timestamps via is_current_proxy_expired before each use, automatically fetching a new proxy from the pool if the current one expires within 30 seconds. This prevents request failures due to expired IPs.

What is the difference between IP_PROXY_POOL_COUNT and the actual number of requests?

IP_PROXY_POOL_COUNT defines how many proxy IPs are fetched and cached per API call to KuaiDaili. It does not limit the number of requests you can make. Once the pool is initialized, get_or_refresh_proxy() cycles through cached proxies, and only contacts the KuaiDaili API again when the local pool runs low or expires.

Can I use KuaiDaili proxy rotation without validation?

Yes. Set ENABLE_VALIDATE_IP = False in config/base_config.py to skip the validation step. In this mode, ProxyIpPool assumes all received proxies are valid and immediately adds them to the IpCache. Validation is recommended for production crawls to ensure connectivity, but disabling it reduces initialization time.

Where does MediaCrawler store the proxy credentials?

The KuaiDaili credentials are read from environment variables (KDL_SECERT_ID, KDL_SIGNATURE, KDL_USER_NAME, KDL_USER_PWD) by the new_kuai_daili_proxy() function in proxy/providers/kuaidl_proxy.py. MediaCrawler does not hardcode these values; it expects them in a .env file or exported in the shell environment before runtime.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}