# How to Implement Multi-Account Rotation with IP Proxy Pools in MediaCrawler

> Learn to implement multi-account rotation with IP proxy pools in MediaCrawler. Automate IP and user credential rotation using built-in proxy pooling and account manager.

- Repository: [程序员阿江-Relakkes/MediaCrawler](https://github.com/NanmiCoder/MediaCrawler)
- Tags: how-to-guide
- Published: 2026-08-14

---

**MediaCrawler provides built-in proxy pooling infrastructure and platform-specific exception handling that you can combine with a lightweight account manager to rotate both IP addresses and user credentials automatically.**

The `NanmiCoder/MediaCrawler` repository separates two critical concerns for large-scale social media scraping: credential management and network anonymity. This separation lets you build a resilient crawler that switches accounts when one gets blocked, while simultaneously routing traffic through rotating IP addresses to avoid detection.

## Understanding the Core Components

MediaCrawler's architecture gives you two extensible systems to leverage:

| Component | Purpose | Key Location |
|-----------|---------|--------------|
| **Proxy Pool** | Load, validate, and rotate IP proxies on-demand | [`proxy/proxy_ip_pool.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/proxy/proxy_ip_pool.py) |
| **Account Credentials** | Per-platform authentication (phone, password, cookies) | [`config/xhs_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/xhs_config.py), [`config/douyin_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/douyin_config.py), etc. |
| **Access Exceptions** | Detect blocks, bans, and rate-limits | [`media_platform/xhs/exception.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/xhs/exception.py) |

The `ProxyIpPool` class handles the networking layer. Your custom `AccountManager` handles the identity layer. When combined inside a retry loop, they create a self-healing crawler.

## Step 1: Initialize the Proxy Pool

The [`proxy/proxy_ip_pool.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/proxy/proxy_ip_pool.py) module exports `create_ip_pool`—a factory that returns a configured `ProxyIpPool` instance.

```python
from proxy.proxy_ip_pool import create_ip_pool

proxy_pool = await create_ip_pool(
    ip_pool_count=10,        # Keep 10 proxies in rotation

    enable_validate_ip=True  # Verify each proxy before use

)

```

Under the hood, `get_or_refresh_proxy()` checks `is_current_proxy_expired` and draws from `self.proxy_list` when needed. When validation is enabled, each candidate is tested against `self.valid_ip_url` before being returned.

## Step 2: Build a Round-Robin Account Manager

MediaCrawler does not include a multi-account manager, but you can add one in approximately 20 lines. Create [`utils/account_manager.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/utils/account_manager.py):

```python
from typing import List, Dict


class AccountManager:
    """Round-robin credential rotation for any platform."""
    
    def __init__(self, accounts: List[Dict]):
        if not accounts:
            raise ValueError("At least one account must be supplied")
        self._accounts = accounts
        self._cur = 0

    def current(self) -> Dict:
        """Return credentials for the active account."""
        return self._accounts[self._cur]

    def next(self) -> Dict:
        """Advance index and return new credentials."""
        self._cur = (self._cur + 1) % len(self._accounts)
        return self.current()

```

Load accounts from environment variables, JSON files, or a secrets manager—extending the pattern already used in [`config/xhs_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/xhs_config.py).

## Step 3: Wire Proxy and Account Rotation Together

The integration happens in your request wrapper. Here's a complete pattern using the XHS client:

```python
import asyncio
from proxy.proxy_ip_pool import create_ip_pool
from tools.httpx_util import make_async_client
from utils.account_manager import AccountManager
from media_platform.xhs.client import XHSClient, PlatformAccessError


async def resilient_crawl():
    # Initialize infrastructure

    proxy_pool = await create_ip_pool(10, enable_validate_ip=True)
    
    accounts = [
        {"phone": "13800138000", "password": "account_1_pass"},
        {"phone": "13900139000", "password": "account_2_pass"},
        {"phone": "13700137000", "password": "account_3_pass"},
    ]
    acc_mgr = AccountManager(accounts)
    
    # Create client with first account

    client = XHSClient(**acc_mgr.current())
    
    for target_url in crawl_queue:
        success = False
        attempts = 0
        max_attempts = len(accounts) * 2  # Allow full rotation twice

        
        while not success and attempts < max_attempts:
            proxy = await proxy_pool.get_or_refresh_proxy()
            proxy_url = f"http://{proxy.ip}:{proxy.port}"
            
            async with make_async_client(proxy=proxy_url) as http:
                try:
                    data = await client.fetch_post_detail(http, target_url)
                    process(data)
                    success = True
                    
                except PlatformAccessError as e:
                    # Block detected: rotate both identity and network

                    attempts += 1
                    print(f"Block detected on account {acc_mgr.current()['phone']}")
                    
                    # Switch account

                    client.update_credentials(**acc_mgr.next())
                    
                    # Force fresh proxy for clean slate

                    await proxy_pool.get_or_refresh_proxy()
                    
        if not success:
            print(f"Failed to fetch {target_url} after exhausting all accounts")

```

### Key Integration Points

- **`make_async_client`** ([`tools/httpx_util.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/httpx_util.py)): Creates an `httpx.AsyncClient` preconfigured with your proxy URL
- **`PlatformAccessError`** ([`media_platform/xhs/exception.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/xhs/exception.py)): Caught to trigger rotation; equivalent exceptions exist for Douyin, Tieba, and Weibo clients
- **`client.update_credentials()`**: Platform clients store credentials in `self.account`; replacing this object changes all subsequent request headers

## Step 4: Persist Rotation State (Optional)

For long-running crawlers, persist the account index to avoid restarting with account 0 after every crash. The repository includes [`test_expiring_local_cache.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/test_expiring_local_cache.py) demonstrating file-based caching:

```python
import json
import os

class PersistentAccountManager(AccountManager):
    def __init__(self, accounts: List[Dict], state_file: str = ".account_idx"):
        super().__init__(accounts)
        self._state_file = state_file
        self._load_state()
    
    def _load_state(self):
        if os.path.exists(self._state_file):
            with open(self._state_file) as f:
                saved = json.load(f)
                self._cur = saved.get("index", 0) % len(self._accounts)
    
    def next(self) -> Dict:
        creds = super().next()
        with open(self._state_file, "w") as f:
            json.dump({"index": self._cur}, f)
        return creds

```

## Proxy Provider Integration

`proxy/providers/` contains ready-made factories for commercial proxy services:

| Provider | Factory Function | File |
|----------|----------------|------|
| 快代理 (Kuai Daili) | `new_kuai_daili_proxy` | [`proxy/providers/kuai_daili.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/proxy/providers/kuai_daili.py) |
| 豌豆HTTP | `new_wandou_http_proxy` | [`proxy/providers/wandou_http.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/proxy/providers/wandou_http.py) |

These return proxy dictionaries that `ProxyIpPool` consumes. Add your own provider by matching this output format:

```python
{"ip": "1.2.3.4", "port": 8080, "protocol": "http"}

```

## Platform-Specific Considerations

Different platforms trigger blocks differently. Adjust your exception handling per client:

- **XiaoHongShu (XHS)**: Raises `PlatformAccessError` for rate-limits and IP blocks
- **Douyin**: Raises authentication errors when session cookies expire
- **Tieba/Weibo**: Similar patterns in their respective [`client.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/client.py) files

Inspect `media_platform/<platform>/client.py` to identify the exact exception class for each target.

## Summary

- **[`proxy/proxy_ip_pool.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/proxy/proxy_ip_pool.py)** provides `create_ip_pool` and `ProxyIpPool.get_or_refresh_proxy()` for IP rotation
- Build a lightweight `AccountManager` to cycle through credential sets using round-robin logic
- Wrap requests with `make_async_client(proxy=...)` and catch `PlatformAccessError` to trigger rotation
- Combine both rotations—switching accounts and refreshing proxies—whenever a security exception occurs
- Optionally persist account index using the caching pattern from [`test_expiring_local_cache.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/test_expiring_local_cache.py)

## Frequently Asked Questions

### How many accounts and proxies do I need for reliable crawling?

Start with **3-5 accounts** and **10 proxies** per account. MediaCrawler's `ProxyIpPool` validates IPs before use, so your effective pool size depends on provider quality. Monitor block rates and scale horizontally—more accounts reduce per-account request volume, while more proxies distribute IP-based risk.

### Can I use the same proxy pool across multiple platforms?

Yes. `ProxyIpPool` is platform-agnostic. Initialize one pool and share it across XHS, Douyin, and Weibo clients. Each `make_async_client` call requests a fresh proxy independently, so concurrent platform scraping naturally distributes traffic across your IP pool.

### What happens if all accounts get blocked?

The example code includes a `max_attempts` limit. When exhausted, the specific URL fails and logs for manual review. Implement exponential backoff or CAPTCHA-solving hooks in the failure path. For critical pipelines, integrate with notification systems (email, Slack, PagerDuty) when account exhaustion occurs.

### Does MediaCrawler support cookie-based account rotation instead of password login?

Yes. Platform clients accept cookie dictionaries in their account parameters. Store multiple cookie sets in your accounts list and rotate them identically. This avoids expensive re-login requests and reduces detection fingerprints compared to repeated password authentication flows.