# How to Set Up Static Proxy Providers in MediaCrawler: Complete Configuration Guide

> Learn how to set up static proxy providers in MediaCrawler with our complete guide. Configure IP proxy settings easily for enhanced crawling.

- Repository: [程序员阿江-Relakkes/MediaCrawler](https://github.com/NanmiCoder/MediaCrawler)
- Tags: how-to-guide
- Published: 2026-06-27

---

**Enable static proxy support in MediaCrawler by setting `ENABLE_IP_PROXY` to `True`, `IP_PROXY_PROVIDER_NAME` to `"static"`, and providing your proxy endpoint in `STATIC_PROXY_URL`.**

MediaCrawler supports multiple proxy provider types, including *kuaidaili*, *wandouhttp*, and **static**. When you configure a static proxy provider, the crawler routes every HTTP request through a single, persistent proxy endpoint you specify. This approach is ideal for environments requiring a dedicated outbound IP address, such as corporate firewalls or rate-limit bypass scenarios.

## Configuration Methods

You can configure static proxy settings through the global configuration file or command-line arguments. Both methods interact with the same underlying variables defined in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py).

### Method 1: Edit base_config.py

The most direct approach is modifying the global configuration values in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py). Set the three required variables to enable and configure your static proxy:

```python

# config/base_config.py

ENABLE_IP_PROXY = True               # Turn on proxy support

IP_PROXY_PROVIDER_NAME = "static"   # Select static provider

STATIC_PROXY_URL = "http://user:passwd@proxy.example.com:8080"

```

These configuration variables are referenced throughout the codebase, including in [`cmd_arg/arg.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/cmd_arg/arg.py) where command-line overrides are processed.

### Method 2: Use Command-Line Arguments

MediaCrawler exposes the same configuration options as CLI flags, allowing you to override settings without editing files. This is useful for containerized deployments or temporary testing:

```bash
python -m MediaCrawler.main \
    --enable_ip_proxy true \
    --ip_proxy_provider_name static \
    --static_proxy_url "http://user:passwd@proxy.example.com:8080"

```

The argument definitions in [`cmd_arg/arg.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/cmd_arg/arg.py) map these flags directly to the configuration variables, ensuring consistency between file-based and command-line configuration.

## How Static Proxy Providers Work

When `IP_PROXY_PROVIDER_NAME` is set to `"static"` (case-insensitive), MediaCrawler initializes the `StaticProxyProvider` class located in [`proxy/proxy_ip_pool.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/proxy/proxy_ip_pool.py). This provider creates a single persistent proxy entry rather than rotating through a pool.

### URL Parsing and Authentication

The `StaticProxyProvider.get_proxy()` method parses `STATIC_PROXY_URL` using Python's `urlparse` module to extract connection details:

```python

# proxy/proxy_ip_pool.py – StaticProxyProvider implementation

proxy_url = getattr(config, "STATIC_PROXY_URL", "")
parsed = urlparse(proxy_url)
scheme = parsed.scheme or "http"
ip = parsed.hostname or ""
port = parsed.port or (443 if scheme == "https" else 80)

```

The implementation supports two URL formats:
- **Unauthenticated**: `http://your_domain:port`
- **Authenticated**: `http://user:password@your_domain:port`

Both `http` and `https` schemes are valid. Credentials are automatically URL-decoded using `unquote()` to handle special characters.

### IpInfoModel Creation

The provider returns an `IpInfoModel` instance with a far-future expiration timestamp, ensuring the proxy never expires from the pool's perspective:

```python
return [
    IpInfoModel(
        ip=ip,
        port=port,
        user=unquote(parsed.username or ""),
        password=unquote(parsed.password or ""),
        protocol=f"{scheme}://",
        expired_time_ts=int(time.time()) + 99999999,  # never expires

    )
]

```

This model is consumed by `ProxyIpPool.get_or_refresh_proxy()`, which supplies the proxy to HTTP clients throughout the application.

## Integration with Crawler Clients

MediaCrawler's HTTP clients inherit from `ProxyRefreshMixin`, which automatically retrieves the current proxy before each request. For example, in [`media_platform/zhihu/client.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/zhihu/client.py) and other platform clients, the mixin calls `self.proxy_ip_pool.get_or_refresh_proxy()` to obtain the static proxy:

```python

# Inside API client implementations

proxy = await self.proxy_ip_pool.get_or_refresh_proxy()
proxy_url = f"{proxy.protocol}{proxy.ip}:{proxy.port}"
async with make_async_client(proxy=proxy_url) as client:
    resp = await client.get("https://api.targetplatform.com/endpoint")

```

This architecture ensures that once you configure the static provider in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py) or via CLI, all requests automatically use your specified proxy without additional code changes.

## Summary

- **Static proxy mode** uses a single, persistent proxy endpoint for all crawler requests
- **Configuration** requires three variables: `ENABLE_IP_PROXY`, `IP_PROXY_PROVIDER_NAME="static"`, and `STATIC_PROXY_URL`
- **Implementation** resides in [`proxy/proxy_ip_pool.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/proxy/proxy_ip_pool.py) via the `StaticProxyProvider` class
- **Supported formats** include authenticated and unauthenticated HTTP/HTTPS URLs
- **Integration** is automatic through `ProxyRefreshMixin` in platform client classes

## Frequently Asked Questions

### What URL formats are supported for static proxies?

MediaCrawler supports `http://domain:port` for unauthenticated proxies and `http://user:password@domain:port` for authenticated connections. Both `http` and `https` schemes are valid. The `StaticProxyProvider` class in [`proxy/proxy_ip_pool.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/proxy/proxy_ip_pool.py) automatically extracts host, port, credentials, and protocol from the URL.

### Does the static proxy ever expire?

No. The `StaticProxyProvider` creates an `IpInfoModel` with an `expired_time_ts` value set to the current time plus 99,999,999 seconds (approximately 3 years). This ensures the proxy remains valid indefinitely within the `ProxyIpPool` lifecycle, eliminating the need for refresh logic used by rotating providers.

### Can I use HTTPS static proxies?

Yes. The URL parser in `StaticProxyProvider` recognizes both `http` and `https` schemes. If no port is specified, it defaults to 443 for HTTPS and 80 for HTTP. The protocol field in `IpInfoModel` preserves your specified scheme for use in actual HTTP requests.

### How do I switch from static to rotating proxy providers?

Change `IP_PROXY_PROVIDER_NAME` to `"kuaidaili"` or `"wandouhttp"` and remove or comment out `STATIC_PROXY_URL`. These providers implement different `get_proxy()` methods that fetch fresh IPs from external APIs rather than using the persistent static configuration.