How to Set Up Static Proxy Providers in MediaCrawler: Complete Configuration Guide

Enable static proxy support in MediaCrawler by setting ENABLE_IP_PROXY to True, IP_PROXY_PROVIDER_NAME to "static", and providing your proxy endpoint in STATIC_PROXY_URL.

MediaCrawler supports multiple proxy provider types, including kuaidaili, wandouhttp, and static. When you configure a static proxy provider, the crawler routes every HTTP request through a single, persistent proxy endpoint you specify. This approach is ideal for environments requiring a dedicated outbound IP address, such as corporate firewalls or rate-limit bypass scenarios.

Configuration Methods

You can configure static proxy settings through the global configuration file or command-line arguments. Both methods interact with the same underlying variables defined in config/base_config.py.

Method 1: Edit base_config.py

The most direct approach is modifying the global configuration values in config/base_config.py. Set the three required variables to enable and configure your static proxy:

# config/base_config.py

ENABLE_IP_PROXY = True               # Turn on proxy support

IP_PROXY_PROVIDER_NAME = "static"   # Select static provider

STATIC_PROXY_URL = "http://user:passwd@proxy.example.com:8080"

These configuration variables are referenced throughout the codebase, including in cmd_arg/arg.py where command-line overrides are processed.

Method 2: Use Command-Line Arguments

MediaCrawler exposes the same configuration options as CLI flags, allowing you to override settings without editing files. This is useful for containerized deployments or temporary testing:

python -m MediaCrawler.main \
    --enable_ip_proxy true \
    --ip_proxy_provider_name static \
    --static_proxy_url "http://user:passwd@proxy.example.com:8080"

The argument definitions in cmd_arg/arg.py map these flags directly to the configuration variables, ensuring consistency between file-based and command-line configuration.

How Static Proxy Providers Work

When IP_PROXY_PROVIDER_NAME is set to "static" (case-insensitive), MediaCrawler initializes the StaticProxyProvider class located in proxy/proxy_ip_pool.py. This provider creates a single persistent proxy entry rather than rotating through a pool.

URL Parsing and Authentication

The StaticProxyProvider.get_proxy() method parses STATIC_PROXY_URL using Python's urlparse module to extract connection details:

# proxy/proxy_ip_pool.py – StaticProxyProvider implementation

proxy_url = getattr(config, "STATIC_PROXY_URL", "")
parsed = urlparse(proxy_url)
scheme = parsed.scheme or "http"
ip = parsed.hostname or ""
port = parsed.port or (443 if scheme == "https" else 80)

The implementation supports two URL formats:

• Unauthenticated: http://your_domain:port
• Authenticated: http://user:password@your_domain:port

Both http and https schemes are valid. Credentials are automatically URL-decoded using unquote() to handle special characters.

IpInfoModel Creation

The provider returns an IpInfoModel instance with a far-future expiration timestamp, ensuring the proxy never expires from the pool's perspective:

return [
    IpInfoModel(
        ip=ip,
        port=port,
        user=unquote(parsed.username or ""),
        password=unquote(parsed.password or ""),
        protocol=f"{scheme}://",
        expired_time_ts=int(time.time()) + 99999999,  # never expires

    )
]

This model is consumed by ProxyIpPool.get_or_refresh_proxy(), which supplies the proxy to HTTP clients throughout the application.

Integration with Crawler Clients

MediaCrawler's HTTP clients inherit from ProxyRefreshMixin, which automatically retrieves the current proxy before each request. For example, in media_platform/zhihu/client.py and other platform clients, the mixin calls self.proxy_ip_pool.get_or_refresh_proxy() to obtain the static proxy:

# Inside API client implementations

proxy = await self.proxy_ip_pool.get_or_refresh_proxy()
proxy_url = f"{proxy.protocol}{proxy.ip}:{proxy.port}"
async with make_async_client(proxy=proxy_url) as client:
    resp = await client.get("https://api.targetplatform.com/endpoint")

This architecture ensures that once you configure the static provider in config/base_config.py or via CLI, all requests automatically use your specified proxy without additional code changes.

Summary

• Static proxy mode uses a single, persistent proxy endpoint for all crawler requests
• Configuration requires three variables: ENABLE_IP_PROXY, IP_PROXY_PROVIDER_NAME="static", and STATIC_PROXY_URL
• Implementation resides in proxy/proxy_ip_pool.py via the StaticProxyProvider class
• Supported formats include authenticated and unauthenticated HTTP/HTTPS URLs
• Integration is automatic through ProxyRefreshMixin in platform client classes

Frequently Asked Questions

What URL formats are supported for static proxies?

MediaCrawler supports http://domain:port for unauthenticated proxies and http://user:password@domain:port for authenticated connections. Both http and https schemes are valid. The StaticProxyProvider class in proxy/proxy_ip_pool.py automatically extracts host, port, credentials, and protocol from the URL.

Does the static proxy ever expire?

No. The StaticProxyProvider creates an IpInfoModel with an expired_time_ts value set to the current time plus 99,999,999 seconds (approximately 3 years). This ensures the proxy remains valid indefinitely within the ProxyIpPool lifecycle, eliminating the need for refresh logic used by rotating providers.

Can I use HTTPS static proxies?

Yes. The URL parser in StaticProxyProvider recognizes both http and https schemes. If no port is specified, it defaults to 443 for HTTPS and 80 for HTTP. The protocol field in IpInfoModel preserves your specified scheme for use in actual HTTP requests.

How do I switch from static to rotating proxy providers?

Change IP_PROXY_PROVIDER_NAME to "kuaidaili" or "wandouhttp" and remove or comment out STATIC_PROXY_URL. These providers implement different get_proxy() methods that fetch fresh IPs from external APIs rather than using the persistent static configuration.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}