How to Implement Custom Proxy Rotation Strategies in Scrapling

Implement custom proxy rotation in Scrapling by creating a callable that matches the RotationStrategy type signature and passing it to the ProxyRotator class, which manages thread-safe proxy selection and integrates with both synchronous and asynchronous fetchers.

Scrapling provides a flexible, extensible proxy management system centered around the ProxyRotator class. By implementing custom proxy rotation strategies, you can control exactly how Scrapling selects proxies for each request—whether you need sticky sessions, weighted distributions, or error-aware failover logic.

Understanding Scrapling's ProxyRotator Architecture

The proxy rotation system lives in scrapling/engines/toolbelt/proxy_rotation.py and consists of several tightly integrated components designed for thread-safe concurrent access.

Core Components

Component	File Path	Purpose
ProxyRotator	scrapling/engines/toolbelt/proxy_rotation.py (line 39)	Thread-safe container that maintains the proxy pool, a lookup map for deduplication, and manages concurrent access via locks.
_get_proxy_key	scrapling/engines/toolbelt/proxy_rotation.py (line 18)	Generates unique keys for proxies (URL strings or server|username for authenticated dicts) to enable fast lookup.
cyclic_rotation	scrapling/engines/toolbelt/proxy_rotation.py (line 33)	Default rotation strategy that iterates sequentially through the list and wraps around at the end.
is_proxy_error	scrapling/engines/toolbelt/proxy_rotation.py (line 27)	Helper function that detects proxy-related failures by scanning error messages for known proxy error substrings.

The RotationStrategy Type Contract

Custom strategies must conform to the RotationStrategy type alias defined in the module:

from typing import Callable, List, Tuple, Union

ProxyType = Union[str, dict]  # URL string or Playwright-style proxy dict

RotationStrategy = Callable[[List[ProxyType], int], Tuple[ProxyType, int]]

The callable receives:

• proxies: The current list of available proxies
• current_idx: The index from the previous rotation (or 0 on first call)

It must return:

• The selected proxy
• The next index to pass on subsequent calls

Implementing a Custom Rotation Strategy

Because strategies are plain callables, you can implement any selection logic without subclassing. The ProxyRotator handles thread safety, so your strategy only needs to focus on selection algorithms.

Basic Sticky Proxy Strategy

A sticky strategy keeps returning the same proxy until explicitly told to advance (e.g., after a failure):

from typing import List, Tuple, Union
from scrapling.engines.toolbelt.proxy_rotation import ProxyRotator

ProxyType = Union[str, dict]

def sticky_rotation(proxies: List[ProxyType], current_idx: int) -> Tuple[ProxyType, int]:
    """Always return the first proxy; index never advances automatically."""
    return proxies[0], current_idx

proxies = [
    "http://primary-proxy:8080",
    "http://backup-proxy:8080",
]

rotator = ProxyRotator(proxies, strategy=sticky_rotation)

# Both calls return the primary proxy

print(rotator.get_proxy())  # → http://primary-proxy:8080

print(rotator.get_proxy())  # → http://primary-proxy:8080

Weighted Round-Robin Strategy

For proxies with different bandwidth limits or reliability scores, implement weighted selection:

import random
from typing import List, Tuple
from scrapling.engines.toolbelt.proxy_rotation import ProxyRotator

def weighted_rotation(
    weighted_proxies: List[Tuple[str, int]], 
    current_idx: int
) -> Tuple[str, int]:
    """
    weighted_proxies is a list of (proxy_url, weight).
    Builds a weighted pool and selects randomly.
    """
    pool = []
    for idx, (proxy, weight) in enumerate(weighted_proxies):
        pool.extend([idx] * weight)
    
    chosen_idx = random.choice(pool)
    return weighted_proxies[chosen_idx][0], chosen_idx

# High-performance proxy gets weight 5, standard proxy gets weight 1

proxy_list = [
    ("http://fast-proxy:8080", 5),
    ("http://slow-proxy:8080", 1),
]

rotator = ProxyRotator(proxy_list, strategy=weighted_rotation)

Error-Aware Failover Strategy

Combine custom rotation with Scrapling's is_proxy_error helper to implement automatic failover on proxy failures:

from typing import List, Tuple, Union
from scrapling.engines.toolbelt.proxy_rotation import (
    ProxyRotator, 
    is_proxy_error
)

ProxyType = Union[str, dict]

class FailoverRotation:
    """Stateful strategy that advances on proxy errors."""
    
    def __init__(self):
        self.last_error = None
    
    def set_last_error(self, error: Exception):
        self.last_error = error
    
    def __call__(
        self, 
        proxies: List[ProxyType], 
        current_idx: int
    ) -> Tuple[ProxyType, int]:
        if self.last_error and is_proxy_error(self.last_error):
            # Advance to next proxy on error

            next_idx = (current_idx + 1) % len(proxies)
            self.last_error = None
            return proxies[next_idx], next_idx
        
        return proxies[current_idx], current_idx

# Usage with error tracking

failover = FailoverRotation()
rotator = ProxyRotator(
    ["http://p1:8080", "http://p2:8080"],
    strategy=failover
)

try:
    proxy = rotator.get_proxy()
    # ... make request ...

except Exception as e:
    failover.set_last_error(e)
    proxy = rotator.get_proxy()  # Automatically gets next proxy

Integrating Custom Strategies with Scrapling Fetchers

Once you've defined a custom strategy, pass the ProxyRotator instance to Scrapling's fetcher classes. The rotator works with both synchronous Fetcher and asynchronous AsyncFetcher:

from scrapling import Fetcher
from scrapling.engines.toolbelt.proxy_rotation import ProxyRotator

# Define your custom rotator

proxies = ["http://proxy1:8080", "http://proxy2:8080"]
rotator = ProxyRotator(proxies, strategy=weighted_rotation)

# Use with Fetcher

fetcher = Fetcher(proxy_rotator=rotator)
page = fetcher.get("https://example.com")

For Playwright-based fetchers that require dictionary-style proxy configurations (with server, username, password keys), the ProxyRotator handles both string URLs and dictionaries transparently:

proxies = [
    {"server": "http://proxy1:8080", "username": "user", "password": "pass"},
    {"server": "http://proxy2:8080", "username": "user", "password": "pass"},
]
rotator = ProxyRotator(proxies)

Summary

• Scrapling's proxy rotation is handled by the ProxyRotator class in scrapling/engines/toolbelt/proxy_rotation.py, which provides thread-safe proxy management and deduplication.
• Custom strategies are implemented as callables matching the RotationStrategy type signature: Callable[[List[ProxyType], int], Tuple[ProxyType, int]].
• The default cyclic_rotation strategy provides simple round-robin behavior, but you can replace it with sticky sessions, weighted selection, or error-aware failover logic.
• Error detection uses the is_proxy_error() helper to identify proxy-related failures, enabling automatic failover in custom strategies.
• Integration requires passing the configured ProxyRotator instance to Scrapling's Fetcher or AsyncFetcher classes via the proxy_rotator parameter.

Frequently Asked Questions

What is the default proxy rotation strategy in Scrapling?

Scrapling uses cyclic_rotation by default, which iterates sequentially through the proxy list and wraps back to the beginning when it reaches the end. This provides simple round-robin distribution across all configured proxies without requiring any custom configuration.

How does Scrapling handle proxy authentication?

The ProxyRotator accepts both plain URL strings (e.g., http://user:pass@proxy:8080) and Playwright-style dictionaries containing server, username, and password keys. The internal _get_proxy_key function generates unique identifiers for each proxy to prevent duplicates, handling both authentication formats transparently.

Can I use proxy rotation with Playwright-based fetchers?

Yes. The ProxyRotator is format-agnostic and works with both standard HTTP fetchers and Playwright-based fetchers. When using Playwright, provide proxy definitions as dictionaries with server, username, and password keys, and pass the rotator instance to your fetcher's proxy_rotator parameter.

How do I detect proxy failures in custom strategies?

Import the is_proxy_error function from scrapling.engines.toolbelt.proxy_rotation. This helper examines exception messages for known proxy-related substrings (connection refused, timeout, authentication errors, etc.) and returns a boolean. Use this inside your custom strategy to trigger failover logic when a proxy error is detected.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}