# How to Implement Custom Proxy Rotation Strategies in Scrapling

> Learn to implement custom proxy rotation strategies in Scrapling. Create a callable for thread-safe proxy selection and integrate with fetchers.

- Repository: [Karim shoair/Scrapling](https://github.com/D4Vinci/Scrapling)
- Tags: how-to-guide
- Published: 2026-03-08

---

**Implement custom proxy rotation in Scrapling by creating a callable that matches the `RotationStrategy` type signature and passing it to the `ProxyRotator` class, which manages thread-safe proxy selection and integrates with both synchronous and asynchronous fetchers.**

Scrapling provides a flexible, extensible proxy management system centered around the `ProxyRotator` class. By implementing custom proxy rotation strategies, you can control exactly how Scrapling selects proxies for each request—whether you need sticky sessions, weighted distributions, or error-aware failover logic.

## Understanding Scrapling's ProxyRotator Architecture

The proxy rotation system lives in [`scrapling/engines/toolbelt/proxy_rotation.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/engines/toolbelt/proxy_rotation.py) and consists of several tightly integrated components designed for thread-safe concurrent access.

### Core Components

| Component | File Path | Purpose |
|-----------|-----------|---------|
| **`ProxyRotator`** | [`scrapling/engines/toolbelt/proxy_rotation.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/engines/toolbelt/proxy_rotation.py) (line 39) | Thread-safe container that maintains the proxy pool, a lookup map for deduplication, and manages concurrent access via locks. |
| **`_get_proxy_key`** | [`scrapling/engines/toolbelt/proxy_rotation.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/engines/toolbelt/proxy_rotation.py) (line 18) | Generates unique keys for proxies (URL strings or `server\|username` for authenticated dicts) to enable fast lookup. |
| **`cyclic_rotation`** | [`scrapling/engines/toolbelt/proxy_rotation.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/engines/toolbelt/proxy_rotation.py) (line 33) | Default rotation strategy that iterates sequentially through the list and wraps around at the end. |
| **`is_proxy_error`** | [`scrapling/engines/toolbelt/proxy_rotation.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/engines/toolbelt/proxy_rotation.py) (line 27) | Helper function that detects proxy-related failures by scanning error messages for known proxy error substrings. |

### The RotationStrategy Type Contract

Custom strategies must conform to the `RotationStrategy` type alias defined in the module:

```python
from typing import Callable, List, Tuple, Union

ProxyType = Union[str, dict]  # URL string or Playwright-style proxy dict

RotationStrategy = Callable[[List[ProxyType], int], Tuple[ProxyType, int]]

```

The callable receives:
- `proxies`: The current list of available proxies
- `current_idx`: The index from the previous rotation (or 0 on first call)

It must return:
- The selected proxy
- The next index to pass on subsequent calls

## Implementing a Custom Rotation Strategy

Because strategies are plain callables, you can implement any selection logic without subclassing. The `ProxyRotator` handles thread safety, so your strategy only needs to focus on selection algorithms.

### Basic Sticky Proxy Strategy

A sticky strategy keeps returning the same proxy until explicitly told to advance (e.g., after a failure):

```python
from typing import List, Tuple, Union
from scrapling.engines.toolbelt.proxy_rotation import ProxyRotator

ProxyType = Union[str, dict]

def sticky_rotation(proxies: List[ProxyType], current_idx: int) -> Tuple[ProxyType, int]:
    """Always return the first proxy; index never advances automatically."""
    return proxies[0], current_idx

proxies = [
    "http://primary-proxy:8080",
    "http://backup-proxy:8080",
]

rotator = ProxyRotator(proxies, strategy=sticky_rotation)

# Both calls return the primary proxy

print(rotator.get_proxy())  # → http://primary-proxy:8080

print(rotator.get_proxy())  # → http://primary-proxy:8080

```

### Weighted Round-Robin Strategy

For proxies with different bandwidth limits or reliability scores, implement weighted selection:

```python
import random
from typing import List, Tuple
from scrapling.engines.toolbelt.proxy_rotation import ProxyRotator

def weighted_rotation(
    weighted_proxies: List[Tuple[str, int]], 
    current_idx: int
) -> Tuple[str, int]:
    """
    weighted_proxies is a list of (proxy_url, weight).
    Builds a weighted pool and selects randomly.
    """
    pool = []
    for idx, (proxy, weight) in enumerate(weighted_proxies):
        pool.extend([idx] * weight)
    
    chosen_idx = random.choice(pool)
    return weighted_proxies[chosen_idx][0], chosen_idx

# High-performance proxy gets weight 5, standard proxy gets weight 1

proxy_list = [
    ("http://fast-proxy:8080", 5),
    ("http://slow-proxy:8080", 1),
]

rotator = ProxyRotator(proxy_list, strategy=weighted_rotation)

```

### Error-Aware Failover Strategy

Combine custom rotation with Scrapling's `is_proxy_error` helper to implement automatic failover on proxy failures:

```python
from typing import List, Tuple, Union
from scrapling.engines.toolbelt.proxy_rotation import (
    ProxyRotator, 
    is_proxy_error
)

ProxyType = Union[str, dict]

class FailoverRotation:
    """Stateful strategy that advances on proxy errors."""
    
    def __init__(self):
        self.last_error = None
    
    def set_last_error(self, error: Exception):
        self.last_error = error
    
    def __call__(
        self, 
        proxies: List[ProxyType], 
        current_idx: int
    ) -> Tuple[ProxyType, int]:
        if self.last_error and is_proxy_error(self.last_error):
            # Advance to next proxy on error

            next_idx = (current_idx + 1) % len(proxies)
            self.last_error = None
            return proxies[next_idx], next_idx
        
        return proxies[current_idx], current_idx

# Usage with error tracking

failover = FailoverRotation()
rotator = ProxyRotator(
    ["http://p1:8080", "http://p2:8080"],
    strategy=failover
)

try:
    proxy = rotator.get_proxy()
    # ... make request ...

except Exception as e:
    failover.set_last_error(e)
    proxy = rotator.get_proxy()  # Automatically gets next proxy

```

## Integrating Custom Strategies with Scrapling Fetchers

Once you've defined a custom strategy, pass the `ProxyRotator` instance to Scrapling's fetcher classes. The rotator works with both synchronous `Fetcher` and asynchronous `AsyncFetcher`:

```python
from scrapling import Fetcher
from scrapling.engines.toolbelt.proxy_rotation import ProxyRotator

# Define your custom rotator

proxies = ["http://proxy1:8080", "http://proxy2:8080"]
rotator = ProxyRotator(proxies, strategy=weighted_rotation)

# Use with Fetcher

fetcher = Fetcher(proxy_rotator=rotator)
page = fetcher.get("https://example.com")

```

For Playwright-based fetchers that require dictionary-style proxy configurations (with `server`, `username`, `password` keys), the `ProxyRotator` handles both string URLs and dictionaries transparently:

```python
proxies = [
    {"server": "http://proxy1:8080", "username": "user", "password": "pass"},
    {"server": "http://proxy2:8080", "username": "user", "password": "pass"},
]
rotator = ProxyRotator(proxies)

```

## Summary

- **Scrapling's proxy rotation** is handled by the `ProxyRotator` class in [`scrapling/engines/toolbelt/proxy_rotation.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/engines/toolbelt/proxy_rotation.py), which provides thread-safe proxy management and deduplication.
- **Custom strategies** are implemented as callables matching the `RotationStrategy` type signature: `Callable[[List[ProxyType], int], Tuple[ProxyType, int]]`.
- **The default `cyclic_rotation`** strategy provides simple round-robin behavior, but you can replace it with sticky sessions, weighted selection, or error-aware failover logic.
- **Error detection** uses the `is_proxy_error()` helper to identify proxy-related failures, enabling automatic failover in custom strategies.
- **Integration** requires passing the configured `ProxyRotator` instance to Scrapling's `Fetcher` or `AsyncFetcher` classes via the `proxy_rotator` parameter.

## Frequently Asked Questions

### What is the default proxy rotation strategy in Scrapling?

Scrapling uses **`cyclic_rotation`** by default, which iterates sequentially through the proxy list and wraps back to the beginning when it reaches the end. This provides simple round-robin distribution across all configured proxies without requiring any custom configuration.

### How does Scrapling handle proxy authentication?

The `ProxyRotator` accepts both plain URL strings (e.g., `http://user:pass@proxy:8080`) and Playwright-style dictionaries containing `server`, `username`, and `password` keys. The internal `_get_proxy_key` function generates unique identifiers for each proxy to prevent duplicates, handling both authentication formats transparently.

### Can I use proxy rotation with Playwright-based fetchers?

Yes. The `ProxyRotator` is format-agnostic and works with both standard HTTP fetchers and Playwright-based fetchers. When using Playwright, provide proxy definitions as dictionaries with `server`, `username`, and `password` keys, and pass the rotator instance to your fetcher's `proxy_rotator` parameter.

### How do I detect proxy failures in custom strategies?

Import the **`is_proxy_error`** function from `scrapling.engines.toolbelt.proxy_rotation`. This helper examines exception messages for known proxy-related substrings (connection refused, timeout, authentication errors, etc.) and returns a boolean. Use this inside your custom strategy to trigger failover logic when a proxy error is detected.