# Routing Requests to Different Sessions by ID in a Multi-Session Scrapling Spider

> Route Scrapling requests to specific sessions by ID. Learn how to manage multiple sessions and assign sid parameters for targeted traffic in your spider.

- Repository: [Karim shoair/Scrapling](https://github.com/D4Vinci/Scrapling)
- Tags: how-to-guide
- Published: 2026-03-08

---

**To route requests to different sessions by ID in Scrapling, register multiple sessions in `configure_sessions()` using `manager.add()`, then assign the `sid` parameter when creating `Request` objects to direct traffic to specific fetcher configurations.**

Scrapling is a modern Python web scraping framework that decouples request logic from HTTP execution. When building complex multi-session spiders, you need fine-grained control over which session handles each request—whether routing API calls through lightweight fetchers or rendering JavaScript through stealth browsers. This guide explains how to implement routing requests to different sessions by ID in a multi-session Scrapling spider using the framework's `SessionManager` and `Request` objects.

## How Session Routing Works in Scrapling

Scrapling's architecture separates **what** a request asks for (URL, callback, metadata) from **how** the HTTP transaction is performed (the underlying session). Three core components handle routing requests to different sessions by ID.

### The Request Object and Session IDs

The `Request` class in [`scrapling/spiders/request.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/spiders/request.py) accepts an optional `sid` parameter during initialization. This identifier tells the engine which session should execute the request.

```python

# scrapling/spiders/request.py

class Request:
    def __init__(self, url, callback=None, sid=None, **kwargs):
        self.url = url
        self.callback = callback
        self.sid = sid  # Session ID for routing

        # ... additional metadata handling

```

When you omit `sid`, the spider falls back to the default session configured in the `SessionManager`.

### The SessionManager Registry

The `SessionManager` class in [`scrapling/spiders/session.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/spiders/session.py) maintains a registry of session objects and handles the routing logic. Key methods include:

- **`add(id, session, default=False, lazy=False)`**: Registers a session with an identifier. Setting `default=True` makes it the fallback for requests without `sid`. Setting `lazy=True` defers session initialization until the first request targets it.
- **`get(sid)`**: Retrieves a session by ID, starting it if marked as lazy.
- **`fetch(request)`**: The core routing method that resolves the effective session ID and delegates the HTTP call.

```python

# scrapling/spiders/session.py

class SessionManager:
    def fetch(self, request):
        # Resolve session ID (fallback to default if None)

        sid = request.sid or self.default_session_id
        session = self.get(sid)
        
        # Delegate to the specific session implementation

        if hasattr(session, 'fetch'):
            return session.fetch(request)
        # ... additional handling for sync sessions

```

### The Spider Configuration Hook

The base `Spider` class in [`scrapling/spiders/spider.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/spiders/spider.py) provides the `configure_sessions` hook. This is where you register multiple sessions for routing.

```python

# scrapling/spiders/spider.py

class Spider:
    def configure_sessions(self, manager: SessionManager) -> None:
        # Default implementation adds a basic FetcherSession

        manager.add("default", FetcherSession(), default=True)
    
    async def start_requests(self):
        # Uses default session ID if sid not specified

        for url in self.start_urls:
            yield Request(url, sid=self._session_manager.default_session_id)

```

## Implementing Multi-Session Routing

To route requests to different sessions by ID, you must register diverse session types in `configure_sessions` and then assign `sid` values when creating requests.

### Registering Multiple Sessions

Override `configure_sessions` to register different fetcher configurations. The first session added becomes the default unless you explicitly mark another with `default=True`.

```python
from scrapling.spiders.spider import Spider
from scrapling.spiders.request import Request
from scrapling.fetchers import FetcherSession, AsyncStealthySession
from scrapling.spiders.session import SessionManager

class MultiSessionSpider(Spider):
    name = "multi-session"
    start_urls = ["https://api.example.com/data", "https://protected-site.com"]

    def configure_sessions(self, manager: SessionManager) -> None:
        # Lightweight session for API calls (default)

        manager.add("api", FetcherSession(), default=True)
        
        # Stealth browser for protected sites (lazy initialization)

        manager.add("stealth", AsyncStealthySession(headless=True), lazy=True)

```

### Directing Requests to Specific Sessions

In `start_requests` or callbacks, instantiate `Request` objects with the `sid` parameter to route them to specific sessions.

```python
    async def start_requests(self):
        # Uses default "api" session

        yield Request("https://api.example.com/data", callback=self.parse_api)
        
        # Explicitly routes to stealth session

        yield Request(
            "https://protected-site.com", 
            callback=self.parse_protected,
            sid="stealth"
        )

```

### Accessing Session Information in Callbacks

The response object retains a reference to the original request, allowing you to inspect which session handled the fetch.

```python
    async def parse_api(self, response):
        # Verify routing worked as expected

        if response.request.sid == "api":
            self.logger.info("API response received via FetcherSession")
        
        yield {"source": "api", "data": response.json()}

    async def parse_protected(self, response):
        # Confirm stealth session handled this request

        self.logger.info(f"Protected site rendered via {response.request.sid}")
        yield {"source": "stealth", "title": response.css("h1::text").get()}

```

## Key Source Files and Implementation Details

Understanding the routing mechanism requires familiarity with these specific files in the Scrapling repository:

| File | Purpose | Key Components |
|------|---------|----------------|
| [`scrapling/spiders/request.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/spiders/request.py) | Defines the `Request` object and its `sid` attribute | `Request.__init__` accepts `sid` parameter for session routing |
| [`scrapling/spiders/session.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/spiders/session.py) | Implements `SessionManager` with registry and routing logic | `add()`, `get()`, `fetch()` methods handle session resolution and lazy initialization |
| [`scrapling/spiders/spider.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/spiders/spider.py) | Base spider class providing the configuration hook | `configure_sessions()` method for registering sessions; `start_requests()` uses default session ID |
| [`scrapling/fetchers/requests.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/fetchers/requests.py) | `FetcherSession` implementation for standard HTTP | Used for lightweight API calls in multi-session setups |
| [`scrapling/fetchers/stealth_chrome.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/fetchers/stealth_chrome.py) | `AsyncStealthySession` for headless browser automation | Used for JavaScript rendering and anti-bot evasion |

The routing logic in `SessionManager.fetch()` specifically checks `request.sid` and falls back to `self.default_session_id` when the attribute is `None`, ensuring that every request is associated with a valid session instance before the HTTP transaction occurs.

## Summary

- **Session routing** in Scrapling relies on the `sid` parameter in `Request` objects to map requests to specific session instances managed by `SessionManager`.
- **Registration** occurs in `Spider.configure_sessions()` using `manager.add()`, where you can designate default sessions and enable lazy initialization to conserve resources.
- **Lazy sessions** defer browser startup or connection pool creation until the first request targeting that session ID is processed, improving efficiency in multi-session spiders.
- **Fallback behavior** automatically routes requests without an explicit `sid` to the default session configured in the manager, ensuring backward compatibility with simple spider implementations.

## Frequently Asked Questions

### What happens if I don't specify a session ID in my Request?

If you omit the `sid` parameter when creating a `Request`, Scrapling automatically routes the request to the default session configured in the `SessionManager`. According to the implementation in [`scrapling/spiders/session.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/spiders/session.py), the `fetch` method resolves the effective session ID using `request.sid or self.default_session_id`, ensuring every request is handled by a valid session instance.

### Can I switch sessions dynamically based on response content?

Yes, you can route subsequent requests to different sessions dynamically within your callback methods. Since callbacks receive the `Response` object containing the original `Request`, you can inspect the content and yield new `Request` objects with explicit `sid` parameters targeting different sessions. For example, if you detect a CAPTCHA challenge in a response, you can yield a new request with `sid="stealth"` to route it through a headless browser session.

### What is the difference between lazy and non-lazy sessions?

Lazy sessions defer initialization until the first request actually targets that session ID, while non-lazy sessions start immediately when the spider begins crawling. When you register a session with `manager.add("stealth", AsyncStealthySession(), lazy=True)`, the browser process doesn't launch until Scrapling encounters a request with `sid="stealth"`. This conserves memory and CPU resources when running spiders that may not use every registered session during a specific crawl.

### How do I access the session object directly in a callback?

While the `Response` object provides access to the request's session ID via `response.request.sid`, direct access to the session instance itself is typically not necessary since the fetch operation has already completed. However, if you need to interact with the session for advanced scenarios (such as manually managing cookies or checking browser state), you can access the active session through the spider's `_session_manager` attribute using `self._session_manager.get(response.request.sid)`. Note that this accesses internal APIs and should be used with caution.