# Configuring Multi-Session Support for Different Session Types in a Single Scrapling Spider

> Learn to configure multi session support for diverse session types in a Scrapling spider. Override configure sessions and route requests using the sid parameter for efficient web scraping.

- Repository: [Karim shoair/Scrapling](https://github.com/D4Vinci/Scrapling)
- Tags: how-to-guide
- Published: 2026-03-08

---

**Override `Spider.configure_sessions` to register multiple session implementations—such as `FetcherSession`, `AsyncDynamicSession`, and `AsyncStealthySession`—then route each `Request` to the appropriate session via the `sid` parameter.**

Configuring multi-session support for different session types in a single Scrapling spider allows you to combine lightweight HTTP clients, full-browser rendering, and stealth automation within one crawl. The D4Vinci/Scrapling repository provides a `SessionManager` that handles session registration, lazy initialization, and lifecycle management, while the `Spider` class exposes a `configure_sessions` hook for declaring which fetchers to use.

## Understanding Scrapling's Session Architecture

The architecture centers on the **`SessionManager`** class in [`scrapling/spiders/session.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/spiders/session.py). This manager maintains a dictionary mapping session IDs to session instances, tracks which session serves as the default, and supports lazy-start semantics for resource-heavy browsers.

When a spider starts, the `CrawlerEngine` (defined in [`scrapling/spiders/engine.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/spiders/engine.py)) instantiates the spider, which in turn creates a `SessionManager` and calls `self.configure_sessions(manager)`. The default implementation in [`scrapling/spiders/spider.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/spiders/spider.py) registers a single `FetcherSession` under the ID `"default"`.

## Registering Multiple Session Types in configure_sessions

To enable multi-session support, override `configure_sessions` in your spider subclass. Use `manager.add(session_id, session_instance, *, default=False, lazy=False)` to register each fetcher:

- **`default=True`** designates the fallback session for requests that omit an explicit `sid`.
- **`lazy=True`** defers the session's `__aenter__` (startup) until the first request actually references that session ID, conserving resources for infrequently used browsers.

```python
from scrapling.spiders import Spider
from scrapling.fetchers import FetcherSession, AsyncDynamicSession, AsyncStealthySession

class MultiSessionSpider(Spider):
    name = "multi_session"
    
    def configure_sessions(self, manager):
        # Lightweight HTTP session for API calls

        manager.add("api", FetcherSession(), default=True)
        
        # Chrome-based rendering for JavaScript-heavy pages

        manager.add(
            "chrome",
            AsyncDynamicSession(
                headless=False,
                disable_resources=True,
                timeout=60_000,
            )
        )
        
        # Stealth session for anti-bot protected sites

        manager.add(
            "stealth",
            AsyncStealthySession(
                headless=True,
                solve_cloudflare=True,
                extra_flags=["--disable-blink-features=AutomationControlled"],
                timeout=90_000,
            ),
            lazy=True,  # Only starts when first "stealth" request is made

        )

```

## Routing Requests to Specific Sessions

The `Request` object in [`scrapling/spiders/request.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/spiders/request.py) carries an optional **`sid`** attribute. When the `CrawlerEngine` processes a request, it passes the request to `SessionManager.fetch(request)`, which resolves the session ID via `manager.get(request.sid)` or falls back to the default session if `sid` is `None`.

Explicitly set `sid` in `start_requests` or when yielding new requests:

```python
async def start_requests(self):
    # Uses default "api" session

    yield self.request("https://api.example.com/data")
    
    # Explicitly routes to Chrome session

    yield self.request(
        "https://spa.example.com/dashboard",
        sid="chrome",
    )
    
    # Triggers lazy initialization of stealth session

    yield self.request(
        "https://protected.example.com",
        sid="stealth",
    )

```

Inside `parse` callbacks, inspect `response.session_id` to determine which session fetched the page:

```python
async def parse(self, response):
    if response.session_id == "stealth":
        self.logger.info("Successfully bypassed protection")
    yield {"url": response.url, "session": response.session_id}

```

## Session Lifecycle and Resource Management

The `SessionManager` handles asynchronous context management automatically. When a spider enters its async context (started by the engine), `SessionManager.start()` executes the `__aenter__` method of every **non-lazy** session. Lazy sessions remain dormant until `SessionManager.fetch` first references them, at which point a lock ensures only one coroutine triggers the startup.

When the spider finishes or pauses, `SessionManager.close()` invokes each session's `__aexit__` to shut down browsers, close HTTP connections, and release file descriptors. This lifecycle is verified in [`tests/spiders/test_session.py`](https://github.com/D4Vinci/Scrapling/blob/main/tests/spiders/test_session.py), which covers edge cases such as idempotent `start()` calls and concurrent lazy initialization.

## Complete Working Example

The following spider demonstrates production-ready multi-session configuration, combining all three session types with explicit routing and lazy initialization:

```python

# multi_session_spider.py

from scrapling.spiders import Spider
from scrapling.fetchers import (
    FetcherSession,
    AsyncDynamicSession,
    AsyncStealthySession,
)

class ProductionSpider(Spider):
    name = "production_spider"
    start_urls = ["https://example.com"]

    def configure_sessions(self, manager):
        # Default: lightweight HTTP client

        manager.add("http", FetcherSession(), default=True)
        
        # Dynamic rendering for SPAs

        manager.add(
            "dynamic",
            AsyncDynamicSession(
                headless=True,
                disable_resources=True,
                timeout=45_000,
            )
        )
        
        # Stealth mode for protected sites (lazy start)

        manager.add(
            "stealth",
            AsyncStealthySession(
                headless=True,
                solve_cloudflare=True,
                timeout=120_000,
            ),
            lazy=True,
        )

    async def start_requests(self):
        # Static/API content via default session

        for url in self.start_urls:
            yield self.request(url)
            
        # JavaScript-heavy page

        yield self.request(
            "https://spa.example.com/products",
            sid="dynamic",
        )
        
        # Anti-bot protected page

        yield self.request(
            "https://protected.example.com/data",
            sid="stealth",
        )

    async def parse(self, response):
        self.logger.info(
            f"Parsed {response.url} using session '{response.session_id}'"
        )
        yield {
            "url": response.url,
            "session_used": response.session_id,
            "title": response.css("title::text").get(),
        }

```

## Key Source Files

| File | Purpose |
|------|---------|
| [`scrapling/spiders/session.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/spiders/session.py) | `SessionManager` implementation – registration, lazy start, fetch routing, and lifecycle management |
| [`scrapling/spiders/spider.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/spiders/spider.py) | Abstract `Spider` class with `configure_sessions` hook and request generation |
| [`scrapling/spiders/request.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/spiders/request.py) | `Request` dataclass carrying URL, `sid`, and metadata |
| [`scrapling/spiders/engine.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/spiders/engine.py) | `CrawlerEngine` orchestrating requests and session resolution |
| [`scrapling/fetchers/requests.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/fetchers/requests.py) | `FetcherSession` – lightweight HTTP client based on `curl_cffi` |
| [`scrapling/fetchers/chrome.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/fetchers/chrome.py) | `AsyncDynamicSession` – Playwright Chrome rendering |
| [`scrapling/fetchers/stealth_chrome.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/fetchers/stealth_chrome.py) | `AsyncStealthySession` – stealth Chromium with anti-detection |
| [`tests/spiders/test_session.py`](https://github.com/D4Vinci/Scrapling/blob/main/tests/spiders/test_session.py) | Unit tests documenting `SessionManager` behavior and edge cases |

## Summary

- **Register sessions** in `configure_sessions` using `manager.add(id, session, default=..., lazy=...)` to mix HTTP clients, dynamic browsers, and stealth implementations.
- **Route requests** by setting `sid="<session_id>"` in `Request` objects; omit `sid` to use the default session.
- **Conserve resources** by marking heavyweight sessions as `lazy=True` so they only start when first referenced.
- **Inspect responses** via `response.session_id` to apply conditional parsing logic based on the fetcher used.
- **Lifecycle management** is automatic—`SessionManager` handles async startup and shutdown for all registered sessions.

## Frequently Asked Questions

### How do I set a default session for requests that don't specify a session ID?

Pass `default=True` when calling `manager.add()` in your `configure_sessions` method. Only one session can be the default; the last one registered with `default=True` wins. Requests that omit the `sid` parameter automatically route to this default session.

### Can I use lazy loading for browser-based sessions to save memory?

Yes. When registering expensive sessions like `AsyncDynamicSession` or `AsyncStealthySession`, pass `lazy=True` to `manager.add()`. The session's `__aenter__` (browser startup) is deferred until the first request explicitly referencing that session ID enters the fetch queue, preventing unnecessary resource consumption during crawls that might not need that session type.

### What happens if I reference a session ID that hasn't been registered?

The `SessionManager.get(sid)` method raises a `KeyError` if the requested session ID does not exist in the registry. To avoid this, ensure all session IDs used in `start_requests` or `parse` callbacks are registered in `configure_sessions`, or implement error handling to fall back to the default session when appropriate.

### How can I inspect which session fetched a response inside my parse callback?

The `Response` object returned to your `parse` method includes a `session_id` attribute populated by `SessionManager.fetch()` in [`scrapling/spiders/session.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/spiders/session.py). Check `response.session_id` to conditionally execute parsing logic—for example, applying additional anti-bot checks when the `"stealth"` session was used, or extracting JSON directly when the `"api"` session was used.