Configuring Multi-Session Support for Different Session Types in a Single Scrapling Spider

Override Spider.configure_sessions to register multiple session implementations—such as FetcherSession, AsyncDynamicSession, and AsyncStealthySession—then route each Request to the appropriate session via the sid parameter.

Configuring multi-session support for different session types in a single Scrapling spider allows you to combine lightweight HTTP clients, full-browser rendering, and stealth automation within one crawl. The D4Vinci/Scrapling repository provides a SessionManager that handles session registration, lazy initialization, and lifecycle management, while the Spider class exposes a configure_sessions hook for declaring which fetchers to use.

Understanding Scrapling's Session Architecture

The architecture centers on the SessionManager class in scrapling/spiders/session.py. This manager maintains a dictionary mapping session IDs to session instances, tracks which session serves as the default, and supports lazy-start semantics for resource-heavy browsers.

When a spider starts, the CrawlerEngine (defined in scrapling/spiders/engine.py) instantiates the spider, which in turn creates a SessionManager and calls self.configure_sessions(manager). The default implementation in scrapling/spiders/spider.py registers a single FetcherSession under the ID "default".

Registering Multiple Session Types in configure_sessions

To enable multi-session support, override configure_sessions in your spider subclass. Use manager.add(session_id, session_instance, *, default=False, lazy=False) to register each fetcher:

• default=True designates the fallback session for requests that omit an explicit sid.
• lazy=True defers the session's __aenter__ (startup) until the first request actually references that session ID, conserving resources for infrequently used browsers.

from scrapling.spiders import Spider
from scrapling.fetchers import FetcherSession, AsyncDynamicSession, AsyncStealthySession

class MultiSessionSpider(Spider):
    name = "multi_session"
    
    def configure_sessions(self, manager):
        # Lightweight HTTP session for API calls

        manager.add("api", FetcherSession(), default=True)
        
        # Chrome-based rendering for JavaScript-heavy pages

        manager.add(
            "chrome",
            AsyncDynamicSession(
                headless=False,
                disable_resources=True,
                timeout=60_000,
            )
        )
        
        # Stealth session for anti-bot protected sites

        manager.add(
            "stealth",
            AsyncStealthySession(
                headless=True,
                solve_cloudflare=True,
                extra_flags=["--disable-blink-features=AutomationControlled"],
                timeout=90_000,
            ),
            lazy=True,  # Only starts when first "stealth" request is made

        )

Routing Requests to Specific Sessions

The Request object in scrapling/spiders/request.py carries an optional sid attribute. When the CrawlerEngine processes a request, it passes the request to SessionManager.fetch(request), which resolves the session ID via manager.get(request.sid) or falls back to the default session if sid is None.

Explicitly set sid in start_requests or when yielding new requests:

async def start_requests(self):
    # Uses default "api" session

    yield self.request("https://api.example.com/data")
    
    # Explicitly routes to Chrome session

    yield self.request(
        "https://spa.example.com/dashboard",
        sid="chrome",
    )
    
    # Triggers lazy initialization of stealth session

    yield self.request(
        "https://protected.example.com",
        sid="stealth",
    )

Inside parse callbacks, inspect response.session_id to determine which session fetched the page:

async def parse(self, response):
    if response.session_id == "stealth":
        self.logger.info("Successfully bypassed protection")
    yield {"url": response.url, "session": response.session_id}

Session Lifecycle and Resource Management

The SessionManager handles asynchronous context management automatically. When a spider enters its async context (started by the engine), SessionManager.start() executes the __aenter__ method of every non-lazy session. Lazy sessions remain dormant until SessionManager.fetch first references them, at which point a lock ensures only one coroutine triggers the startup.

When the spider finishes or pauses, SessionManager.close() invokes each session's __aexit__ to shut down browsers, close HTTP connections, and release file descriptors. This lifecycle is verified in tests/spiders/test_session.py, which covers edge cases such as idempotent start() calls and concurrent lazy initialization.

Complete Working Example

The following spider demonstrates production-ready multi-session configuration, combining all three session types with explicit routing and lazy initialization:

# multi_session_spider.py

from scrapling.spiders import Spider
from scrapling.fetchers import (
    FetcherSession,
    AsyncDynamicSession,
    AsyncStealthySession,
)

class ProductionSpider(Spider):
    name = "production_spider"
    start_urls = ["https://example.com"]

    def configure_sessions(self, manager):
        # Default: lightweight HTTP client

        manager.add("http", FetcherSession(), default=True)
        
        # Dynamic rendering for SPAs

        manager.add(
            "dynamic",
            AsyncDynamicSession(
                headless=True,
                disable_resources=True,
                timeout=45_000,
            )
        )
        
        # Stealth mode for protected sites (lazy start)

        manager.add(
            "stealth",
            AsyncStealthySession(
                headless=True,
                solve_cloudflare=True,
                timeout=120_000,
            ),
            lazy=True,
        )

    async def start_requests(self):
        # Static/API content via default session

        for url in self.start_urls:
            yield self.request(url)
            
        # JavaScript-heavy page

        yield self.request(
            "https://spa.example.com/products",
            sid="dynamic",
        )
        
        # Anti-bot protected page

        yield self.request(
            "https://protected.example.com/data",
            sid="stealth",
        )

    async def parse(self, response):
        self.logger.info(
            f"Parsed {response.url} using session '{response.session_id}'"
        )
        yield {
            "url": response.url,
            "session_used": response.session_id,
            "title": response.css("title::text").get(),
        }

Key Source Files

File	Purpose
scrapling/spiders/session.py	SessionManager implementation – registration, lazy start, fetch routing, and lifecycle management
scrapling/spiders/spider.py	Abstract Spider class with configure_sessions hook and request generation
scrapling/spiders/request.py	Request dataclass carrying URL, sid, and metadata
scrapling/spiders/engine.py	CrawlerEngine orchestrating requests and session resolution
scrapling/fetchers/requests.py	FetcherSession – lightweight HTTP client based on curl_cffi
scrapling/fetchers/chrome.py	AsyncDynamicSession – Playwright Chrome rendering
scrapling/fetchers/stealth_chrome.py	AsyncStealthySession – stealth Chromium with anti-detection
tests/spiders/test_session.py	Unit tests documenting SessionManager behavior and edge cases

Summary

• Register sessions in configure_sessions using manager.add(id, session, default=..., lazy=...) to mix HTTP clients, dynamic browsers, and stealth implementations.
• Route requests by setting sid="<session_id>" in Request objects; omit sid to use the default session.
• Conserve resources by marking heavyweight sessions as lazy=True so they only start when first referenced.
• Inspect responses via response.session_id to apply conditional parsing logic based on the fetcher used.
• Lifecycle management is automatic—SessionManager handles async startup and shutdown for all registered sessions.

Frequently Asked Questions

How do I set a default session for requests that don't specify a session ID?

Pass default=True when calling manager.add() in your configure_sessions method. Only one session can be the default; the last one registered with default=True wins. Requests that omit the sid parameter automatically route to this default session.

Can I use lazy loading for browser-based sessions to save memory?

Yes. When registering expensive sessions like AsyncDynamicSession or AsyncStealthySession, pass lazy=True to manager.add(). The session's __aenter__ (browser startup) is deferred until the first request explicitly referencing that session ID enters the fetch queue, preventing unnecessary resource consumption during crawls that might not need that session type.

What happens if I reference a session ID that hasn't been registered?

The SessionManager.get(sid) method raises a KeyError if the requested session ID does not exist in the registry. To avoid this, ensure all session IDs used in start_requests or parse callbacks are registered in configure_sessions, or implement error handling to fall back to the default session when appropriate.

How can I inspect which session fetched a response inside my parse callback?

The Response object returned to your parse method includes a session_id attribute populated by SessionManager.fetch() in scrapling/spiders/session.py. Check response.session_id to conditionally execute parsing logic—for example, applying additional anti-bot checks when the "stealth" session was used, or extracting JSON directly when the "api" session was used.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}