Routing Requests to Different Sessions by ID in a Multi-Session Scrapling Spider

To route requests to different sessions by ID in Scrapling, register multiple sessions in configure_sessions() using manager.add(), then assign the sid parameter when creating Request objects to direct traffic to specific fetcher configurations.

Scrapling is a modern Python web scraping framework that decouples request logic from HTTP execution. When building complex multi-session spiders, you need fine-grained control over which session handles each request—whether routing API calls through lightweight fetchers or rendering JavaScript through stealth browsers. This guide explains how to implement routing requests to different sessions by ID in a multi-session Scrapling spider using the framework's SessionManager and Request objects.

How Session Routing Works in Scrapling

Scrapling's architecture separates what a request asks for (URL, callback, metadata) from how the HTTP transaction is performed (the underlying session). Three core components handle routing requests to different sessions by ID.

The Request Object and Session IDs

The Request class in scrapling/spiders/request.py accepts an optional sid parameter during initialization. This identifier tells the engine which session should execute the request.

# scrapling/spiders/request.py

class Request:
    def __init__(self, url, callback=None, sid=None, **kwargs):
        self.url = url
        self.callback = callback
        self.sid = sid  # Session ID for routing

        # ... additional metadata handling

When you omit sid, the spider falls back to the default session configured in the SessionManager.

The SessionManager Registry

The SessionManager class in scrapling/spiders/session.py maintains a registry of session objects and handles the routing logic. Key methods include:

• add(id, session, default=False, lazy=False): Registers a session with an identifier. Setting default=True makes it the fallback for requests without sid. Setting lazy=True defers session initialization until the first request targets it.
• get(sid): Retrieves a session by ID, starting it if marked as lazy.
• fetch(request): The core routing method that resolves the effective session ID and delegates the HTTP call.

# scrapling/spiders/session.py

class SessionManager:
    def fetch(self, request):
        # Resolve session ID (fallback to default if None)

        sid = request.sid or self.default_session_id
        session = self.get(sid)
        
        # Delegate to the specific session implementation

        if hasattr(session, 'fetch'):
            return session.fetch(request)
        # ... additional handling for sync sessions

The Spider Configuration Hook

The base Spider class in scrapling/spiders/spider.py provides the configure_sessions hook. This is where you register multiple sessions for routing.

# scrapling/spiders/spider.py

class Spider:
    def configure_sessions(self, manager: SessionManager) -> None:
        # Default implementation adds a basic FetcherSession

        manager.add("default", FetcherSession(), default=True)
    
    async def start_requests(self):
        # Uses default session ID if sid not specified

        for url in self.start_urls:
            yield Request(url, sid=self._session_manager.default_session_id)

Implementing Multi-Session Routing

To route requests to different sessions by ID, you must register diverse session types in configure_sessions and then assign sid values when creating requests.

Registering Multiple Sessions

Override configure_sessions to register different fetcher configurations. The first session added becomes the default unless you explicitly mark another with default=True.

from scrapling.spiders.spider import Spider
from scrapling.spiders.request import Request
from scrapling.fetchers import FetcherSession, AsyncStealthySession
from scrapling.spiders.session import SessionManager

class MultiSessionSpider(Spider):
    name = "multi-session"
    start_urls = ["https://api.example.com/data", "https://protected-site.com"]

    def configure_sessions(self, manager: SessionManager) -> None:
        # Lightweight session for API calls (default)

        manager.add("api", FetcherSession(), default=True)
        
        # Stealth browser for protected sites (lazy initialization)

        manager.add("stealth", AsyncStealthySession(headless=True), lazy=True)

Directing Requests to Specific Sessions

In start_requests or callbacks, instantiate Request objects with the sid parameter to route them to specific sessions.

    async def start_requests(self):
        # Uses default "api" session

        yield Request("https://api.example.com/data", callback=self.parse_api)
        
        # Explicitly routes to stealth session

        yield Request(
            "https://protected-site.com", 
            callback=self.parse_protected,
            sid="stealth"
        )

Accessing Session Information in Callbacks

The response object retains a reference to the original request, allowing you to inspect which session handled the fetch.

    async def parse_api(self, response):
        # Verify routing worked as expected

        if response.request.sid == "api":
            self.logger.info("API response received via FetcherSession")
        
        yield {"source": "api", "data": response.json()}

    async def parse_protected(self, response):
        # Confirm stealth session handled this request

        self.logger.info(f"Protected site rendered via {response.request.sid}")
        yield {"source": "stealth", "title": response.css("h1::text").get()}

Key Source Files and Implementation Details

Understanding the routing mechanism requires familiarity with these specific files in the Scrapling repository:

File	Purpose	Key Components
scrapling/spiders/request.py	Defines the Request object and its sid attribute	Request.__init__ accepts sid parameter for session routing
scrapling/spiders/session.py	Implements SessionManager with registry and routing logic	add(), get(), fetch() methods handle session resolution and lazy initialization
scrapling/spiders/spider.py	Base spider class providing the configuration hook	configure_sessions() method for registering sessions; start_requests() uses default session ID
scrapling/fetchers/requests.py	FetcherSession implementation for standard HTTP	Used for lightweight API calls in multi-session setups
scrapling/fetchers/stealth_chrome.py	AsyncStealthySession for headless browser automation	Used for JavaScript rendering and anti-bot evasion

The routing logic in SessionManager.fetch() specifically checks request.sid and falls back to self.default_session_id when the attribute is None, ensuring that every request is associated with a valid session instance before the HTTP transaction occurs.

Summary

• Session routing in Scrapling relies on the sid parameter in Request objects to map requests to specific session instances managed by SessionManager.
• Registration occurs in Spider.configure_sessions() using manager.add(), where you can designate default sessions and enable lazy initialization to conserve resources.
• Lazy sessions defer browser startup or connection pool creation until the first request targeting that session ID is processed, improving efficiency in multi-session spiders.
• Fallback behavior automatically routes requests without an explicit sid to the default session configured in the manager, ensuring backward compatibility with simple spider implementations.

Frequently Asked Questions

What happens if I don't specify a session ID in my Request?

If you omit the sid parameter when creating a Request, Scrapling automatically routes the request to the default session configured in the SessionManager. According to the implementation in scrapling/spiders/session.py, the fetch method resolves the effective session ID using request.sid or self.default_session_id, ensuring every request is handled by a valid session instance.

Can I switch sessions dynamically based on response content?

Yes, you can route subsequent requests to different sessions dynamically within your callback methods. Since callbacks receive the Response object containing the original Request, you can inspect the content and yield new Request objects with explicit sid parameters targeting different sessions. For example, if you detect a CAPTCHA challenge in a response, you can yield a new request with sid="stealth" to route it through a headless browser session.

What is the difference between lazy and non-lazy sessions?

Lazy sessions defer initialization until the first request actually targets that session ID, while non-lazy sessions start immediately when the spider begins crawling. When you register a session with manager.add("stealth", AsyncStealthySession(), lazy=True), the browser process doesn't launch until Scrapling encounters a request with sid="stealth". This conserves memory and CPU resources when running spiders that may not use every registered session during a specific crawl.

How do I access the session object directly in a callback?

While the Response object provides access to the request's session ID via response.request.sid, direct access to the session instance itself is typically not necessary since the fetch operation has already completed. However, if you need to interact with the session for advanced scenarios (such as manually managing cookies or checking browser state), you can access the active session through the spider's _session_manager attribute using self._session_manager.get(response.request.sid). Note that this accesses internal APIs and should be used with caution.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}