How to Implement Multi-Page Scraping in SpiderCreator: Listings + Details Guide

SpiderCreator generates Scrapy spiders capable of multi-page scraping by analyzing browser recordings of pagination and detail-page navigation, then producing candidate spiders with separate callbacks for listings and detail pages that pass partial items via the meta parameter.

SpiderCreator is an open-source framework that transforms natural-language task descriptions into functional Scrapy spiders. When your data extraction spans both listing pages and individual detail pages, the framework automatically generates the necessary pagination logic and cross-page data passing mechanisms required for cohesive multi-page scraping.

How SpiderCreator Generates Multi-Page Spiders

The framework follows a structured pipeline to convert browser interactions into executable spider code. According to the spidercreator source code, the core flow involves:

1. Session Recording: Browser sessions are captured and stored (utils/recordings.py).
2. Recording Interpretation: The system analyzes recordings (planning/rec_filtering.py) and constructs a mind-map of the site structure (pipeline/mindmap.py).
3. Draft Generation: A draft Scrapy spider is requested from the LLM (pipeline/spider_draft.py).
4. Candidate Creation: The draft is transformed into candidate spiders targeting individual "regions of interest" (ROIs), such as pagination links or detail pages (pipeline/roiclf_spcandmkr.py).
5. Execution and Verification: Each candidate spider is executed (ctxexec/exec_sp.py) and the best performer is selected after verification (pipeline/verification_pipeline.py).

For multi-page scenarios involving listings and details, the ROI-based classification in step 4 is critical. When recordings show clicks on "Next" buttons or "View details" links, the classifier identifies these patterns and generates candidate spiders with separate callbacks for pagination and detail extraction.

The Architecture of Listings + Details Scraping

Effective multi-page scraping requires handling two distinct navigation patterns: paginating through listing pages and drilling down into individual detail pages.

Handling Pagination Across Listing Pages

The generated spider typically handles pagination by extracting the "next page" link and yielding a new request to the same parsing callback. In pipeline/spider_draft.py, the LLM analyzes recordings of repeated "Next" clicks and often generates a while loop or LinkExtractor rule to follow pagination chains.

If the generated code lacks pagination logic, you can manually add a rule that extracts the next page link:

next_page = response.css("a.next::attr(href)").get()
if next_page:
    yield response.follow(next_page, callback=self.parse)

Extracting Data from Detail Pages

Detail page extraction requires passing partially collected data from the listing callback to the detail callback. The ROI classifier (pipeline/roiclf_spcandmkr.py) creates a detail-page candidate spider when recordings show clicks on listing items.

The standard pattern involves passing the incomplete Item via the meta parameter:

# In the listing callback

item = {
    "title": product.css("h2::text").get(),
    "price": product.css("span.price::text").get(),
}
detail_url = product.css("a::attr(href)").get()
yield response.follow(
    detail_url,
    callback=self.parse_detail,
    meta={"item": item},  # Pass the incomplete item forward

)

In parse_detail, retrieve the item via response.meta["item"], enrich it with detail-specific fields, and yield the completed item.

Implementing Multi-Page Scraping Step-by-Step

To generate a spider that handles both listings and details, you must explicitly describe the navigation flow in your task description and then verify the generated callbacks.

Step 1: Describe the Task with Pagination Instructions

The LLM generates code based on the browser_use_task description. To trigger multi-page logic, include explicit instructions about pagination and detail navigation:

MULTI_PAGE_TASK = """
Navigate to https://tiendainglesa.com.uy/ homepage.
Extract all products on the homepage with visible attributes.
While a "Next" button exists, click it to load the next listings page.
For each product click the "View details" link and extract the full description, SKU and all images.
Stop after you have collected data for at least 10 products.
"""

Running create_spider(browser_use_task=MULTI_PAGE_TASK) from main.py initiates the pipeline. The recordings of "Next" clicks and "View details" clicks inform the ROI classifier to generate candidate spiders with separate pagination and detail callbacks.

Step 2: Review the Generated Spider Structure

After execution, the final spider is written to results/<task_id>/spider_code.py. Inspect this file for two critical components:

1. Pagination logic: Look for response.follow calls that reference next-page selectors or LinkExtractor rules.
2. Detail callbacks: Verify that the listing parser extracts detail URLs and yields requests with meta={"item": item}.

The generated structure typically resembles:

import scrapy

class ListingSpider(scrapy.Spider):
    name = "listing_spider"
    start_urls = ["https://example.com/products"]

    def parse(self, response):
        for product in response.css("div.product"):
            item = {
                "title": product.css("h2::text").get(),
                "price": product.css("span.price::text").get(),
                "list_url": response.url,
            }
            detail_url = product.css("a::attr(href)").get()
            if detail_url:
                yield response.follow(
                    detail_url,
                    callback=self.parse_detail,
                    meta={"item": item},
                )
        
        next_page = response.css("a.next::attr(href)").get()
        if next_page:
            yield response.follow(next_page, callback=self.parse)

    def parse_detail(self, response):
        item = response.meta["item"]
        item.update({
            "description": response.css("div.description::text").get(),
            "sku": response.css("span.sku::text").get(),
            "images": response.css("img.gallery::attr(src)").getall(),
        })
        yield item

Step 3: Pass Items Between Callbacks Using Meta

The critical mechanism for multi-page scraping is the meta parameter. When the listing callback yields a request to the detail page, it must pass the partially filled item:

yield response.follow(
    detail_url,
    callback=self.parse_detail,
    meta={"item": item},  # Pass the partial item forward

)

In parse_detail, retrieve the item via response.meta["item"], add detail-specific fields (like SKU or full description), and yield the completed item. This pattern ensures data integrity across the two-page extraction flow.

Alternative Approach: Using CrawlSpider for Link Following

For sites with consistent pagination and detail link patterns, you can guide the LLM to generate a CrawlSpider instead of a basic Spider. This approach uses declarative Rule objects rather than manual callback chaining.

from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor

class ListingCrawlSpider(CrawlSpider):
    name = "listing_crawl"
    start_urls = ["https://example.com/products"]

    rules = [
        Rule(LinkExtractor(restrict_css="a.next"), follow=True),  # Pagination

        Rule(
            LinkExtractor(restrict_css="a.detail"),
            callback="parse_detail",
        ),
    ]

    def parse_detail(self, response):
        yield {
            "title": response.css("h1::text").get(),
            "price": response.css("span.price::text").get(),
            "description": response.css("div.desc::text").get(),
            "images": response.css("img::attr(src)").getall(),
        }

To trigger this pattern, include explicit instructions in your browser_use_task such as "use Scrapy CrawlSpider rules" or "use LinkExtractor for pagination."

Key Files in the Multi-Page Pipeline

Understanding the source code helps you debug and extend generated spiders. Here are the critical files involved in multi-page spider generation:

File	Role in Multi-Page Creation	Location
pipeline/spider_draft.py	Sends recordings and mind-maps to the LLM to generate the initial Scrapy draft with pagination logic.	[pipeline/spider_draft.py](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/spider_draft.py)
pipeline/roiclf_spcandmkr.py	Analyzes Regions of Interest (ROIs) like "Next" buttons and detail links to create candidate spiders with separate callbacks.	[pipeline/roiclf_spcandmkr.py](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/roiclf_spcandmkr.py)
ctxexec/exec_sp.py	Executes candidate spiders to verify which implementation correctly handles pagination and detail page extraction.	[ctxexec/exec_sp.py](https://github.com/carlosplanchon/spidercreator/blob/main/ctxexec/exec_sp.py)
main.py	Entry point that orchestrates the pipeline; call create_spider() with your multi-page task description.	[main.py](https://github.com/carlosplanchon/spidercreator/blob/main/main.py)
results/<task_id>/spider_code.py	The final generated spider file where you manually fine-tune pagination selectors or meta passing.	(generated at runtime)

Summary

• SpiderCreator automates multi-page scraping by analyzing browser recordings of pagination and detail navigation, then generating candidate spiders through the pipeline in main.py.
• The framework creates separate callbacks for listing pages and detail pages, using Scrapy's meta parameter to pass partially filled items between them.
• To trigger automatic generation of multi-page logic, describe both pagination ("Next" button) and detail extraction ("View details") actions in your browser_use_task string.
• If the generated spider requires adjustment, edit results/<task_id>/spider_code.py to add manual pagination loops or refine CSS selectors for detail links.
• For sites with consistent link patterns, you can guide the LLM to generate a CrawlSpider with declarative Rule objects instead of manual callback chaining.

Frequently Asked Questions

How does SpiderCreator detect when to generate pagination logic?

SpiderCreator detects pagination through the ROI classifier in pipeline/roiclf_spcandmkr.py. When your browser recordings show repeated clicks on a "Next" button or page numbers, the classifier identifies these as Regions of Interest and generates candidate spiders that include pagination loops or LinkExtractor rules. Explicitly stating "click the Next button until it disappears" in your task description ensures the recordings capture this behavior for the classifier to analyze.

What is the correct way to pass data between the listing and detail callbacks?

Use Scrapy's meta parameter to pass the partially extracted item from the listing callback to the detail callback. In the listing parser, yield a request with meta={"item": item} where item contains the fields extracted from the listing page. In the detail callback, retrieve the item using item = response.meta["item"], then add the detail-specific fields and yield the completed item. This pattern ensures data integrity across the multi-page extraction flow.

Can I modify the generated spider if it misses the detail page extraction?

Yes, you can manually edit the generated spider located at results/<task_id>/spider_code.py. If the spider lacks detail page handling, add a callback method (e.g., parse_detail) and modify the listing parser to extract detail URLs and yield requests with the appropriate callback and meta data. If pagination is missing, add a selector for the next page link and yield a follow request back to the parse method.

Is CrawlSpider better than a basic Spider for multi-page scraping?

CrawlSpider is preferable when the site uses consistent, predictable URL patterns for pagination and detail links, as it allows declarative rules via LinkExtractor rather than manual callback chaining. However, for complex sites where you need fine-grained control over which links to follow or when to stop, a basic Spider with custom parse methods and explicit response.follow calls (as generated by default) provides more flexibility. You can guide SpiderCreator toward either approach by specifying "use CrawlSpider rules" or "manual pagination" in your task description.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}