# How to Implement Multi-Page Scraping in SpiderCreator: Listings + Details Guide

> Learn multi-page scraping in SpiderCreator. Automate listing and detail page extraction with generated Scrapy spiders. Analyze browser recordings for seamless data collection.

- Repository: [Carlos A. Planchón/spidercreator](https://github.com/carlosplanchon/spidercreator)
- Tags: how-to-guide
- Published: 2026-02-26

---

**SpiderCreator generates Scrapy spiders capable of multi-page scraping by analyzing browser recordings of pagination and detail-page navigation, then producing candidate spiders with separate callbacks for listings and detail pages that pass partial items via the `meta` parameter.**

SpiderCreator is an open-source framework that transforms natural-language task descriptions into functional Scrapy spiders. When your data extraction spans both listing pages and individual detail pages, the framework automatically generates the necessary pagination logic and cross-page data passing mechanisms required for cohesive multi-page scraping.

## How SpiderCreator Generates Multi-Page Spiders

The framework follows a structured pipeline to convert browser interactions into executable spider code. According to the `spidercreator` source code, the core flow involves:

1. **Session Recording**: Browser sessions are captured and stored ([`utils/recordings.py`](https://github.com/carlosplanchon/spidercreator/blob/main/utils/recordings.py)).
2. **Recording Interpretation**: The system analyzes recordings ([`planning/rec_filtering.py`](https://github.com/carlosplanchon/spidercreator/blob/main/planning/rec_filtering.py)) and constructs a **mind-map** of the site structure ([`pipeline/mindmap.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/mindmap.py)).
3. **Draft Generation**: A draft Scrapy spider is requested from the LLM ([`pipeline/spider_draft.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/spider_draft.py)).
4. **Candidate Creation**: The draft is transformed into **candidate spiders** targeting individual "regions of interest" (ROIs), such as pagination links or detail pages ([`pipeline/roiclf_spcandmkr.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/roiclf_spcandmkr.py)).
5. **Execution and Verification**: Each candidate spider is executed ([`ctxexec/exec_sp.py`](https://github.com/carlosplanchon/spidercreator/blob/main/ctxexec/exec_sp.py)) and the best performer is selected after verification ([`pipeline/verification_pipeline.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/verification_pipeline.py)).

For multi-page scenarios involving listings and details, the **ROI-based classification** in step 4 is critical. When recordings show clicks on "Next" buttons or "View details" links, the classifier identifies these patterns and generates candidate spiders with separate callbacks for pagination and detail extraction.

## The Architecture of Listings + Details Scraping

Effective multi-page scraping requires handling two distinct navigation patterns: paginating through listing pages and drilling down into individual detail pages.

### Handling Pagination Across Listing Pages

The generated spider typically handles pagination by extracting the "next page" link and yielding a new request to the same parsing callback. In [`pipeline/spider_draft.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/spider_draft.py), the LLM analyzes recordings of repeated "Next" clicks and often generates a `while` loop or `LinkExtractor` rule to follow pagination chains.

If the generated code lacks pagination logic, you can manually add a rule that extracts the next page link:

```python
next_page = response.css("a.next::attr(href)").get()
if next_page:
    yield response.follow(next_page, callback=self.parse)

```

### Extracting Data from Detail Pages

Detail page extraction requires passing partially collected data from the listing callback to the detail callback. The **ROI classifier** ([`pipeline/roiclf_spcandmkr.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/roiclf_spcandmkr.py)) creates a *detail-page* candidate spider when recordings show clicks on listing items.

The standard pattern involves passing the incomplete `Item` via the **`meta`** parameter:

```python

# In the listing callback

item = {
    "title": product.css("h2::text").get(),
    "price": product.css("span.price::text").get(),
}
detail_url = product.css("a::attr(href)").get()
yield response.follow(
    detail_url,
    callback=self.parse_detail,
    meta={"item": item},  # Pass the incomplete item forward

)

```

In `parse_detail`, retrieve the item via `response.meta["item"]`, enrich it with detail-specific fields, and yield the completed item.

## Implementing Multi-Page Scraping Step-by-Step

To generate a spider that handles both listings and details, you must explicitly describe the navigation flow in your task description and then verify the generated callbacks.

### Step 1: Describe the Task with Pagination Instructions

The LLM generates code based on the **browser_use_task** description. To trigger multi-page logic, include explicit instructions about pagination and detail navigation:

```python
MULTI_PAGE_TASK = """
Navigate to https://tiendainglesa.com.uy/ homepage.
Extract all products on the homepage with visible attributes.
While a "Next" button exists, click it to load the next listings page.
For each product click the "View details" link and extract the full description, SKU and all images.
Stop after you have collected data for at least 10 products.
"""

```

Running `create_spider(browser_use_task=MULTI_PAGE_TASK)` from [`main.py`](https://github.com/carlosplanchon/spidercreator/blob/main/main.py) initiates the pipeline. The recordings of "Next" clicks and "View details" clicks inform the ROI classifier to generate candidate spiders with separate pagination and detail callbacks.

### Step 2: Review the Generated Spider Structure

After execution, the final spider is written to `results/<task_id>/spider_code.py`. Inspect this file for two critical components:

1. **Pagination logic**: Look for `response.follow` calls that reference next-page selectors or `LinkExtractor` rules.
2. **Detail callbacks**: Verify that the listing parser extracts detail URLs and yields requests with `meta={"item": item}`.

The generated structure typically resembles:

```python
import scrapy

class ListingSpider(scrapy.Spider):
    name = "listing_spider"
    start_urls = ["https://example.com/products"]

    def parse(self, response):
        for product in response.css("div.product"):
            item = {
                "title": product.css("h2::text").get(),
                "price": product.css("span.price::text").get(),
                "list_url": response.url,
            }
            detail_url = product.css("a::attr(href)").get()
            if detail_url:
                yield response.follow(
                    detail_url,
                    callback=self.parse_detail,
                    meta={"item": item},
                )
        
        next_page = response.css("a.next::attr(href)").get()
        if next_page:
            yield response.follow(next_page, callback=self.parse)

    def parse_detail(self, response):
        item = response.meta["item"]
        item.update({
            "description": response.css("div.description::text").get(),
            "sku": response.css("span.sku::text").get(),
            "images": response.css("img.gallery::attr(src)").getall(),
        })
        yield item

```

### Step 3: Pass Items Between Callbacks Using Meta

The critical mechanism for multi-page scraping is the **`meta`** parameter. When the listing callback yields a request to the detail page, it must pass the partially filled item:

```python
yield response.follow(
    detail_url,
    callback=self.parse_detail,
    meta={"item": item},  # Pass the partial item forward

)

```

In `parse_detail`, retrieve the item via `response.meta["item"]`, add detail-specific fields (like SKU or full description), and yield the completed item. This pattern ensures data integrity across the two-page extraction flow.

## Alternative Approach: Using CrawlSpider for Link Following

For sites with consistent pagination and detail link patterns, you can guide the LLM to generate a **CrawlSpider** instead of a basic Spider. This approach uses declarative `Rule` objects rather than manual callback chaining.

```python
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor

class ListingCrawlSpider(CrawlSpider):
    name = "listing_crawl"
    start_urls = ["https://example.com/products"]

    rules = [
        Rule(LinkExtractor(restrict_css="a.next"), follow=True),  # Pagination

        Rule(
            LinkExtractor(restrict_css="a.detail"),
            callback="parse_detail",
        ),
    ]

    def parse_detail(self, response):
        yield {
            "title": response.css("h1::text").get(),
            "price": response.css("span.price::text").get(),
            "description": response.css("div.desc::text").get(),
            "images": response.css("img::attr(src)").getall(),
        }

```

To trigger this pattern, include explicit instructions in your **browser_use_task** such as "use Scrapy CrawlSpider rules" or "use LinkExtractor for pagination."

## Key Files in the Multi-Page Pipeline

Understanding the source code helps you debug and extend generated spiders. Here are the critical files involved in multi-page spider generation:

| File | Role in Multi-Page Creation | Location |
|------|----------------------------|----------|
| [`pipeline/spider_draft.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/spider_draft.py) | Sends recordings and mind-maps to the LLM to generate the initial Scrapy draft with pagination logic. | [[`pipeline/spider_draft.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/spider_draft.py)](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/spider_draft.py) |
| [`pipeline/roiclf_spcandmkr.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/roiclf_spcandmkr.py) | Analyzes Regions of Interest (ROIs) like "Next" buttons and detail links to create candidate spiders with separate callbacks. | [[`pipeline/roiclf_spcandmkr.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/roiclf_spcandmkr.py)](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/roiclf_spcandmkr.py) |
| [`ctxexec/exec_sp.py`](https://github.com/carlosplanchon/spidercreator/blob/main/ctxexec/exec_sp.py) | Executes candidate spiders to verify which implementation correctly handles pagination and detail page extraction. | [[`ctxexec/exec_sp.py`](https://github.com/carlosplanchon/spidercreator/blob/main/ctxexec/exec_sp.py)](https://github.com/carlosplanchon/spidercreator/blob/main/ctxexec/exec_sp.py) |
| [`main.py`](https://github.com/carlosplanchon/spidercreator/blob/main/main.py) | Entry point that orchestrates the pipeline; call `create_spider()` with your multi-page task description. | [[`main.py`](https://github.com/carlosplanchon/spidercreator/blob/main/main.py)](https://github.com/carlosplanchon/spidercreator/blob/main/main.py) |
| `results/<task_id>/spider_code.py` | The final generated spider file where you manually fine-tune pagination selectors or meta passing. | (generated at runtime) |

## Summary

- **SpiderCreator** automates multi-page scraping by analyzing browser recordings of pagination and detail navigation, then generating candidate spiders through the pipeline in [`main.py`](https://github.com/carlosplanchon/spidercreator/blob/main/main.py).
- The framework creates separate callbacks for **listing pages** and **detail pages**, using Scrapy's `meta` parameter to pass partially filled items between them.
- To trigger automatic generation of multi-page logic, describe both pagination ("Next" button) and detail extraction ("View details") actions in your **browser_use_task** string.
- If the generated spider requires adjustment, edit `results/<task_id>/spider_code.py` to add manual pagination loops or refine CSS selectors for detail links.
- For sites with consistent link patterns, you can guide the LLM to generate a **CrawlSpider** with declarative `Rule` objects instead of manual callback chaining.

## Frequently Asked Questions

### How does SpiderCreator detect when to generate pagination logic?

SpiderCreator detects pagination through the **ROI classifier** in [`pipeline/roiclf_spcandmkr.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/roiclf_spcandmkr.py). When your browser recordings show repeated clicks on a "Next" button or page numbers, the classifier identifies these as Regions of Interest and generates candidate spiders that include pagination loops or `LinkExtractor` rules. Explicitly stating "click the Next button until it disappears" in your task description ensures the recordings capture this behavior for the classifier to analyze.

### What is the correct way to pass data between the listing and detail callbacks?

Use Scrapy's **`meta`** parameter to pass the partially extracted item from the listing callback to the detail callback. In the listing parser, yield a request with `meta={"item": item}` where `item` contains the fields extracted from the listing page. In the detail callback, retrieve the item using `item = response.meta["item"]`, then add the detail-specific fields and yield the completed item. This pattern ensures data integrity across the multi-page extraction flow.

### Can I modify the generated spider if it misses the detail page extraction?

Yes, you can manually edit the generated spider located at `results/<task_id>/spider_code.py`. If the spider lacks detail page handling, add a callback method (e.g., `parse_detail`) and modify the listing parser to extract detail URLs and yield requests with the appropriate callback and meta data. If pagination is missing, add a selector for the next page link and yield a follow request back to the `parse` method.

### Is CrawlSpider better than a basic Spider for multi-page scraping?

**CrawlSpider** is preferable when the site uses consistent, predictable URL patterns for pagination and detail links, as it allows declarative rules via `LinkExtractor` rather than manual callback chaining. However, for complex sites where you need fine-grained control over which links to follow or when to stop, a basic **Spider** with custom `parse` methods and explicit `response.follow` calls (as generated by default) provides more flexibility. You can guide SpiderCreator toward either approach by specifying "use CrawlSpider rules" or "manual pagination" in your task description.