# How Spider Creator Handles Dynamic Content and Infinite Scroll Pages

> Spider Creator records browser interactions like scrolls and JS executions to generate Scrapy spiders that replicate dynamic content and infinite scroll page behavior using Playwright or Selenium.

- Repository: [Carlos A. Planchón/spidercreator](https://github.com/carlosplanchon/spidercreator)
- Tags: how-to-guide
- Published: 2026-02-26

---

**Spider Creator handles dynamic content and infinite scroll pages by recording every browser interaction—including scrolls and JavaScript executions—then generating Scrapy spiders that replicate this behavior using Playwright or Selenium.**

Spider Creator is an open-source framework that automates web scraping by recording LLM-driven browser agent interactions. When you need to handle dynamic content and infinite scroll pages in Spider Creator, the system captures each scroll event as a discrete recording step, allowing the pipeline to detect newly loaded regions of interest and emit spiders that implement scroll-until-exhausted logic.

## Recording Browser Interactions for Dynamic Pages

The foundation of dynamic content handling lies in [`main/record_activity.py`](https://github.com/carlosplanchon/spidercreator/blob/main/main/record_activity.py), where the `Agent` class (a wrapper around `browser_use.Agent`) executes browser actions and captures the resulting HTML.

### Capturing Scroll Events and JavaScript Execution

Each time the agent performs an action—whether clicking, filling forms, or scrolling—the `record_activity()` function (lines 55-57) captures the updated page state. For infinite scroll scenarios, you can inject JavaScript directly through the browser context:

```python

# Inside the agent's execution loop in record_activity.py

while not reached_end:
    await agent.browser_context.evaluate(
        "window.scrollTo(0, document.body.scrollHeight)"
    )
    await asyncio.sleep(2)  # Allow time for AJAX content to load

    
    # Record the new state - this posts to the FastAPI endpoint

    await record_activity(agent)
    
    # Check if new content was loaded (optional)

    if len(agent.state.history.urls()) == previous_url_count:
        reached_end = True

```

This pattern ensures that every scroll iteration generates a discrete recording containing the freshly loaded HTML.

### Storing Step-by-Step HTML Snapshots

The [`main/receive_bu_data.py`](https://github.com/carlosplanchon/spidercreator/blob/main/main/receive_bu_data.py) module exposes a FastAPI endpoint at `/post_agent_history_step` that receives a JSON payload for every agent step. Each payload includes the `website_html` field, which is persisted as a sequential JSON file ([`1.json`](https://github.com/carlosplanchon/spidercreator/blob/main/1.json), [`2.json`](https://github.com/carlosplanchon/spidercreator/blob/main/2.json), etc.) in a task-specific folder. This storage mechanism provides the pipeline with a complete history of the page's dynamic evolution.

## Processing Dynamic Content in the Pipeline

Once recordings are stored, the pipeline processes them to identify regions of interest (ROI) that appear across multiple scroll states.

### Building DOM Representations from Recordings

The [`main/pipeline/make_dom_repr.py`](https://github.com/carlosplanchon/spidercreator/blob/main/main/pipeline/make_dom_repr.py) module provides the `make_dom_representation()` function, which constructs a `DomRepresentation` object from the cumulative HTML of the latest recording. Because this representation is built from the full HTML snapshot captured after each scroll, newly added nodes from infinite scroll loads are automatically included in the DOM tree.

The system utilizes the `betterhtmlchunking` library to segment the DOM into discrete regions, creating `tree_regions_system.sorted_roi_by_pos_xpath` which contains all detectable content blocks.

### Classifying Regions of Interest

In [`main/pipeline/roiclf_spcandmkr.py`](https://github.com/carlosplanchon/spidercreator/blob/main/main/pipeline/roiclf_spcandmkr.py), the `classify_roi_html_create_cand_spider()` function iterates over `dom_repr.tree_regions_system.sorted_roi_by_pos_xpath` to classify each region. As the DOM grows with each scroll iteration, new ROI objects appear automatically in this sorted list. The classifier evaluates each region against the planning JSON to determine if it contains target data, ensuring that content loaded via infinite scroll is treated identically to initially visible content.

## Generating Spiders with Infinite Scroll Logic

The final stage converts the recorded interactions into executable Scrapy code that replicates the scrolling behavior.

### Augmenting the Scrapy Creation Prompt

The `SCRAPY_CREATION_PROMPT` in [`main/pipeline/roiclf_spcandmkr.py`](https://github.com/carlosplanchon/spidercreator/blob/main/main/pipeline/roiclf_spcandmkr.py) serves as the template for LLM-driven spider generation. To handle infinite scroll, you can enrich this prompt with explicit scrolling instructions:

```python
SCRAPY_CREATION_PROMPT = """
Write a Scrapy spider that:

1. Loads the start URL using Playwright.
2. Repeatedly scrolls to the bottom of the page until no new items appear,
   detecting exhaustion by comparing item counts between scrolls.
3. Extracts the fields defined in {extracted_content_on_rec}.
4. Yields each item as a Scrapy Item.
"""

```

When `classify_roi_html_create_cand_spider()` invokes the LLM with this augmented prompt, the resulting spider code includes the scroll-until-exhausted logic.

### Implementing Scroll-Until-Exhausted in Generated Spiders

The generated spider, typically output to [`spider_code.py`](https://github.com/carlosplanchon/spidercreator/blob/main/spider_code.py) via [`main/exec_funcs.py`](https://github.com/carlosplanchon/spidercreator/blob/main/main/exec_funcs.py), utilizes Scrapy-Playwright to execute JavaScript scrolling. An example implementation that the LLM might generate based on the prompt above:

```python
import scrapy
from scrapy_playwright.page import PageMethod

class InfiniteScrollSpider(scrapy.Spider):
    name = "infinite_scroll"
    start_urls = ["https://example.com/catalog"]
    
    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(
                url,
                meta={
                    "playwright": True,
                    "playwright_page_methods": [
                        PageMethod("wait_for_selector", ".product-card")
                    ]
                }
            )
    
    async def parse(self, response):
        page = response.meta["playwright_page"]
        previous_count = 0
        
        while True:
            # Scroll to bottom

            await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
            await page.wait_for_timeout(2000)  # Wait for AJAX

            
            # Count current items

            items = page.locator(".product-card")
            current_count = await items.count()
            
            if current_count == previous_count:
                break  # No new content loaded

            
            previous_count = current_count
        
        # Extract all items after scrolling complete

        for i in range(previous_count):
            item = items.nth(i)
            yield {
                "title": await item.locator(".title").text_content(),
                "price": await item.locator(".price").text_content()
            }

```

This pattern ensures the spider exhausts the infinite scroll before extracting data, matching the behavior recorded during the agent phase.

## Key Files for Dynamic Content Handling

Understanding the repository structure helps you locate the specific components responsible for infinite scroll support:

- **[`main/record_activity.py`](https://github.com/carlosplanchon/spidercreator/blob/main/main/record_activity.py)** – Orchestrates the LLM Agent and captures HTML/screenshots after every browser action, including scroll events.
- **[`main/receive_bu_data.py`](https://github.com/carlosplanchon/spidercreator/blob/main/main/receive_bu_data.py)** – FastAPI endpoint that receives step-by-step JSON payloads containing `website_html` and persists them as sequential recordings.
- **[`main/utils/recordings.py`](https://github.com/carlosplanchon/spidercreator/blob/main/main/utils/recordings.py)** – Provides `load_recordings()` to retrieve the stored step sequences for downstream processing.
- **[`main/pipeline/make_dom_repr.py`](https://github.com/carlosplanchon/spidercreator/blob/main/main/pipeline/make_dom_repr.py)** – Builds `DomRepresentation` objects from the latest HTML snapshots, automatically including dynamically loaded nodes.
- **[`main/pipeline/roiclf_spcandmkr.py`](https://github.com/carlosplanchon/spidercreator/blob/main/main/pipeline/roiclf_spcandmkr.py)** – Contains `classify_roi_html_create_cand_spider()` and `SCRAPY_CREATION_PROMPT` for generating scroll-aware spider code.
- **[`main/pipeline/spider_draft.py`](https://github.com/carlosplanchon/spidercreator/blob/main/main/pipeline/spider_draft.py)** – Generates initial spider drafts from the full recording list before ROI classification.
- **[`main/exec_funcs.py`](https://github.com/carlosplanchon/spidercreator/blob/main/main/exec_funcs.py)** – High-level `create_spider()` function (line 48) that orchestrates the entire pipeline and outputs [`spider_code.py`](https://github.com/carlosplanchon/spidercreator/blob/main/spider_code.py).

## Summary

Spider Creator handles dynamic content and infinite scroll pages through a systematic recording and generation pipeline:

- **Step-wise recording** captures HTML after every scroll or JavaScript execution, ensuring dynamically loaded content is preserved in [`record_activity.py`](https://github.com/carlosplanchon/spidercreator/blob/main/record_activity.py).
- **Sequential storage** via [`receive_bu_data.py`](https://github.com/carlosplanchon/spidercreator/blob/main/receive_bu_data.py) creates discrete JSON snapshots ([`1.json`](https://github.com/carlosplanchon/spidercreator/blob/main/1.json), [`2.json`](https://github.com/carlosplanchon/spidercreator/blob/main/2.json), etc.) for each page state.
- **DOM representation** in [`make_dom_repr.py`](https://github.com/carlosplanchon/spidercreator/blob/main/make_dom_repr.py) automatically includes new nodes from infinite scroll loads when building the `DomRepresentation`.
- **ROI classification** through [`roiclf_spcandmkr.py`](https://github.com/carlosplanchon/spidercreator/blob/main/roiclf_spcandmkr.py) treats dynamically loaded content identically to static content, classifying regions based on the full HTML snapshot.
- **Scroll-aware generation** via augmented `SCRAPY_CREATION_PROMPT` produces spiders that implement scroll-until-exhausted logic using Scrapy-Playwright.

## Frequently Asked Questions

### Does Spider Creator require manual scrolling during the recording phase?

No, manual scrolling is not required. The LLM-driven `Agent` in [`main/record_activity.py`](https://github.com/carlosplanchon/spidercreator/blob/main/main/record_activity.py) can autonomously execute scroll actions using `agent.browser_context.evaluate()` to run JavaScript like `window.scrollTo(0, document.body.scrollHeight)`. Each scroll triggers `record_activity()`, which posts the updated HTML to the FastAPI endpoint, capturing the dynamically loaded content automatically.

### Can Spider Creator handle JavaScript-heavy single-page applications (SPAs)?

Yes, Spider Creator is designed for JavaScript-heavy applications. Because the recording phase uses a real browser agent (via `browser_use`), it executes all JavaScript and AJAX calls naturally. The [`main/receive_bu_data.py`](https://github.com/carlosplanchon/spidercreator/blob/main/main/receive_bu_data.py) endpoint captures the rendered HTML after each interaction, and the [`main/pipeline/make_dom_repr.py`](https://github.com/carlosplanchon/spidercreator/blob/main/main/pipeline/make_dom_repr.py) module processes the full DOM including dynamically injected elements typical of SPAs.

### How does the generated spider know when to stop scrolling?

The generated spider implements a scroll-until-exhausted algorithm based on the `SCRAPY_CREATION_PROMPT` template in [`main/pipeline/roiclf_spcandmkr.py`](https://github.com/carlosplanchon/spidercreator/blob/main/main/pipeline/roiclf_spcandmkr.py). Typically, the spider counts the number of items (e.g., product cards) before and after each scroll using Playwright's `page.locator().count()`. If the count remains unchanged after scrolling and waiting for network activity, the loop breaks, indicating all content has loaded.

### Is Playwright the only browser automation option for the generated spiders?

While the examples and default prompts in Spider Creator emphasize Scrapy-Playwright for handling infinite scroll and JavaScript execution, the architecture is not strictly limited to Playwright. The `SCRAPY_CREATION_PROMPT` can be modified to generate Selenium-based callbacks or even bare Scrapy with Splash requests. However, Playwright is the recommended and most tested approach for handling complex dynamic content in the current implementation.