How Spider Creator Handles Dynamic Content and Infinite Scroll Pages

Spider Creator handles dynamic content and infinite scroll pages by recording every browser interaction—including scrolls and JavaScript executions—then generating Scrapy spiders that replicate this behavior using Playwright or Selenium.

Spider Creator is an open-source framework that automates web scraping by recording LLM-driven browser agent interactions. When you need to handle dynamic content and infinite scroll pages in Spider Creator, the system captures each scroll event as a discrete recording step, allowing the pipeline to detect newly loaded regions of interest and emit spiders that implement scroll-until-exhausted logic.

Recording Browser Interactions for Dynamic Pages

The foundation of dynamic content handling lies in main/record_activity.py, where the Agent class (a wrapper around browser_use.Agent) executes browser actions and captures the resulting HTML.

Capturing Scroll Events and JavaScript Execution

Each time the agent performs an action—whether clicking, filling forms, or scrolling—the record_activity() function (lines 55-57) captures the updated page state. For infinite scroll scenarios, you can inject JavaScript directly through the browser context:

# Inside the agent's execution loop in record_activity.py

while not reached_end:
    await agent.browser_context.evaluate(
        "window.scrollTo(0, document.body.scrollHeight)"
    )
    await asyncio.sleep(2)  # Allow time for AJAX content to load

    
    # Record the new state - this posts to the FastAPI endpoint

    await record_activity(agent)
    
    # Check if new content was loaded (optional)

    if len(agent.state.history.urls()) == previous_url_count:
        reached_end = True

This pattern ensures that every scroll iteration generates a discrete recording containing the freshly loaded HTML.

Storing Step-by-Step HTML Snapshots

The main/receive_bu_data.py module exposes a FastAPI endpoint at /post_agent_history_step that receives a JSON payload for every agent step. Each payload includes the website_html field, which is persisted as a sequential JSON file (1.json, 2.json, etc.) in a task-specific folder. This storage mechanism provides the pipeline with a complete history of the page's dynamic evolution.

Processing Dynamic Content in the Pipeline

Once recordings are stored, the pipeline processes them to identify regions of interest (ROI) that appear across multiple scroll states.

Building DOM Representations from Recordings

The main/pipeline/make_dom_repr.py module provides the make_dom_representation() function, which constructs a DomRepresentation object from the cumulative HTML of the latest recording. Because this representation is built from the full HTML snapshot captured after each scroll, newly added nodes from infinite scroll loads are automatically included in the DOM tree.

The system utilizes the betterhtmlchunking library to segment the DOM into discrete regions, creating tree_regions_system.sorted_roi_by_pos_xpath which contains all detectable content blocks.

Classifying Regions of Interest

In main/pipeline/roiclf_spcandmkr.py, the classify_roi_html_create_cand_spider() function iterates over dom_repr.tree_regions_system.sorted_roi_by_pos_xpath to classify each region. As the DOM grows with each scroll iteration, new ROI objects appear automatically in this sorted list. The classifier evaluates each region against the planning JSON to determine if it contains target data, ensuring that content loaded via infinite scroll is treated identically to initially visible content.

Generating Spiders with Infinite Scroll Logic

The final stage converts the recorded interactions into executable Scrapy code that replicates the scrolling behavior.

Augmenting the Scrapy Creation Prompt

The SCRAPY_CREATION_PROMPT in main/pipeline/roiclf_spcandmkr.py serves as the template for LLM-driven spider generation. To handle infinite scroll, you can enrich this prompt with explicit scrolling instructions:

SCRAPY_CREATION_PROMPT = """
Write a Scrapy spider that:

1. Loads the start URL using Playwright.
2. Repeatedly scrolls to the bottom of the page until no new items appear,
   detecting exhaustion by comparing item counts between scrolls.
3. Extracts the fields defined in {extracted_content_on_rec}.
4. Yields each item as a Scrapy Item.
"""

When classify_roi_html_create_cand_spider() invokes the LLM with this augmented prompt, the resulting spider code includes the scroll-until-exhausted logic.

Implementing Scroll-Until-Exhausted in Generated Spiders

The generated spider, typically output to spider_code.py via main/exec_funcs.py, utilizes Scrapy-Playwright to execute JavaScript scrolling. An example implementation that the LLM might generate based on the prompt above:

import scrapy
from scrapy_playwright.page import PageMethod

class InfiniteScrollSpider(scrapy.Spider):
    name = "infinite_scroll"
    start_urls = ["https://example.com/catalog"]
    
    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(
                url,
                meta={
                    "playwright": True,
                    "playwright_page_methods": [
                        PageMethod("wait_for_selector", ".product-card")
                    ]
                }
            )
    
    async def parse(self, response):
        page = response.meta["playwright_page"]
        previous_count = 0
        
        while True:
            # Scroll to bottom

            await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
            await page.wait_for_timeout(2000)  # Wait for AJAX

            
            # Count current items

            items = page.locator(".product-card")
            current_count = await items.count()
            
            if current_count == previous_count:
                break  # No new content loaded

            
            previous_count = current_count
        
        # Extract all items after scrolling complete

        for i in range(previous_count):
            item = items.nth(i)
            yield {
                "title": await item.locator(".title").text_content(),
                "price": await item.locator(".price").text_content()
            }

This pattern ensures the spider exhausts the infinite scroll before extracting data, matching the behavior recorded during the agent phase.

Key Files for Dynamic Content Handling

Understanding the repository structure helps you locate the specific components responsible for infinite scroll support:

• main/record_activity.py – Orchestrates the LLM Agent and captures HTML/screenshots after every browser action, including scroll events.
• main/receive_bu_data.py – FastAPI endpoint that receives step-by-step JSON payloads containing website_html and persists them as sequential recordings.
• main/utils/recordings.py – Provides load_recordings() to retrieve the stored step sequences for downstream processing.
• main/pipeline/make_dom_repr.py – Builds DomRepresentation objects from the latest HTML snapshots, automatically including dynamically loaded nodes.
• main/pipeline/roiclf_spcandmkr.py – Contains classify_roi_html_create_cand_spider() and SCRAPY_CREATION_PROMPT for generating scroll-aware spider code.
• main/pipeline/spider_draft.py – Generates initial spider drafts from the full recording list before ROI classification.
• main/exec_funcs.py – High-level create_spider() function (line 48) that orchestrates the entire pipeline and outputs spider_code.py.

Summary

Spider Creator handles dynamic content and infinite scroll pages through a systematic recording and generation pipeline:

• Step-wise recording captures HTML after every scroll or JavaScript execution, ensuring dynamically loaded content is preserved in record_activity.py.
• Sequential storage via receive_bu_data.py creates discrete JSON snapshots (1.json, 2.json, etc.) for each page state.
• DOM representation in make_dom_repr.py automatically includes new nodes from infinite scroll loads when building the DomRepresentation.
• ROI classification through roiclf_spcandmkr.py treats dynamically loaded content identically to static content, classifying regions based on the full HTML snapshot.
• Scroll-aware generation via augmented SCRAPY_CREATION_PROMPT produces spiders that implement scroll-until-exhausted logic using Scrapy-Playwright.

Frequently Asked Questions

Does Spider Creator require manual scrolling during the recording phase?

No, manual scrolling is not required. The LLM-driven Agent in main/record_activity.py can autonomously execute scroll actions using agent.browser_context.evaluate() to run JavaScript like window.scrollTo(0, document.body.scrollHeight). Each scroll triggers record_activity(), which posts the updated HTML to the FastAPI endpoint, capturing the dynamically loaded content automatically.

Can Spider Creator handle JavaScript-heavy single-page applications (SPAs)?

Yes, Spider Creator is designed for JavaScript-heavy applications. Because the recording phase uses a real browser agent (via browser_use), it executes all JavaScript and AJAX calls naturally. The main/receive_bu_data.py endpoint captures the rendered HTML after each interaction, and the main/pipeline/make_dom_repr.py module processes the full DOM including dynamically injected elements typical of SPAs.

How does the generated spider know when to stop scrolling?

The generated spider implements a scroll-until-exhausted algorithm based on the SCRAPY_CREATION_PROMPT template in main/pipeline/roiclf_spcandmkr.py. Typically, the spider counts the number of items (e.g., product cards) before and after each scroll using Playwright's page.locator().count(). If the count remains unchanged after scrolling and waiting for network activity, the loop breaks, indicating all content has loaded.

Is Playwright the only browser automation option for the generated spiders?

While the examples and default prompts in Spider Creator emphasize Scrapy-Playwright for handling infinite scroll and JavaScript execution, the architecture is not strictly limited to Playwright. The SCRAPY_CREATION_PROMPT can be modified to generate Selenium-based callbacks or even bare Scrapy with Splash requests. However, Playwright is the recommended and most tested approach for handling complex dynamic content in the current implementation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}