# How Spider Creator Uses Playwright to Scrape JavaScript-Rendered Pages

> Spider Creator harnesses Playwright's three-layer architecture to scrape JavaScript-rendered pages. Discover how it captures, processes, and extracts data from dynamic websites efficiently.

- Repository: [Carlos A. Planchón/spidercreator](https://github.com/carlosplanchon/spidercreator)
- Tags: deep-dive
- Published: 2026-02-26

---

**Spider Creator implements a three-layer Playwright architecture that records fully rendered DOM from JavaScript-heavy sites, processes the static HTML through an LLM pipeline, and emits standalone Playwright spiders that wait for network idle states before extraction.**

Spider Creator, an open-source web scraping framework available at carlosplanchon/spidercreator, solves the challenge of extracting data from modern JavaScript-rendered pages through deep Playwright integration. Unlike traditional scrapers that parse static server-side HTML, this tool captures the final DOM state after all client-side scripts execute, enabling reliable scraping of single-page applications and dynamically loaded content.

## The Three-Layer Playwright Architecture

Spider Creator’s ability to handle JavaScript-rendered content relies on three distinct architectural layers that transform dynamic browser states into extractable data structures.

### Layer 1: Browser-Use Recording with Playwright

The first layer leverages the **browser-use** agent to drive Playwright browser sessions during task recording. As the agent executes user-defined natural language tasks, it records interactions and captures the fully rendered DOM after JavaScript execution completes. In [`record_activity.py`](https://github.com/carlosplanchon/spidercreator/blob/main/record_activity.py), the system calls `get_page_html()` and `take_screenshot()` methods on the `browser_context` object to extract the final page state.

```python

# From record_activity.py (lines 55-56)

website_html: str = await agent_obj.browser_context.get_page_html()
website_screenshot: str = await agent_obj.browser_context.take_screenshot()

```

### Layer 2: Pipeline Processing of Rendered DOM

Once captured, the rendered HTML flows into [`spidercreator.py`](https://github.com/carlosplanchon/spidercreator/blob/main/spidercreator.py), which orchestrates the processing pipeline. The system chunks the static HTML representation, compresses it into a DOM structure, and feeds it through LLM-driven stages including mind-map generation, XPath planning, and candidate spider verification. Because the input HTML reflects the fully rendered page rather than the initial server response, the generated selectors target the final DOM structure that exists after JavaScript execution.

### Layer 3: Runtime Playwright Execution in Generated Spiders

The final layer ensures that the generated spider code itself uses Playwright to handle JavaScript at runtime. The emitted spiders are standalone Playwright scripts that launch a browser, navigate to the target URL, and explicitly wait for the network to become idle before extraction. According to the [`README.md`](https://github.com/carlosplanchon/spidercreator/blob/main/README.md) (lines 83-87 and 143-147), the generated code uses `sync_playwright()` with `page.wait_for_load_state('networkidle')` to guarantee all JavaScript has executed before applying the planned selectors.

```python

# Generated spider structure (from README.md)

from playwright.sync_api import sync_playwright
from parsel import Selector

class ExampleSpider:
    def fetch(self, page, url):
        page.goto(url)
        page.wait_for_load_state('networkidle')  # Ensures JS execution

        return page.content()

    def parse(self, html):
        sel = Selector(text=html)
        titles = sel.xpath("//h2[@class='article-title']/text()").getall()
        return titles

if __name__ == "__main__":
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page()
        html = ExampleSpider().fetch(page, "https://example.com")
        data = ExampleSpider().parse(html)
        print(data)

```

## Recording JavaScript-Rendered Content in record_activity.py

The [`record_activity.py`](https://github.com/carlosplanchon/spidercreator/blob/main/record_activity.py) file serves as the entry point for capturing dynamic content. It utilizes the `browser-use` library’s Playwright-powered `Agent` to execute natural language tasks while the `BrowserContext` helper exposes methods to capture the rendered state. This approach ensures that content loaded via AJAX, React, Vue, or other client-side frameworks is fully materialized before the HTML reaches the processing pipeline.

## How Generated Spiders Handle Dynamic Content

When Spider Creator emits the final spider code, it embeds Playwright directly into the extraction logic. The generated scripts import `sync_playwright` and implement a `fetch` method that navigates to the target URL and waits for the `networkidle` state. This state indicates that network connections have remained idle for at least 500 milliseconds, signaling that JavaScript has finished rendering the DOM. Only then does the spider extract the HTML and apply the XPath or CSS selectors planned during the pipeline stage.

## Complete Workflow Example

To illustrate how Spider Creator handles JavaScript-rendered pages from task definition to execution, consider the following end-to-end workflow. The user defines a natural language task targeting a JavaScript-heavy site, triggers the recording phase, and receives a Playwright-based spider capable of handling dynamic content.

```python
from spidercreator import create_spider

# Define a natural-language task for a JavaScript-heavy page

TASK = """
Navigate to https://example.com.
Scroll to the bottom to trigger infinite-scroll loading.
Extract the titles of all articles that appear.
"""

# Record the interaction (Browser-Use uses Playwright under the hood)

task_id = create_spider(browser_use_task=TASK)   # Records rendered HTML

# The generated spider is written to results/<task_id>/spider_code.py

# This file contains the sync_playwright implementation that handles

# JavaScript rendering via page.wait_for_load_state('networkidle')

```

## Summary

- Spider Creator integrates Playwright through a three-layer architecture that captures, processes, and executes against JavaScript-rendered content.
- The `browser-use` agent records fully rendered DOM via `get_page_html()` and `take_screenshot()` in [`record_activity.py`](https://github.com/carlosplanchon/spidercreator/blob/main/record_activity.py) (lines 55-56).
- The pipeline processes static HTML representations to generate selectors targeting the final DOM structure rather than initial server responses.
- Generated spiders are standalone Playwright scripts that use `page.wait_for_load_state('networkidle')` to ensure JavaScript execution completes before extraction.
- This design enables reliable scraping of single-page applications, infinite-scroll feeds, and dynamically loaded content without manual headless browser configuration.

## Frequently Asked Questions

### Does Spider Creator require Playwright to be installed separately?

Yes, Playwright is a required dependency for both the recording phase and the generated spiders. The `browser-use` package relies on Playwright to drive browser sessions during task recording, and the emitted spider code imports `sync_playwright` directly to handle JavaScript-rendered pages at runtime.

### Can Spider Creator handle infinite scroll and single-page applications?

Absolutely. Because the generated spiders use Playwright with `page.wait_for_load_state('networkidle')`, they automatically wait for all JavaScript-driven content to load before extraction. During the recording phase, the `browser-use` agent can execute scroll actions to trigger lazy-loaded content, and the resulting spider replicates these interactions to capture dynamically generated data.

### How does the generated spider differ from traditional static HTML scrapers?

Traditional scrapers parse the initial HTML response from the server, which often lacks content rendered by client-side JavaScript. In contrast, spiders generated by Spider Creator are full Playwright scripts that launch a headless browser, execute all page scripts, and extract data from the final DOM. This approach eliminates the need for manual AJAX handling or API reverse-engineering when scraping modern web applications.

### What specific Playwright methods does Spider Creator use to ensure JavaScript execution?

During recording, Spider Creator utilizes `browser_context.get_page_html()` and `browser_context.take_screenshot()` from the `browser-use` package, which internally calls Playwright’s `page.content()` and `page.screenshot()` methods. In the generated spiders, the critical method is `page.wait_for_load_state('networkidle')`, which pauses execution until network activity ceases, indicating that JavaScript has finished rendering the page content.