How Browser Use Records Browser Activity for Spider Generation in SpiderCreator

Browser Use serves as the execution and observation layer that drives a real browser, captures every LLM interaction and page state, and streams structured recordings to the SpiderCreator pipeline for automated spider synthesis.

The carlosplanchon/spidercreator repository automates the creation of Playwright spiders by observing how an LLM navigates a website. At the heart of this observation layer lies Browser Use, a library that controls a real browser instance and exposes hooks for intercepting every step of the agent's execution.

The Three-Stage Recording Pipeline

The integration between Browser Use and SpiderCreator operates through a tightly-coupled, three-stage pipeline that transforms live browsing sessions into deterministic, machine-readable logs.

Stage 1: Capturing Live Sessions with the Browser Use Agent

The recording process begins in record_activity.py, which instantiates a browser_use.Agent and registers an on_step_start hook. This hook executes at the beginning of every agent step, extracting the current page HTML via agent_obj.browser_context.get_page_html(), capturing a screenshot with take_screenshot(), and serializing the agent's internal history—including model thoughts, outputs, actions, and extracted content.

The assembled payload is then POSTed to a local FastAPI endpoint:

# record_activity.py

async def record_activity(agent_obj):
    # Capture page state

    website_html = await agent_obj.browser_context.get_page_html()
    website_screenshot = await agent_obj.browser_context.take_screenshot()
    
    # Pull LLM-generated history elements

    model_thoughts = obj_to_json(agent_obj.state.history.model_thoughts())
    model_outputs = obj_to_json(agent_obj.state.history.model_outputs())
    model_actions = obj_to_json(agent_obj.state.history.model_actions())
    extracted_content = obj_to_json(agent_obj.state.history.extracted_content())
    urls = obj_to_json(agent_obj.state.history.urls())
    
    # Assemble the step summary

    model_step_summary = {
        "website_html": website_html,
        "website_screenshot": website_screenshot,
        "url": urls[-1] if urls else None,
        "model_thoughts": model_thoughts[-1] if model_thoughts else None,
        "model_outputs": model_outputs[-1] if model_outputs else None,
        "model_actions": model_actions[-1] if model_actions else None,
        "extracted_content": extracted_content[-1] if extracted_content else None,
    }
    
    # Send to the local receiver

    send_agent_history_step(data=model_step_summary)

The agent is started with this hook registered:

await agent.run(on_step_start=record_activity, max_steps=30)

Stage 2: Persisting Step Data via FastAPI Endpoint

The FastAPI server implemented in receive_bu_data.py exposes the /post_agent_history_step endpoint. When it receives a JSON payload from the recording hook, it creates a sequentially numbered file (e.g., 1.json, 2.json) in a designated recordings folder. The numbering logic ensures chronological ordering by scanning existing files, extracting their numeric stems, and incrementing the maximum value found.

# receive_bu_data.py

@app.post("/post_agent_history_step")
async def post_agent_history_step(request: Request):
    data = await request.json()
    recordings_folder = Path(app.state.folder_name)
    recordings_folder.mkdir(parents=True, exist_ok=True)

    # Determine next numeric filename

    existing_numbers = [int(p.stem) for p in recordings_folder.iterdir()
                         if p.is_file() and p.suffix == ".json"]
    next_number = max(existing_numbers) + 1 if existing_numbers else 1

    file_path = recordings_folder / f"{next_number}.json"
    with file_path.open("w") as f:
        json.dump(data, f, indent=2)

    return {"status": "ok", "message": f"Saved to {file_path}"}

Stage 3: Loading Recordings into the SpiderCreator Pipeline

Once the browsing session completes, the SpiderCreator pipeline consumes the persisted recordings through utils/recordings.py. The load_recordings function walks the recordings directory, sorts files numerically to preserve execution order, and deserializes each JSON step into a Python dictionary. This ordered list feeds directly into downstream modules that perform DOM compression, candidate spider generation, and verification.

# utils/recordings.py

def load_recordings(directory: str):
    recordings = []
    for filename in sorted(os.listdir(directory), key=lambda x: int(x.split('.')[0])):
        if filename.endswith(".json"):
            filepath = os.path.join(directory, filename)
            with open(filepath, "r", encoding="utf-8") as file:
                recordings.append(json.load(file))
    return recordings

A downstream module can now reconstruct the session:

from utils.recordings import load_recordings

steps = load_recordings("recordings")   # returns List[Dict] ordered by execution

# feed `steps` into the spider-creation pipeline

What Data Does Browser Use Capture for Spider Generation?

The integration captures a rich, multi-modal dataset essential for robust spider synthesis:

• Full DOM snapshots: Raw HTML from get_page_html() enables XPath and CSS selector extraction.
• Visual evidence: Base64-encoded screenshots from take_screenshot() support visual debugging and verification.
• LLM reasoning chain: The agent's history includes model thoughts, outputs, and actions, providing semantic context that maps user goals to concrete scraping logic.
• Navigation state: URLs and extracted content track the precise traversal path through the target website.

According to the SpiderCreator source code, this data is assembled into a model_step_summary dictionary and streamed step-by-step to the receiving server, ensuring no interaction is lost during the LLM's autonomous browsing session.

Summary

• Browser Use functions as the execution and observation layer that drives real browser automation for SpiderCreator.
• The record_activity.py hook captures HTML, screenshots, and complete LLM history at every step via browser_use.Agent.
• receive_bu_data.py persists these steps as sequentially numbered JSON files via a FastAPI endpoint.
• utils/recordings.py loads and orders these recordings for the spider synthesis pipeline.
• The captured data provides both structural (DOM) and semantic (LLM intent) signals required to generate robust Playwright spiders.

Frequently Asked Questions

What is Browser Use in the context of SpiderCreator?

Browser Use is the open-source Python library that powers the LLM-driven browser automation in SpiderCreator. It provides the Agent class that executes high-level browsing commands and exposes lifecycle hooks—such as on_step_start—that enable SpiderCreator to intercept and record every interaction for later analysis and code generation.

How does the recording hook capture page state during execution?

The hook registered in record_activity.py executes at the start of every agent step and calls agent_obj.browser_context.get_page_html() to extract the DOM and take_screenshot() for visual capture. It also serializes the agent's internal history object to capture the LLM's thoughts, actions, and extracted content, assembling everything into a JSON payload that is posted to the local FastAPI receiver.

What file format are the recordings stored in?

Recordings are stored as individual JSON files with numeric filenames (e.g., 1.json, 2.json) in a designated directory. Each file represents a single step of the agent's execution and contains the HTML snapshot, base64-encoded screenshot, current URL, and the LLM's reasoning data for that specific point in the browsing session.

How does SpiderCreator use the recorded Browser Use data?

SpiderCreator loads the recorded JSON steps via utils/recordings.py, which sorts them chronologically to reconstruct the browsing session. The pipeline then analyzes the DOM snapshots to extract CSS selectors and XPath expressions, uses the screenshots for visual verification, and interprets the LLM history to understand the semantic intent behind each action, ultimately synthesizing a Playwright spider that replicates the observed behavior.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}