Spider Creator Multi-Stage XPath Planning: How It Works Internally

Spider Creator's multi-stage XPath planning uses a two-phase LLM approach—first generating a free-form planning sketch, then enforcing a strict Pydantic schema—to create validated extraction plans that are tokenized and aligned with original recordings for reliable execution.

Spider Creator, developed by Carlos Planchón in the carlosplanchon/spidercreator repository, implements a sophisticated multi-stage XPath planning system that bridges high-level LLM reasoning with precise DOM extraction. This architecture separates the creative planning phase from structured execution preparation, ensuring that XPath extraction plans are both semantically rich and mechanically valid.

Stage 1: Generate a Raw (Non-Structured) Planning Sketch

The pipeline begins with non-structured planning to gather raw XPath ideas without schema constraints. The make_non_structured_planning function in pipeline/xpath_builder_planning.py initiates this brainstorming phase.

from pipeline.xpath_builder_planning import make_non_structured_planning

raw_plan = make_non_structured_planning(
    mermaid_code=mermaid,               # mind-map of the whole crawl

    scrapy_spider=scrapy_spider_draft   # draft spider code

)

This function constructs a HumanMessage using the XPATH_BUILDER_PLANNING_PROMPT and sends it to the base LLM (o3_llm). The LLM returns free-form text describing required XPaths, allowing creative exploration without premature structural constraints.

Stage 2: Enforce Structured Planning with Pydantic

The second stage transforms raw ideas into a machine-validated plan using structured planning. The make_structured_planning function wraps the LLM with o3_llm.with_structured_output(Planning), enforcing a strict Pydantic schema.

from pipeline.xpath_builder_planning import make_structured_planning, Planning

structured_plan: Planning = make_structured_planning(
    mermaid_code=mermaid,
    scrapy_spider_draft=scrapy_spider_draft
)

The schema defines three nested Pydantic models that enforce precise JSON structure:

• Action: Contains action_description, example_xpaths_you_might_need, and verify fields
• InUrl: Contains url and an action_list (list of Action objects)
• Planning: Contains in_url_list (list of InUrl objects) and a summary string

This guarantees that every URL and XPath action conforms to the expected format, eliminating malformed outputs before execution begins.

Stage 3: Tokenize and Validate the Structured Plan

Once structured, the plan requires tokenization to become iterable. The PlanningTokenizer class in planning/plan_tokenizer.py converts the Planning object into discrete frames while validating URLs.

from planning.plan_tokenizer import PlanningTokenizer

tokenizer = PlanningTokenizer(structured_planning=structured_plan)

# Move the cursor to the first valid URL frame

tokenizer.advance_planning_until_valid_url()

The tokenizer maintains an internal index (planning_idx) and categorizes each frame as VALID_URL, INVALID_URL, or EOF. The advance_planning_until_valid_url method uses tldextract and validators.url to verify domain legitimacy, automatically skipping malformed or placeholder URLs.

Stage 4: Filter Original Browser Recordings

Raw browser recordings contain noise such as about:blank pages and redundant navigation events. The RecordingInterpreter class in planning/rec_filtering.py sanitizes these inputs before alignment.

from planning.rec_filtering import RecordingInterpreter

rec_itpr = RecordingInterpreter(recordings=recordings_data)
rec_itpr.start()                     # builds filtered_recording

filtered = rec_itpr.get_filtered_recordings_list()

The interpreter iterates through raw recordings, skipping placeholder pages and extracting only essential fields: url, model_thoughts, model_actions, and extracted_content. This filtering ensures that the alignment stage compares planning steps against meaningful page states rather than browser initialization noise.

Stage 5: Align Planning Steps with Recordings

The critical bridge between planning and execution occurs in make_planner_idx_to_recording_idx within planning/planner_to_rec.py. This function creates a bidirectional mapping between planner indices and filtered recording indices.

from planning.planner_to_rec import make_planner_idx_to_recording_idx

idx_map = make_planner_idx_to_recording_idx(
    recordings_itpr=rec_itpr,
    planning_tokenizer=tokenizer
)

The alignment algorithm walks through the structured plan and filtered recordings in parallel. When a planner URL matches a recording URL and hasn't been previously mapped, it stores the association planner_idx → recording_idx. This ensures that each XPath planning step executes against the correct historical page snapshot, maintaining context integrity throughout the crawl sequence.

Stage 6: Execute Planning Steps Against Contextual HTML

With the index map established, the final execution loop in spidercreator.py retrieves the correct HTML for each planning step. The system extracts the page DOM, passes it along with the current Action JSON to classify_roi_html_create_cand_spider, and validates results against the verify field defined in the structured plan.

Successful candidate spiders are combined into the final Scrapy spider implementation. This execution stage depends entirely on the index map built during the alignment phase, demonstrating how Spider Creator's multi-stage XPath planning creates a reliable pipeline from LLM reasoning to working extraction code.

End-to-End XPath Planning Example

The following example demonstrates the complete multi-stage pipeline:

# 1. Load recordings (already done earlier)

recordings = load_recordings("recordings/example_task")

# 2. Build the mind-map and spider draft (omitted for brevity)

mermaid = make_mermaid_mindmap(recordings)
spider_draft = make_scrapy_spider_draft(recordings, mermaid)

# 3. Structured planning (the core of XPath planning)

structured_plan = make_structured_planning(mermaid, spider_draft)

# 4. Tokenise and validate

tokenizer = PlanningTokenizer(structured_plan)
tokenizer.advance_planning_until_valid_url()

# 5. Filter recordings

rec_itpr = RecordingInterpreter(recordings)
rec_itpr.start()

# 6. Align planner with recordings

planner_to_rec = make_planner_idx_to_recording_idx(rec_itpr, tokenizer)

# 7. Inspect the mapping

print(planner_to_rec)

# Example output: {0: 3, 1: 7, 2: 12}

# → Planner step 0 uses recording 3, etc.

Key Files in the Repository

File	Role
pipeline/xpath_builder_planning.py	Prompt handling, Planning schema, non-structured and structured planning functions
planning/plan_tokenizer.py	Tokenises Planning objects, validates URLs, advances frames
planning/rec_filtering.py	Cleans raw recordings, creates filtered_recording
planning/planner_to_rec.py	Aligns planner indices with filtered recordings
spidercreator.py	Orchestrates the whole pipeline (mind-map → planning → execution)

Summary

• Spider Creator implements a two-phase LLM approach to XPath planning: first generating free-form ideas, then enforcing strict Pydantic schemas.
• The non-structured planning stage (make_non_structured_planning) allows creative exploration without schema constraints.
• The structured planning stage (make_structured_planning) validates outputs against Planning, InUrl, and Action models.
• PlanningTokenizer converts structured plans into iterable frames, validating URLs using tldextract and validators.url.
• RecordingInterpreter filters raw browser recordings to remove noise like about:blank pages.
• make_planner_idx_to_recording_idx creates precise mappings between planning steps and historical page states, ensuring each XPath executes against the correct HTML snapshot.

Frequently Asked Questions

What is the difference between non-structured and structured planning in Spider Creator?

Non-structured planning uses make_non_structured_planning to generate free-form text descriptions of required XPaths without schema constraints, allowing the LLM to brainstorm creatively. Structured planning uses make_structured_planning with Pydantic models (Planning, InUrl, Action) to enforce a strict JSON schema, ensuring machine-readable outputs that can be tokenized and executed reliably.

How does Spider Creator validate URLs during XPath planning?

URL validation occurs in the PlanningTokenizer class within planning/plan_tokenizer.py. The advance_planning_until_valid_url method uses tldextract to extract domain components and validators.url to check URL legitimacy. Invalid URLs or placeholders like about:blank are automatically skipped, ensuring only legitimate page frames enter the execution pipeline.

What Pydantic models define Spider Creator's structured planning schema?

The structured planning schema in pipeline/xpath_builder_planning.py defines three nested Pydantic models: Action (containing action_description, example_xpaths_you_might_need, and verify), InUrl (containing url and an action_list of Action objects), and Planning (containing in_url_list and a summary string). These models enforce type safety and precise JSON structure when the LLM generates extraction plans.

How does Spider Creator align planning steps with browser recordings?

The alignment occurs in make_planner_idx_to_recording_idx within planning/planner_to_rec.py. This function walks through the structured plan and filtered recordings in parallel, creating a mapping planner_idx → recording_idx when URLs match. This ensures each XPath planning step executes against the correct historical page snapshot, maintaining context integrity throughout the crawl sequence.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}