# Spider Creator Multi-Stage XPath Planning: How It Works Internally

> Discover how Spider Creator's multi-stage XPath planning works internally. Learn about its two-phase LLM approach for reliable data extraction plans. Read now.

- Repository: [Carlos A. Planchón/spidercreator](https://github.com/carlosplanchon/spidercreator)
- Tags: internals
- Published: 2026-02-26

---

**Spider Creator's multi-stage XPath planning uses a two-phase LLM approach—first generating a free-form planning sketch, then enforcing a strict Pydantic schema—to create validated extraction plans that are tokenized and aligned with original recordings for reliable execution.**

Spider Creator, developed by Carlos Planchón in the `carlosplanchon/spidercreator` repository, implements a sophisticated multi-stage XPath planning system that bridges high-level LLM reasoning with precise DOM extraction. This architecture separates the creative planning phase from structured execution preparation, ensuring that XPath extraction plans are both semantically rich and mechanically valid.

## Stage 1: Generate a Raw (Non-Structured) Planning Sketch

The pipeline begins with **non-structured planning** to gather raw XPath ideas without schema constraints. The `make_non_structured_planning` function in [`pipeline/xpath_builder_planning.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/xpath_builder_planning.py) initiates this brainstorming phase.

```python
from pipeline.xpath_builder_planning import make_non_structured_planning

raw_plan = make_non_structured_planning(
    mermaid_code=mermaid,               # mind-map of the whole crawl

    scrapy_spider=scrapy_spider_draft   # draft spider code

)

```

This function constructs a **HumanMessage** using the `XPATH_BUILDER_PLANNING_PROMPT` and sends it to the base LLM (`o3_llm`). The LLM returns free-form text describing required XPaths, allowing creative exploration without premature structural constraints.

## Stage 2: Enforce Structured Planning with Pydantic

The second stage transforms raw ideas into a machine-validated plan using **structured planning**. The `make_structured_planning` function wraps the LLM with `o3_llm.with_structured_output(Planning)`, enforcing a strict Pydantic schema.

```python
from pipeline.xpath_builder_planning import make_structured_planning, Planning

structured_plan: Planning = make_structured_planning(
    mermaid_code=mermaid,
    scrapy_spider_draft=scrapy_spider_draft
)

```

The schema defines three nested **Pydantic** models that enforce precise JSON structure:

- **Action**: Contains `action_description`, `example_xpaths_you_might_need`, and `verify` fields
- **InUrl**: Contains `url` and an `action_list` (list of Action objects)
- **Planning**: Contains `in_url_list` (list of InUrl objects) and a `summary` string

This guarantees that every URL and XPath action conforms to the expected format, eliminating malformed outputs before execution begins.

## Stage 3: Tokenize and Validate the Structured Plan

Once structured, the plan requires **tokenization** to become iterable. The `PlanningTokenizer` class in [`planning/plan_tokenizer.py`](https://github.com/carlosplanchon/spidercreator/blob/main/planning/plan_tokenizer.py) converts the `Planning` object into discrete frames while validating URLs.

```python
from planning.plan_tokenizer import PlanningTokenizer

tokenizer = PlanningTokenizer(structured_planning=structured_plan)

# Move the cursor to the first valid URL frame

tokenizer.advance_planning_until_valid_url()

```

The tokenizer maintains an internal index (`planning_idx`) and categorizes each frame as `VALID_URL`, `INVALID_URL`, or `EOF`. The `advance_planning_until_valid_url` method uses **tldextract** and **validators.url** to verify domain legitimacy, automatically skipping malformed or placeholder URLs.

## Stage 4: Filter Original Browser Recordings

Raw browser recordings contain noise such as `about:blank` pages and redundant navigation events. The `RecordingInterpreter` class in [`planning/rec_filtering.py`](https://github.com/carlosplanchon/spidercreator/blob/main/planning/rec_filtering.py) sanitizes these inputs before alignment.

```python
from planning.rec_filtering import RecordingInterpreter

rec_itpr = RecordingInterpreter(recordings=recordings_data)
rec_itpr.start()                     # builds filtered_recording

filtered = rec_itpr.get_filtered_recordings_list()

```

The interpreter iterates through raw recordings, skipping placeholder pages and extracting only essential fields: `url`, `model_thoughts`, `model_actions`, and `extracted_content`. This filtering ensures that the alignment stage compares planning steps against meaningful page states rather than browser initialization noise.

## Stage 5: Align Planning Steps with Recordings

The critical bridge between planning and execution occurs in `make_planner_idx_to_recording_idx` within [`planning/planner_to_rec.py`](https://github.com/carlosplanchon/spidercreator/blob/main/planning/planner_to_rec.py). This function creates a bidirectional mapping between planner indices and filtered recording indices.

```python
from planning.planner_to_rec import make_planner_idx_to_recording_idx

idx_map = make_planner_idx_to_recording_idx(
    recordings_itpr=rec_itpr,
    planning_tokenizer=tokenizer
)

```

The alignment algorithm walks through the structured plan and filtered recordings in parallel. When a planner URL matches a recording URL and hasn't been previously mapped, it stores the association `planner_idx → recording_idx`. This ensures that each XPath planning step executes against the correct historical page snapshot, maintaining context integrity throughout the crawl sequence.

## Stage 6: Execute Planning Steps Against Contextual HTML

With the index map established, the final execution loop in [`spidercreator.py`](https://github.com/carlosplanchon/spidercreator/blob/main/spidercreator.py) retrieves the correct HTML for each planning step. The system extracts the page DOM, passes it along with the current `Action` JSON to `classify_roi_html_create_cand_spider`, and validates results against the `verify` field defined in the structured plan.

Successful candidate spiders are combined into the final Scrapy spider implementation. This execution stage depends entirely on the **index map** built during the alignment phase, demonstrating how Spider Creator's multi-stage XPath planning creates a reliable pipeline from LLM reasoning to working extraction code.

## End-to-End XPath Planning Example

The following example demonstrates the complete multi-stage pipeline:

```python

# 1. Load recordings (already done earlier)

recordings = load_recordings("recordings/example_task")

# 2. Build the mind-map and spider draft (omitted for brevity)

mermaid = make_mermaid_mindmap(recordings)
spider_draft = make_scrapy_spider_draft(recordings, mermaid)

# 3. Structured planning (the core of XPath planning)

structured_plan = make_structured_planning(mermaid, spider_draft)

# 4. Tokenise and validate

tokenizer = PlanningTokenizer(structured_plan)
tokenizer.advance_planning_until_valid_url()

# 5. Filter recordings

rec_itpr = RecordingInterpreter(recordings)
rec_itpr.start()

# 6. Align planner with recordings

planner_to_rec = make_planner_idx_to_recording_idx(rec_itpr, tokenizer)

# 7. Inspect the mapping

print(planner_to_rec)

# Example output: {0: 3, 1: 7, 2: 12}

# → Planner step 0 uses recording 3, etc.

```

## Key Files in the Repository

| File | Role |
|------|------|
| [`pipeline/xpath_builder_planning.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/xpath_builder_planning.py) | Prompt handling, `Planning` schema, non-structured and structured planning functions |
| [`planning/plan_tokenizer.py`](https://github.com/carlosplanchon/spidercreator/blob/main/planning/plan_tokenizer.py) | Tokenises `Planning` objects, validates URLs, advances frames |
| [`planning/rec_filtering.py`](https://github.com/carlosplanchon/spidercreator/blob/main/planning/rec_filtering.py) | Cleans raw recordings, creates `filtered_recording` |
| [`planning/planner_to_rec.py`](https://github.com/carlosplanchon/spidercreator/blob/main/planning/planner_to_rec.py) | Aligns planner indices with filtered recordings |
| [`spidercreator.py`](https://github.com/carlosplanchon/spidercreator/blob/main/spidercreator.py) | Orchestrates the whole pipeline (mind-map → planning → execution) |

## Summary

- **Spider Creator** implements a two-phase LLM approach to XPath planning: first generating free-form ideas, then enforcing strict Pydantic schemas.
- The **non-structured planning stage** (`make_non_structured_planning`) allows creative exploration without schema constraints.
- The **structured planning stage** (`make_structured_planning`) validates outputs against `Planning`, `InUrl`, and `Action` models.
- **PlanningTokenizer** converts structured plans into iterable frames, validating URLs using `tldextract` and `validators.url`.
- **RecordingInterpreter** filters raw browser recordings to remove noise like `about:blank` pages.
- **make_planner_idx_to_recording_idx** creates precise mappings between planning steps and historical page states, ensuring each XPath executes against the correct HTML snapshot.

## Frequently Asked Questions

### What is the difference between non-structured and structured planning in Spider Creator?

**Non-structured planning** uses `make_non_structured_planning` to generate free-form text descriptions of required XPaths without schema constraints, allowing the LLM to brainstorm creatively. **Structured planning** uses `make_structured_planning` with Pydantic models (`Planning`, `InUrl`, `Action`) to enforce a strict JSON schema, ensuring machine-readable outputs that can be tokenized and executed reliably.

### How does Spider Creator validate URLs during XPath planning?

URL validation occurs in the `PlanningTokenizer` class within [`planning/plan_tokenizer.py`](https://github.com/carlosplanchon/spidercreator/blob/main/planning/plan_tokenizer.py). The `advance_planning_until_valid_url` method uses `tldextract` to extract domain components and `validators.url` to check URL legitimacy. Invalid URLs or placeholders like `about:blank` are automatically skipped, ensuring only legitimate page frames enter the execution pipeline.

### What Pydantic models define Spider Creator's structured planning schema?

The structured planning schema in [`pipeline/xpath_builder_planning.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/xpath_builder_planning.py) defines three nested Pydantic models: **Action** (containing `action_description`, `example_xpaths_you_might_need`, and `verify`), **InUrl** (containing `url` and an `action_list` of Action objects), and **Planning** (containing `in_url_list` and a `summary` string). These models enforce type safety and precise JSON structure when the LLM generates extraction plans.

### How does Spider Creator align planning steps with browser recordings?

The alignment occurs in `make_planner_idx_to_recording_idx` within [`planning/planner_to_rec.py`](https://github.com/carlosplanchon/spidercreator/blob/main/planning/planner_to_rec.py). This function walks through the structured plan and filtered recordings in parallel, creating a mapping `planner_idx → recording_idx` when URLs match. This ensures each XPath planning step executes against the correct historical page snapshot, maintaining context integrity throughout the crawl sequence.