# Structured and Non-Structured Planning Approaches in Spider Creator: Key Differences Explained

> Understand structured vs non-structured planning in Spider Creator. Discover how raw text or Pydantic models drive your automated pipeline. Learn the key differences now.

- Repository: [Carlos A. Planchón/spidercreator](https://github.com/carlosplanchon/spidercreator)
- Tags: deep-dive
- Published: 2026-02-26

---

**The core difference is that non-structured planning returns raw text for human inspection, while structured planning returns a Pydantic model that drives the automated pipeline.**

Spider Creator, an open-source web scraping framework by `carlosplanchon/spidercreator`, extracts XPaths from target websites using two distinct planning strategies. Understanding how these approaches differ is essential for developers building reliable, automated spider generation pipelines.

## What Is Non-Structured Planning in Spider Creator?

Non-structured planning produces a free-form textual description of the actions and XPaths required to scrape a target site. This approach prioritizes human readability over machine consumption.

### Implementation and Output

In [`pipeline/xpath_builder_planning.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/xpath_builder_planning.py), the `make_non_structured_planning()` function calls the LLM using a standard invocation:

```python
from pipeline.xpath_builder_planning import make_non_structured_planning

# Returns a raw string describing needed XPaths

non_structured_plan = make_non_structured_planning(
    mermaid_code=mermaid_code,
    scrapy_spider=scrapy_spider,
)

```

The function calls `o3_llm.invoke()` and returns the raw `content` string. Because the output is a `str` with no built-in validation, the pipeline at [`spidercreator.py`](https://github.com/carlosplanchon/spidercreator/blob/main/spidercreator.py) lines 96-103 only prints this output for human inspection. It cannot be parsed programmatically for downstream automation.

## What Is Structured Planning in Spider Creator?

Structured planning produces a **typed, schema-enforced** representation using Pydantic models. This approach enables programmatic consumption and drives the automated spider generation pipeline.

### The Planning Schema

The structured approach relies on three Pydantic models defined in [`pipeline/xpath_builder_planning.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/xpath_builder_planning.py) (lines 47-66):

- **`Action`**: Contains `action_description` and `example_xpaths_you_might_need`
- **`InUrl`**: Contains `url` and a list of `Action` objects (`action_list`)
- **`Planning`**: The root model containing `in_url_list` (list of `InUrl` objects) and a `summary`

### Implementation with Structured Output

The `make_structured_planning()` function uses LangChain's structured output capability:

```python
from pipeline.xpath_builder_planning import make_structured_planning, Planning

# Returns a Pydantic Planning model

structured_plan: Planning = make_structured_planning(
    mermaid_code=mermaid_code,
    scrapy_spider_draft=scrapy_spider,
)

# Programmatically iterate over URLs and actions

for in_url in structured_plan.in_url_list:
    print(f"Processing URL: {in_url.url}")
    for action in in_url.action_list:
        print(f"  Action: {action.action_description}")

```

This calls `o3_llm.with_structured_output(Planning).invoke()`, ensuring the LLM output conforms to the schema.

## Key Differences Between Structured and Non-Structured Planning

| Feature | Non-Structured Planning | Structured Planning |
|---------|------------------------|---------------------|
| **Output Type** | Raw `str` (natural language) | `Planning` Pydantic model |
| **Schema Enforcement** | None | Strict Pydantic validation |
| **LLM Invocation** | `o3_llm.invoke()` | `o3_llm.with_structured_output(Planning).invoke()` |
| **Downstream Usage** | Human inspection only ([`spidercreator.py`](https://github.com/carlosplanchon/spidercreator/blob/main/spidercreator.py) lines 96-103) | Drives automated pipeline via `PlanningTokenizer` |
| **URL Validation** | No validation | `validate_url` applied to each `InUrl` |
| **Error Handling** | Ambiguous text may be malformed | Type-safe access to `in_url_list` and `action_list` |

## How Structured Planning Drives the Spider Creator Pipeline

The structured planning approach is essential for the automated spider generation workflow. After `make_structured_planning()` returns the `Planning` model at [`spidercreator.py`](https://github.com/carlosplanchon/spidercreator/blob/main/spidercreator.py) lines 104-119, the pipeline performs these operations:

1. **Tokenization**: The `PlanningTokenizer` (from [`planning/plan_tokenizer.py`](https://github.com/carlosplanchon/spidercreator/blob/main/planning/plan_tokenizer.py)) consumes the structured plan and validates each URL using `validate_url`.

2. **State Management**: The tokenizer maintains `PlanningState` to track which `InUrl` is currently being processed.

3. **Index Mapping**: `make_planner_idx_to_recording_idx()` aligns planner indices with filtered recordings, ensuring the pipeline executes the exact plan for each URL.

4. **Execution**: The validated, structured data feeds into ROI extraction (`make_dom_representation`), candidate spider generation (`classify_roi_html_create_cand_spider`), and verification (`verify_spider_exec_result`).

Without the structured planning approach, these downstream components could not reliably parse XPaths or map actions to specific URLs.

## Summary

- **Non-structured planning** in Spider Creator returns raw text via `make_non_structured_planning()`, suitable only for human debugging at [`spidercreator.py`](https://github.com/carlosplanchon/spidercreator/blob/main/spidercreator.py) lines 96-103.
- **Structured planning** returns a Pydantic `Planning` model via `make_structured_planning()`, enabling programmatic consumption through `PlanningTokenizer` and downstream pipeline stages.
- The structured approach enforces URL validation, maintains type safety, and drives automated spider generation, while the non-structured approach serves as a human-readable diagnostic tool.

## Frequently Asked Questions

### What is the main difference between structured and non-structured planning in Spider Creator?

The main difference is that **non-structured planning** produces a raw string output meant for human inspection, while **structured planning** produces a Pydantic `Planning` model that can be programmatically traversed and validated. The structured approach uses `o3_llm.with_structured_output(Planning).invoke()` to enforce schema compliance, whereas the non-structured approach simply calls `o3_llm.invoke()` and returns the text content.

### Can I use non-structured planning for production spider generation?

No, non-structured planning is not suitable for production automation. According to the source code in [`spidercreator.py`](https://github.com/carlosplanchon/spidercreator/blob/main/spidercreator.py) (lines 96-103), the non-structured output is only printed for human inspection and is not parsed further by the pipeline. Without a defined schema, the system cannot reliably extract XPaths, validate URLs, or map planner indices to recordings. Production workflows require the structured planning approach to feed data into `PlanningTokenizer` and subsequent verification stages.

### How does the PlanningTokenizer validate URLs in structured planning?

The `PlanningTokenizer` class defined in [`planning/plan_tokenizer.py`](https://github.com/carlosplanchon/spidercreator/blob/main/planning/plan_tokenizer.py) validates URLs using the `validate_url` function (lines 17-66). When processing a structured `Planning` model, the tokenizer iterates over each `InUrl` object in the `in_url_list`, applies URL validation to ensure the URL is well-formed, and maintains a `PlanningState` to track valid frames. Only URLs passing validation are yielded for downstream processing, ensuring that the spider generation pipeline operates on reliable, well-formed data.

### Where are the planning methods defined in the Spider Creator codebase?

Both planning methods are defined in [`pipeline/xpath_builder_planning.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/xpath_builder_planning.py). The **non-structured planner** is implemented in `make_non_structured_planning()` at lines 27-44, which returns a raw string. The **structured planner** is implemented in `make_structured_planning()` at lines 69-90, which returns a Pydantic `Planning` model. The underlying schema definitions for the structured approach (`Action`, `InUrl`, `Planning`) are located at lines 47-66 in the same file.