Structured and Non-Structured Planning Approaches in Spider Creator: Key Differences Explained

The core difference is that non-structured planning returns raw text for human inspection, while structured planning returns a Pydantic model that drives the automated pipeline.

Spider Creator, an open-source web scraping framework by carlosplanchon/spidercreator, extracts XPaths from target websites using two distinct planning strategies. Understanding how these approaches differ is essential for developers building reliable, automated spider generation pipelines.

What Is Non-Structured Planning in Spider Creator?

Non-structured planning produces a free-form textual description of the actions and XPaths required to scrape a target site. This approach prioritizes human readability over machine consumption.

Implementation and Output

In pipeline/xpath_builder_planning.py, the make_non_structured_planning() function calls the LLM using a standard invocation:

from pipeline.xpath_builder_planning import make_non_structured_planning

# Returns a raw string describing needed XPaths

non_structured_plan = make_non_structured_planning(
    mermaid_code=mermaid_code,
    scrapy_spider=scrapy_spider,
)

The function calls o3_llm.invoke() and returns the raw content string. Because the output is a str with no built-in validation, the pipeline at spidercreator.py lines 96-103 only prints this output for human inspection. It cannot be parsed programmatically for downstream automation.

What Is Structured Planning in Spider Creator?

Structured planning produces a typed, schema-enforced representation using Pydantic models. This approach enables programmatic consumption and drives the automated spider generation pipeline.

The Planning Schema

The structured approach relies on three Pydantic models defined in pipeline/xpath_builder_planning.py (lines 47-66):

• Action: Contains action_description and example_xpaths_you_might_need
• InUrl: Contains url and a list of Action objects (action_list)
• Planning: The root model containing in_url_list (list of InUrl objects) and a summary

Implementation with Structured Output

The make_structured_planning() function uses LangChain's structured output capability:

from pipeline.xpath_builder_planning import make_structured_planning, Planning

# Returns a Pydantic Planning model

structured_plan: Planning = make_structured_planning(
    mermaid_code=mermaid_code,
    scrapy_spider_draft=scrapy_spider,
)

# Programmatically iterate over URLs and actions

for in_url in structured_plan.in_url_list:
    print(f"Processing URL: {in_url.url}")
    for action in in_url.action_list:
        print(f"  Action: {action.action_description}")

This calls o3_llm.with_structured_output(Planning).invoke(), ensuring the LLM output conforms to the schema.

Key Differences Between Structured and Non-Structured Planning

Feature	Non-Structured Planning	Structured Planning
Output Type	Raw str (natural language)	Planning Pydantic model
Schema Enforcement	None	Strict Pydantic validation
LLM Invocation	o3_llm.invoke()	o3_llm.with_structured_output(Planning).invoke()
Downstream Usage	Human inspection only (spidercreator.py lines 96-103)	Drives automated pipeline via PlanningTokenizer
URL Validation	No validation	validate_url applied to each InUrl
Error Handling	Ambiguous text may be malformed	Type-safe access to in_url_list and action_list

How Structured Planning Drives the Spider Creator Pipeline

The structured planning approach is essential for the automated spider generation workflow. After make_structured_planning() returns the Planning model at spidercreator.py lines 104-119, the pipeline performs these operations:

1. Tokenization: The PlanningTokenizer (from planning/plan_tokenizer.py) consumes the structured plan and validates each URL using validate_url.

2. State Management: The tokenizer maintains PlanningState to track which InUrl is currently being processed.

3. Index Mapping: make_planner_idx_to_recording_idx() aligns planner indices with filtered recordings, ensuring the pipeline executes the exact plan for each URL.

4. Execution: The validated, structured data feeds into ROI extraction (make_dom_representation), candidate spider generation (classify_roi_html_create_cand_spider), and verification (verify_spider_exec_result).

Without the structured planning approach, these downstream components could not reliably parse XPaths or map actions to specific URLs.

Summary

• Non-structured planning in Spider Creator returns raw text via make_non_structured_planning(), suitable only for human debugging at spidercreator.py lines 96-103.
• Structured planning returns a Pydantic Planning model via make_structured_planning(), enabling programmatic consumption through PlanningTokenizer and downstream pipeline stages.
• The structured approach enforces URL validation, maintains type safety, and drives automated spider generation, while the non-structured approach serves as a human-readable diagnostic tool.

Frequently Asked Questions

What is the main difference between structured and non-structured planning in Spider Creator?

The main difference is that non-structured planning produces a raw string output meant for human inspection, while structured planning produces a Pydantic Planning model that can be programmatically traversed and validated. The structured approach uses o3_llm.with_structured_output(Planning).invoke() to enforce schema compliance, whereas the non-structured approach simply calls o3_llm.invoke() and returns the text content.

Can I use non-structured planning for production spider generation?

No, non-structured planning is not suitable for production automation. According to the source code in spidercreator.py (lines 96-103), the non-structured output is only printed for human inspection and is not parsed further by the pipeline. Without a defined schema, the system cannot reliably extract XPaths, validate URLs, or map planner indices to recordings. Production workflows require the structured planning approach to feed data into PlanningTokenizer and subsequent verification stages.

How does the PlanningTokenizer validate URLs in structured planning?

The PlanningTokenizer class defined in planning/plan_tokenizer.py validates URLs using the validate_url function (lines 17-66). When processing a structured Planning model, the tokenizer iterates over each InUrl object in the in_url_list, applies URL validation to ensure the URL is well-formed, and maintains a PlanningState to track valid frames. Only URLs passing validation are yielded for downstream processing, ensuring that the spider generation pipeline operates on reliable, well-formed data.

Where are the planning methods defined in the Spider Creator codebase?

Both planning methods are defined in pipeline/xpath_builder_planning.py. The non-structured planner is implemented in make_non_structured_planning() at lines 27-44, which returns a raw string. The structured planner is implemented in make_structured_planning() at lines 69-90, which returns a Pydantic Planning model. The underlying schema definitions for the structured approach (Action, InUrl, Planning) are located at lines 47-66 in the same file.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}