# How to Debug XPath Extraction Issues in Spiders Generated by Spider Creator

> Fix XPath extraction problems in Spider Creator. Inspect the plan, verify spider execution, and adjust DOM chunk sizes or prompts for accurate data results.

- Repository: [Carlos A. Planchón/spidercreator](https://github.com/carlosplanchon/spidercreator)
- Tags: how-to-guide
- Published: 2026-02-26

---

**To debug XPath extraction issues in Spider Creator, inspect the structured plan in [`pipeline/xpath_builder_planning.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/xpath_builder_planning.py), verify candidate spider execution using `verify_spider_exec_result` from [`pipeline/verify_sp_execution.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/verify_sp_execution.py), and adjust DOM chunk sizes or planning prompts if elements are missed.**

Spider Creator is an open-source framework that automates Scrapy spider generation through LLM-driven planning and verification. When XPath selectors fail to extract target data, the framework provides specific debugging hooks across its pipeline stages to identify and resolve extraction failures.

## Understanding the Spider Creator Pipeline Architecture

Spider Creator processes web scraping tasks through five distinct stages. Understanding these stages helps isolate where XPath extraction breaks down.

### Stage 1: Planning and XPath Generation

The planning stage converts a mind-map of the target page into a structured extraction plan. In [`pipeline/xpath_builder_planning.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/xpath_builder_planning.py), the `Planning` model defines the `Action` class containing `action_description`, `example_xpaths_you_might_need`, and `verify` fields. The LLM prompt in this file requests specific XPaths for each action, making it the first place to check when selectors are missing or malformed.

### Stage 2: ROI Classification

Region-of-Interest classification occurs in [`pipeline/roiclf_spcandmkr.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/roiclf_spcandmkr.py). This stage classifies DOM fragments to determine if they contain the expected content and generates candidate spider fragments including the XPaths that will be used for extraction.

### Stage 3: Candidate Spider Execution

Candidate spiders are executed locally via [`ctxexec/exec_sp.py`](https://github.com/carlosplanchon/spidercreator/blob/main/ctxexec/exec_sp.py) and [`ctxexec/cand_sp_exec.py`](https://github.com/carlosplanchon/spidercreator/blob/main/ctxexec/cand_sp_exec.py). Raw outputs are collected in `CAND_SPIDER_EXEC_RESULTS` for subsequent verification.

### Stage 4: Verification

The verification stage in [`pipeline/verify_sp_execution.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/verify_sp_execution.py) runs a structured LLM (typically `gpt-4o`) to compare raw spider output against verification criteria. The `verify_spider_exec_result` function returns an `XPathExecutionVerificationResult` containing a score (0-100) and detailed explanation of extraction failures.

## Step-by-Step Debugging Workflow

When XPath extraction fails, follow this systematic approach to identify and resolve the issue.

### Step 1: Identify the Failing Action in the Plan

Run [`spidercreator.py`](https://github.com/carlosplanchon/spidercreator/blob/main/spidercreator.py) and examine the console output showing the plan dump. Locate the `action_description` whose `verify` field mentions the missing data (e.g., "price of the property"). Check the `example_xpaths_you_might_need` list to see what selectors were requested during planning.

### Step 2: Inspect Generated Spider Code

For the selected planner index, examine the candidate spider fragments printed by [`spidercreator.py`](https://github.com/carlosplanchon/spidercreator/blob/main/spidercreator.py):

```python
for key, chunk in CAND_SPIDER_CREATION_RESULTS.items():
    if chunk not in [False, None]:
        print(f"\n{'-' * 50} {key}")
        print(f"Have desired content?: {chunk.result}\n")
        if chunk.spider_code is not None:
            print(chunk.spider_code)  # Generated spider code with XPaths

        print("--- EXPLANATION ---")
        print(chunk.explanation)

```

If the XPath is missing or malformed, the issue originated in the planning stage.

### Step 3: Run Manual Verification

Invoke the verification function directly to see detailed LLM reasoning:

```python
from pipeline.verify_sp_execution import verify_spider_exec_result

result = verify_spider_exec_result(
    spider_code_runnable=generated_spider_code,
    verification_criteria="price should be a number like 123 456",
    extracted_content_on_rec="The price is 123 456",
    spider_output=raw_spider_output
)

print(result.model_dump())

```

The `score` and `explanation` fields reveal whether the XPath returned empty strings or unexpected values.

### Step 4: Check Verification Criteria

The criteria are assembled in [`verification_pipeline.py`](https://github.com/carlosplanchon/spidercreator/blob/main/verification_pipeline.py) via `get_verification_criteria`:

```python
from pipeline.verification_pipeline import get_verification_criteria

plan_json = planning_object.model_dump()
criteria = get_verification_criteria(plan_json)

print("Verification criteria for this plan:")
print(criteria)

```

If the wording is ambiguous (e.g., "get the price" vs. "extract the numeric price value"), the LLM may generate mismatched scores. Refine the `verify` field in your action definitions for specificity.

### Step 5: Examine ROI Boundaries

Before classification, check if the target element falls within the correct ROI:

```python
print(dom_repr.render_system.get_roi_text_render_with_pos_xpath(roi_idx=idx))

```

If the ROI does not include the target element, the generated XPath will never find it. Adjust `MAX_NODE_REPR_LENGTH` in [`pipeline/make_dom_repr.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/make_dom_repr.py):

```python
from pipeline.make_dom_repr import make_dom_representation

dom = make_dom_representation(
    website_html=html_string,
    MAX_NODE_REPR_LENGTH=65536  # Increase from default 32768

)

```

### Step 6: Refine Planning Prompts

If the LLM generates invalid selectors, modify the prompt in [`pipeline/xpath_builder_planning.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/xpath_builder_planning.py). The `XPATH_BUILDER_PLANNING_PROMPT` asks:

```

Based on the prior information, grouped by URL, what xpaths I have to build, and for which action?

```

Add constraints such as:
- "Make sure each XPath ends with `/text()` when the target is a plain string"
- "Prefer `contains(@class, 'price')` over absolute paths"
- "Avoid using indices like `[1]` unless absolutely necessary"

## Practical Code Examples

### Invoking the Verification LLM Directly

```python
from pipeline.verify_sp_execution import verify_spider_exec_result

spider_code = """
import scrapy

class MySpider(scrapy.Spider):
    name = "myspider"
    start_urls = ["https://example.com"]
    def parse(self, response):
        price = response.xpath("//div[@class='price']/text()").get()
        yield {"price": price}
"""

result = verify_spider_exec_result(
    spider_code_runnable=spider_code,
    verification_criteria="price must be a numeric string (e.g., 123456)",
    extracted_content_on_rec="The price is 123456",
    spider_output="price: 123456"
)

print(f"Score: {result.score}")
print(f"Explanation: {result.explanation}")

```

### Extracting Verification Criteria

```python
from pipeline.verification_pipeline import get_verification_criteria

plan_json = planning_object.model_dump()
criteria = get_verification_criteria(plan_json)

print("Verification criteria for this plan:")
print(criteria)

```

### Adjusting DOM Chunk Sizes

```python
from pipeline.make_dom_repr import make_dom_representation

dom = make_dom_representation(
    website_html=html_string,
    MAX_NODE_REPR_LENGTH=65536  # Increase from default 32768

)

```

## Key Files Reference

| File | Purpose | Link |
|------|---------|------|
| [`pipeline/xpath_builder_planning.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/xpath_builder_planning.py) | Builds the structured planning model and prompts the LLM for XPaths. | [view](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/xpath_builder_planning.py) |
| [`pipeline/verify_sp_execution.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/verify_sp_execution.py) | Runs the verification LLM and returns `XPathExecutionVerificationResult`. | [view](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/verify_sp_execution.py) |
| [`pipeline/verification_pipeline.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/verification_pipeline.py) | Extracts verification criteria and orchestrates the verification process. | [view](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/verification_pipeline.py) |
| [`pipeline/roiclf_spcandmkr.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/roiclf_spcandmkr.py) | Classifies DOM regions and generates candidate spider fragments. | [view](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/roiclf_spcandmkr.py) |
| [`pipeline/make_dom_repr.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/make_dom_repr.py) | Generates the DOM representation used for ROI extraction. | [view](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/make_dom_repr.py) |
| [`spidercreator.py`](https://github.com/carlosplanchon/spidercreator/blob/main/spidercreator.py) | Orchestrates the workflow and displays debugging information. | [view](https://github.com/carlosplanchon/spidercreator/blob/main/spidercreator.py) |

## Summary

- **Spider Creator** generates spiders through a five-stage pipeline: planning, ROI classification, candidate execution, verification, and sorting.
- To debug XPath failures, start by examining the structured plan in [`pipeline/xpath_builder_planning.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/xpath_builder_planning.py) to identify which action's verification criteria failed.
- Inspect generated spider code from `CAND_SPIDER_CREATION_RESULTS` to confirm XPath syntax and structure.
- Use `verify_spider_exec_result` from [`pipeline/verify_sp_execution.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/verify_sp_execution.py) to get detailed LLM scoring and explanations of extraction failures.
- Adjust `MAX_NODE_REPR_LENGTH` in [`pipeline/make_dom_repr.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/make_dom_repr.py) if target elements fall outside ROI boundaries.
- Refine the `XPATH_BUILDER_PLANNING_PROMPT` in [`pipeline/xpath_builder_planning.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/xpath_builder_planning.py) to generate more robust selectors.

## Frequently Asked Questions

### How do I know which XPath is failing in Spider Creator?

Check the console output of [`spidercreator.py`](https://github.com/carlosplanchon/spidercreator/blob/main/spidercreator.py) for the plan dump showing `action_description` and `verify` fields. Match the verification criteria describing your missing data to the corresponding `example_xpaths_you_might_need` entry. Then examine the `CAND_SPIDER_CREATION_RESULTS` output to see the actual XPath used in the generated spider code.

### Why does my XPath return empty results even though the element exists?

The target element likely falls outside the Region-of-Interest (ROI) boundaries created during DOM representation. In [`pipeline/make_dom_repr.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/make_dom_repr.py), increase `MAX_NODE_REPR_LENGTH` beyond the default 32768 bytes to ensure the element remains within a single ROI chunk. Alternatively, check [`roiclf_spcandmkr.py`](https://github.com/carlosplanchon/spidercreator/blob/main/roiclf_spcandmkr.py) to verify the ROI classification correctly identifies the region containing your target data.

### Can I manually test XPaths before running the full pipeline?

Yes. Import `verify_spider_exec_result` from [`pipeline/verify_sp_execution.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/verify_sp_execution.py) and call it with your spider code, verification criteria, and sample output. This returns an `XPathExecutionVerificationResult` with a 0-100 score and detailed explanation without requiring a full pipeline run through [`spidercreator.py`](https://github.com/carlosplanchon/spidercreator/blob/main/spidercreator.py).

### How do I fix XPath generation when the LLM produces invalid selectors?

Modify the `XPATH_BUILDER_PLANNING_PROMPT` in [`pipeline/xpath_builder_planning.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/xpath_builder_planning.py) to include specific constraints. Add instructions such as "Use `contains(@class, 'target')` instead of absolute paths" or "Append `/text()` for string extraction" to guide the LLM toward valid, robust selectors. Then regenerate the plan by running `python main.py --task_id <your_task>`.