How to Debug XPath Extraction Issues in Spiders Generated by Spider Creator

To debug XPath extraction issues in Spider Creator, inspect the structured plan in pipeline/xpath_builder_planning.py, verify candidate spider execution using verify_spider_exec_result from pipeline/verify_sp_execution.py, and adjust DOM chunk sizes or planning prompts if elements are missed.

Spider Creator is an open-source framework that automates Scrapy spider generation through LLM-driven planning and verification. When XPath selectors fail to extract target data, the framework provides specific debugging hooks across its pipeline stages to identify and resolve extraction failures.

Understanding the Spider Creator Pipeline Architecture

Spider Creator processes web scraping tasks through five distinct stages. Understanding these stages helps isolate where XPath extraction breaks down.

Stage 1: Planning and XPath Generation

The planning stage converts a mind-map of the target page into a structured extraction plan. In pipeline/xpath_builder_planning.py, the Planning model defines the Action class containing action_description, example_xpaths_you_might_need, and verify fields. The LLM prompt in this file requests specific XPaths for each action, making it the first place to check when selectors are missing or malformed.

Stage 2: ROI Classification

Region-of-Interest classification occurs in pipeline/roiclf_spcandmkr.py. This stage classifies DOM fragments to determine if they contain the expected content and generates candidate spider fragments including the XPaths that will be used for extraction.

Stage 3: Candidate Spider Execution

Candidate spiders are executed locally via ctxexec/exec_sp.py and ctxexec/cand_sp_exec.py. Raw outputs are collected in CAND_SPIDER_EXEC_RESULTS for subsequent verification.

Stage 4: Verification

The verification stage in pipeline/verify_sp_execution.py runs a structured LLM (typically gpt-4o) to compare raw spider output against verification criteria. The verify_spider_exec_result function returns an XPathExecutionVerificationResult containing a score (0-100) and detailed explanation of extraction failures.

Step-by-Step Debugging Workflow

When XPath extraction fails, follow this systematic approach to identify and resolve the issue.

Step 1: Identify the Failing Action in the Plan

Run spidercreator.py and examine the console output showing the plan dump. Locate the action_description whose verify field mentions the missing data (e.g., "price of the property"). Check the example_xpaths_you_might_need list to see what selectors were requested during planning.

Step 2: Inspect Generated Spider Code

For the selected planner index, examine the candidate spider fragments printed by spidercreator.py:

for key, chunk in CAND_SPIDER_CREATION_RESULTS.items():
    if chunk not in [False, None]:
        print(f"\n{'-' * 50} {key}")
        print(f"Have desired content?: {chunk.result}\n")
        if chunk.spider_code is not None:
            print(chunk.spider_code)  # Generated spider code with XPaths

        print("--- EXPLANATION ---")
        print(chunk.explanation)

If the XPath is missing or malformed, the issue originated in the planning stage.

Step 3: Run Manual Verification

Invoke the verification function directly to see detailed LLM reasoning:

from pipeline.verify_sp_execution import verify_spider_exec_result

result = verify_spider_exec_result(
    spider_code_runnable=generated_spider_code,
    verification_criteria="price should be a number like 123 456",
    extracted_content_on_rec="The price is 123 456",
    spider_output=raw_spider_output
)

print(result.model_dump())

The score and explanation fields reveal whether the XPath returned empty strings or unexpected values.

Step 4: Check Verification Criteria

The criteria are assembled in verification_pipeline.py via get_verification_criteria:

from pipeline.verification_pipeline import get_verification_criteria

plan_json = planning_object.model_dump()
criteria = get_verification_criteria(plan_json)

print("Verification criteria for this plan:")
print(criteria)

If the wording is ambiguous (e.g., "get the price" vs. "extract the numeric price value"), the LLM may generate mismatched scores. Refine the verify field in your action definitions for specificity.

Step 5: Examine ROI Boundaries

Before classification, check if the target element falls within the correct ROI:

print(dom_repr.render_system.get_roi_text_render_with_pos_xpath(roi_idx=idx))

If the ROI does not include the target element, the generated XPath will never find it. Adjust MAX_NODE_REPR_LENGTH in pipeline/make_dom_repr.py:

from pipeline.make_dom_repr import make_dom_representation

dom = make_dom_representation(
    website_html=html_string,
    MAX_NODE_REPR_LENGTH=65536  # Increase from default 32768

)

Step 6: Refine Planning Prompts

If the LLM generates invalid selectors, modify the prompt in pipeline/xpath_builder_planning.py. The XPATH_BUILDER_PLANNING_PROMPT asks:

Based on the prior information, grouped by URL, what xpaths I have to build, and for which action?

Add constraints such as:

• "Make sure each XPath ends with /text() when the target is a plain string"
• "Prefer contains(@class, 'price') over absolute paths"
• "Avoid using indices like [1] unless absolutely necessary"

Practical Code Examples

Invoking the Verification LLM Directly

from pipeline.verify_sp_execution import verify_spider_exec_result

spider_code = """
import scrapy

class MySpider(scrapy.Spider):
    name = "myspider"
    start_urls = ["https://example.com"]
    def parse(self, response):
        price = response.xpath("//div[@class='price']/text()").get()
        yield {"price": price}
"""

result = verify_spider_exec_result(
    spider_code_runnable=spider_code,
    verification_criteria="price must be a numeric string (e.g., 123456)",
    extracted_content_on_rec="The price is 123456",
    spider_output="price: 123456"
)

print(f"Score: {result.score}")
print(f"Explanation: {result.explanation}")

Extracting Verification Criteria

from pipeline.verification_pipeline import get_verification_criteria

plan_json = planning_object.model_dump()
criteria = get_verification_criteria(plan_json)

print("Verification criteria for this plan:")
print(criteria)

Adjusting DOM Chunk Sizes

from pipeline.make_dom_repr import make_dom_representation

dom = make_dom_representation(
    website_html=html_string,
    MAX_NODE_REPR_LENGTH=65536  # Increase from default 32768

)

Key Files Reference

File	Purpose	Link
pipeline/xpath_builder_planning.py	Builds the structured planning model and prompts the LLM for XPaths.	view
pipeline/verify_sp_execution.py	Runs the verification LLM and returns XPathExecutionVerificationResult.	view
pipeline/verification_pipeline.py	Extracts verification criteria and orchestrates the verification process.	view
pipeline/roiclf_spcandmkr.py	Classifies DOM regions and generates candidate spider fragments.	view
pipeline/make_dom_repr.py	Generates the DOM representation used for ROI extraction.	view
spidercreator.py	Orchestrates the workflow and displays debugging information.	view

Summary

• Spider Creator generates spiders through a five-stage pipeline: planning, ROI classification, candidate execution, verification, and sorting.
• To debug XPath failures, start by examining the structured plan in pipeline/xpath_builder_planning.py to identify which action's verification criteria failed.
• Inspect generated spider code from CAND_SPIDER_CREATION_RESULTS to confirm XPath syntax and structure.
• Use verify_spider_exec_result from pipeline/verify_sp_execution.py to get detailed LLM scoring and explanations of extraction failures.
• Adjust MAX_NODE_REPR_LENGTH in pipeline/make_dom_repr.py if target elements fall outside ROI boundaries.
• Refine the XPATH_BUILDER_PLANNING_PROMPT in pipeline/xpath_builder_planning.py to generate more robust selectors.

Frequently Asked Questions

How do I know which XPath is failing in Spider Creator?

Check the console output of spidercreator.py for the plan dump showing action_description and verify fields. Match the verification criteria describing your missing data to the corresponding example_xpaths_you_might_need entry. Then examine the CAND_SPIDER_CREATION_RESULTS output to see the actual XPath used in the generated spider code.

Why does my XPath return empty results even though the element exists?

The target element likely falls outside the Region-of-Interest (ROI) boundaries created during DOM representation. In pipeline/make_dom_repr.py, increase MAX_NODE_REPR_LENGTH beyond the default 32768 bytes to ensure the element remains within a single ROI chunk. Alternatively, check roiclf_spcandmkr.py to verify the ROI classification correctly identifies the region containing your target data.

Can I manually test XPaths before running the full pipeline?

Yes. Import verify_spider_exec_result from pipeline/verify_sp_execution.py and call it with your spider code, verification criteria, and sample output. This returns an XPathExecutionVerificationResult with a 0-100 score and detailed explanation without requiring a full pipeline run through spidercreator.py.

How do I fix XPath generation when the LLM produces invalid selectors?

Modify the XPATH_BUILDER_PLANNING_PROMPT in pipeline/xpath_builder_planning.py to include specific constraints. Add instructions such as "Use contains(@class, 'target') instead of absolute paths" or "Append /text() for string extraction" to guide the LLM toward valid, robust selectors. Then regenerate the plan by running python main.py --task_id <your_task>.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}