# Spider Verification Failures in SpiderCreator: Common Causes and Fixes

> Fix spider verification failures in SpiderCreator. Learn common causes like empty outputs or XPath issues and find solutions for normalizing output and validating plans.

- Repository: [Carlos A. Planchón/spidercreator](https://github.com/carlosplanchon/spidercreator)
- Tags: troubleshooting
- Published: 2026-02-26

---

**Spider verification failures in SpiderCreator typically stem from empty execution outputs, incorrect verification criteria in the action plan, or XPath mismatches, and can be resolved by normalizing spider output, validating plan verification strings, and ensuring complete data capture before LLM scoring.**

SpiderCreator automates web scraping through an LLM-powered verification pipeline that evaluates candidate spiders against ground-truth recordings. When **spider verification failures** occur, they usually trace back to six specific issues in the extraction logic or evaluation criteria that prevent the system from confirming successful data capture.

## How Spider Verification Works in SpiderCreator

The verification subsystem resides primarily in [`pipeline/verification_pipeline.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/verification_pipeline.py). The orchestrator function `run_verification_on_cand_spider_exec_results` (lines 41-71) builds verification criteria by calling `get_verification_criteria` (lines 11-18), which concatenates the `verify` fields from each action in the structured plan. This criteria string, along with the spider's output and the ground-truth `extracted_content_on_rec`, feeds into `verify_spider_exec_result` from [`pipeline/verify_sp_execution.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/verify_sp_execution.py) (lines 59-78). The LLM verifier returns a score from 0 to 100 with an explanation comparing the actual extraction against expected results.

## Common Reasons for Spider Verification Failures

### Empty or Missing Spider Output

The pipeline skips verification entirely when `spider_output.strip() == ""` (line 56 of [`verification_pipeline.py`](https://github.com/carlosplanchon/spidercreator/blob/main/verification_pipeline.py)). When candidate spiders fail to return data—either through execution errors or empty selectors—the result is never scored, causing the pipeline to treat the candidate as a failed extraction. This represents one of the most frequent **spider verification failures** in production runs.

### Incorrect Verification Criteria

The `get_verification_criteria` function pulls verification strings from the `action["verify"]` fields defined in the plan (lines 12-15). If the `action_list` in [`pipeline/xpath_builder_planning.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/xpath_builder_planning.py) contains ambiguous or mismatched criteria, the LLM evaluates spiders against incorrect expectations. This leads to false negatives where technically correct extractions receive low scores because they do not match the wrong verification criteria.

### Mismatched Content Format and Noise

Verification fails when spider output contains raw HTML, extra whitespace, or formatting differences compared to the ground-truth recording. The LLM verifier judges relevance and completeness strictly (prompt description lines 35-38 in [`verify_sp_execution.py`](https://github.com/carlosplanchon/spidercreator/blob/main/verify_sp_execution.py)), meaning spiders returning `"<div>Data</div>"` instead of `"Data"` may score poorly even when the semantic content matches.

### Faulty XPath and Selector Logic

Candidate spiders generated by `classify_roi_html_create_cand_spider` may contain XPaths targeting non-existent DOM elements. These faulty selectors execute without crashing but return no data or wrong content, inevitably failing verification because the output cannot satisfy the criteria defined in the plan's `verify` fields.

### LLM Response Variability

The `XPATH_EXECUTION_VERIFICATION_PROMPT` used for scoring is static (lines 10-40), but the LLM's interpretation varies between runs. Ambiguous prompt language causes inconsistent scoring, with the model occasionally misinterpreting whether specific criteria have been met, leading to sporadic **spider verification failures** across identical spider executions.

### Timeout and Data Truncation

The pipeline sleeps 5 seconds between verifications to avoid rate limits (line 50), but heavy spiders may still produce truncated output. In [`spidercreator.py`](https://github.com/carlosplanchon/spidercreator/blob/main/spidercreator.py) (lines 204-206), results are sliced to `[:5000]` characters, potentially cutting off required verification data before it reaches the LLM evaluator.

## How to Fix Spider Verification Failures

### Ensure Complete Output Capture

Remove the `[:5000]` truncation limit in [`spidercreator.py`](https://github.com/carlosplanchon/spidercreator/blob/main/spidercreator.py) (lines 204-206) when storing results for verification. Alternatively, write complete outputs to temporary files and reference those files in the verification call. This prevents the verifier from receiving partial data that misses required fields.

### Validate Verification Criteria Before Execution

After `make_structured_planning` generates the plan in [`pipeline/xpath_builder_planning.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/xpath_builder_planning.py), implement sanity checks ensuring each `verify` string is non-empty and semantically aligned with its corresponding action description. This prevents the LLM from evaluating against incorrect expectations that guarantee **spider verification failures**.

### Normalize Spider Output

Insert a normalization helper in [`pipeline/verification_pipeline.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/verification_pipeline.py) before line 56 to strip HTML tags, collapse whitespace, and serialize data to consistent formats. This reduces format mismatches that trigger false negatives during the LLM comparison.

### Improve Candidate Generation Quality

Tighten the ROI classification logic in [`roiclf_spcandmkr.py`](https://github.com/carlosplanchon/spidercreator/blob/main/roiclf_spcandmkr.py) to reduce generation of XPaths that never hit target elements. Better initial candidate spiders produce valid outputs that satisfy verification criteria without requiring multiple retry cycles.

### Clarify LLM Prompt Instructions

Update `XPATH_EXECUTION_VERIFICATION_PROMPT` (lines 10-40) to explicitly require JSON output with the schema `{"score": 0-100, "explanation": "string"}`. This reduces hallucination and ensures deterministic scoring behavior across different verification runs.

### Handle Empty Output Gracefully

Modify `execute_cand_spiders` in [`ctxexec/exec_sp.py`](https://github.com/carlosplanchon/spidercreator/blob/main/ctxexec/exec_sp.py) to return descriptive placeholders like `"No data extracted"` instead of empty strings when spiders fail. This prevents the verification pipeline from skipping evaluation entirely and provides diagnostic context for debugging failures.

## Code Examples for Debugging Verification Issues

### Extract Verification Criteria from a Plan

```python
from pipeline.verification_pipeline import get_verification_criteria

# `plan_json` is the dict produced by `plan.model_dump()`

criteria = get_verification_criteria(plan_json)
print("Verification criteria:\n", criteria)

```

This calls `get_verification_criteria` (lines 11-18) to concatenate all `action["verify"]` fields from the structured plan.

### Run Verification on Candidate Spider Results

```python
from pipeline.verification_pipeline import run_verification_on_cand_spider_exec_results

# CAND_SPIDER_EXEC_RESULTS ← dict[int, CandSpiderExecResult] from execute_cand_spiders()

# extracted_content_on_rec ← ground-truth from the recording

# criteria ← string from the previous step

verification_results = run_verification_on_cand_spider_exec_results(
    CAND_SPIDER_EXEC_RESULTS,
    extracted_content_on_rec,
    criteria,
)

# `verification_results` is a dict[int, XPathExecutionVerificationResult]

for idx, result in verification_results.items():
    print(f"Spider {idx} → score: {result.score}, explanation: {result.explanation}")

```

This invokes `run_verification_on_cand_spider_exec_results` (lines 41-71), which internally calls `verify_spider_exec_result` from [`pipeline/verify_sp_execution.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/verify_sp_execution.py).

### Normalize Spider Output Before Verification

```python
import re
from bs4 import BeautifulSoup

def normalise_output(raw: str) -> str:
    # Strip HTML tags

    text = BeautifulSoup(raw, "html.parser").get_text(separator=" ")
    # Collapse whitespace

    return re.sub(r"\s+", " ", text).strip()

# Example usage inside the verification loop

spider_output = normalise_output(cand_spider_exec_obj.spider_code_output_with_local_addresses)

```

Placing this helper just before line 56 in [`verification_pipeline.py`](https://github.com/carlosplanchon/spidercreator/blob/main/verification_pipeline.py) ensures the LLM receives clean, normalized text free from HTML artifacts that could trigger false **spider verification failures**.

## Summary

- **Empty outputs** bypass scoring entirely due to the `strip() == ""` check in [`verification_pipeline.py`](https://github.com/carlosplanchon/spidercreator/blob/main/verification_pipeline.py) line 56.
- **Incorrect criteria** in the plan's `verify` fields cause the LLM to evaluate against wrong expectations.
- **Format mismatches** and HTML noise in spider output trigger strict penalties from the verifier.
- **Truncation** at `[:5000]` characters in [`spidercreator.py`](https://github.com/carlosplanchon/spidercreator/blob/main/spidercreator.py) may hide critical data needed for successful verification.
- **Fixes** include normalizing output, validating plan criteria, removing truncation limits, and clarifying LLM prompts for consistent JSON responses.

## Frequently Asked Questions

### Why does SpiderCreator skip verification for some spiders?

The pipeline skips verification when `spider_output.strip() == ""` at line 56 of [`pipeline/verification_pipeline.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/verification_pipeline.py). If the candidate spider returns empty content—whether from execution failure or XPath errors—the system treats it as a verification failure without invoking the LLM scorer.

### How can I improve verification consistency across different runs?

Clarify the `XPATH_EXECUTION_VERIFICATION_PROMPT` in [`pipeline/verify_sp_execution.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/verify_sp_execution.py) (lines 10-40) to require strict JSON output with explicit `score` and `explanation` fields. Additionally, normalize spider output before verification to remove HTML tags and whitespace variations that cause scoring inconsistency.

### What causes low verification scores even when the extracted data looks correct?

This typically occurs when the `verify` criteria in the plan do not match the actual extraction goals, or when the spider output format differs from the ground-truth recording. Check that `get_verification_criteria` in [`pipeline/verification_pipeline.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/verification_pipeline.py) is pulling the correct `action["verify"]` strings, and ensure your spider normalizes data to match the expected format.

### Where is the verification score calculated in SpiderCreator?

The score calculation happens in `verify_spider_exec_result` within [`pipeline/verify_sp_execution.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/verify_sp_execution.py) (lines 59-78). This function sends the spider output, ground-truth content, and verification criteria to the LLM, which returns a numerical score from 0 to 100 based on the `XPATH_EXECUTION_VERIFICATION_PROMPT` instructions.