Spider Verification Failures in SpiderCreator: Common Causes and Fixes

Spider verification failures in SpiderCreator typically stem from empty execution outputs, incorrect verification criteria in the action plan, or XPath mismatches, and can be resolved by normalizing spider output, validating plan verification strings, and ensuring complete data capture before LLM scoring.

SpiderCreator automates web scraping through an LLM-powered verification pipeline that evaluates candidate spiders against ground-truth recordings. When spider verification failures occur, they usually trace back to six specific issues in the extraction logic or evaluation criteria that prevent the system from confirming successful data capture.

How Spider Verification Works in SpiderCreator

The verification subsystem resides primarily in pipeline/verification_pipeline.py. The orchestrator function run_verification_on_cand_spider_exec_results (lines 41-71) builds verification criteria by calling get_verification_criteria (lines 11-18), which concatenates the verify fields from each action in the structured plan. This criteria string, along with the spider's output and the ground-truth extracted_content_on_rec, feeds into verify_spider_exec_result from pipeline/verify_sp_execution.py (lines 59-78). The LLM verifier returns a score from 0 to 100 with an explanation comparing the actual extraction against expected results.

Common Reasons for Spider Verification Failures

Empty or Missing Spider Output

The pipeline skips verification entirely when spider_output.strip() == "" (line 56 of verification_pipeline.py). When candidate spiders fail to return data—either through execution errors or empty selectors—the result is never scored, causing the pipeline to treat the candidate as a failed extraction. This represents one of the most frequent spider verification failures in production runs.

Incorrect Verification Criteria

The get_verification_criteria function pulls verification strings from the action["verify"] fields defined in the plan (lines 12-15). If the action_list in pipeline/xpath_builder_planning.py contains ambiguous or mismatched criteria, the LLM evaluates spiders against incorrect expectations. This leads to false negatives where technically correct extractions receive low scores because they do not match the wrong verification criteria.

Mismatched Content Format and Noise

Verification fails when spider output contains raw HTML, extra whitespace, or formatting differences compared to the ground-truth recording. The LLM verifier judges relevance and completeness strictly (prompt description lines 35-38 in verify_sp_execution.py), meaning spiders returning "<div>Data</div>" instead of "Data" may score poorly even when the semantic content matches.

Faulty XPath and Selector Logic

Candidate spiders generated by classify_roi_html_create_cand_spider may contain XPaths targeting non-existent DOM elements. These faulty selectors execute without crashing but return no data or wrong content, inevitably failing verification because the output cannot satisfy the criteria defined in the plan's verify fields.

LLM Response Variability

The XPATH_EXECUTION_VERIFICATION_PROMPT used for scoring is static (lines 10-40), but the LLM's interpretation varies between runs. Ambiguous prompt language causes inconsistent scoring, with the model occasionally misinterpreting whether specific criteria have been met, leading to sporadic spider verification failures across identical spider executions.

Timeout and Data Truncation

The pipeline sleeps 5 seconds between verifications to avoid rate limits (line 50), but heavy spiders may still produce truncated output. In spidercreator.py (lines 204-206), results are sliced to [:5000] characters, potentially cutting off required verification data before it reaches the LLM evaluator.

How to Fix Spider Verification Failures

Ensure Complete Output Capture

Remove the [:5000] truncation limit in spidercreator.py (lines 204-206) when storing results for verification. Alternatively, write complete outputs to temporary files and reference those files in the verification call. This prevents the verifier from receiving partial data that misses required fields.

Validate Verification Criteria Before Execution

After make_structured_planning generates the plan in pipeline/xpath_builder_planning.py, implement sanity checks ensuring each verify string is non-empty and semantically aligned with its corresponding action description. This prevents the LLM from evaluating against incorrect expectations that guarantee spider verification failures.

Normalize Spider Output

Insert a normalization helper in pipeline/verification_pipeline.py before line 56 to strip HTML tags, collapse whitespace, and serialize data to consistent formats. This reduces format mismatches that trigger false negatives during the LLM comparison.

Improve Candidate Generation Quality

Tighten the ROI classification logic in roiclf_spcandmkr.py to reduce generation of XPaths that never hit target elements. Better initial candidate spiders produce valid outputs that satisfy verification criteria without requiring multiple retry cycles.

Clarify LLM Prompt Instructions

Update XPATH_EXECUTION_VERIFICATION_PROMPT (lines 10-40) to explicitly require JSON output with the schema {"score": 0-100, "explanation": "string"}. This reduces hallucination and ensures deterministic scoring behavior across different verification runs.

Handle Empty Output Gracefully

Modify execute_cand_spiders in ctxexec/exec_sp.py to return descriptive placeholders like "No data extracted" instead of empty strings when spiders fail. This prevents the verification pipeline from skipping evaluation entirely and provides diagnostic context for debugging failures.

Code Examples for Debugging Verification Issues

Extract Verification Criteria from a Plan

from pipeline.verification_pipeline import get_verification_criteria

# `plan_json` is the dict produced by `plan.model_dump()`

criteria = get_verification_criteria(plan_json)
print("Verification criteria:\n", criteria)

This calls get_verification_criteria (lines 11-18) to concatenate all action["verify"] fields from the structured plan.

Run Verification on Candidate Spider Results

from pipeline.verification_pipeline import run_verification_on_cand_spider_exec_results

# CAND_SPIDER_EXEC_RESULTS ← dict[int, CandSpiderExecResult] from execute_cand_spiders()

# extracted_content_on_rec ← ground-truth from the recording

# criteria ← string from the previous step

verification_results = run_verification_on_cand_spider_exec_results(
    CAND_SPIDER_EXEC_RESULTS,
    extracted_content_on_rec,
    criteria,
)

# `verification_results` is a dict[int, XPathExecutionVerificationResult]

for idx, result in verification_results.items():
    print(f"Spider {idx} → score: {result.score}, explanation: {result.explanation}")

This invokes run_verification_on_cand_spider_exec_results (lines 41-71), which internally calls verify_spider_exec_result from pipeline/verify_sp_execution.py.

Normalize Spider Output Before Verification

import re
from bs4 import BeautifulSoup

def normalise_output(raw: str) -> str:
    # Strip HTML tags

    text = BeautifulSoup(raw, "html.parser").get_text(separator=" ")
    # Collapse whitespace

    return re.sub(r"\s+", " ", text).strip()

# Example usage inside the verification loop

spider_output = normalise_output(cand_spider_exec_obj.spider_code_output_with_local_addresses)

Placing this helper just before line 56 in verification_pipeline.py ensures the LLM receives clean, normalized text free from HTML artifacts that could trigger false spider verification failures.

Summary

• Empty outputs bypass scoring entirely due to the strip() == "" check in verification_pipeline.py line 56.
• Incorrect criteria in the plan's verify fields cause the LLM to evaluate against wrong expectations.
• Format mismatches and HTML noise in spider output trigger strict penalties from the verifier.
• Truncation at [:5000] characters in spidercreator.py may hide critical data needed for successful verification.
• Fixes include normalizing output, validating plan criteria, removing truncation limits, and clarifying LLM prompts for consistent JSON responses.

Frequently Asked Questions

Why does SpiderCreator skip verification for some spiders?

The pipeline skips verification when spider_output.strip() == "" at line 56 of pipeline/verification_pipeline.py. If the candidate spider returns empty content—whether from execution failure or XPath errors—the system treats it as a verification failure without invoking the LLM scorer.

How can I improve verification consistency across different runs?

Clarify the XPATH_EXECUTION_VERIFICATION_PROMPT in pipeline/verify_sp_execution.py (lines 10-40) to require strict JSON output with explicit score and explanation fields. Additionally, normalize spider output before verification to remove HTML tags and whitespace variations that cause scoring inconsistency.

What causes low verification scores even when the extracted data looks correct?

This typically occurs when the verify criteria in the plan do not match the actual extraction goals, or when the spider output format differs from the ground-truth recording. Check that get_verification_criteria in pipeline/verification_pipeline.py is pulling the correct action["verify"] strings, and ensure your spider normalizes data to match the expected format.

Where is the verification score calculated in SpiderCreator?

The score calculation happens in verify_spider_exec_result within pipeline/verify_sp_execution.py (lines 59-78). This function sends the spider output, ground-truth content, and verification criteria to the LLM, which returns a numerical score from 0 to 100 based on the XPATH_EXECUTION_VERIFICATION_PROMPT instructions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}