How to Test and Validate Generated Spiders in SpiderCreator: A 3-Stage Verification Pipeline

SpiderCreator validates Playwright spiders through isolated execution, LLM-based scoring against verification criteria, and deterministic ranking to select the highest-quality candidate.

Ensuring that automatically generated web spiders extract the correct data requires rigorous testing beyond static code analysis. The carlosplanchon/spidercreator repository implements a robust validation pipeline that executes each candidate spider in a sandboxed environment, evaluates its output against user-defined criteria, and ranks results using an LLM judge. This approach eliminates fragile spiders before they reach production.

The Three-Stage Validation Pipeline

SpiderCreator’s verification workflow operates in three distinct phases, moving from execution to evaluation to selection.

Stage 1: Isolated Execution of Candidate Spiders

Every candidate spider runs inside a virtual execution environment (a lightweight local server) to prevent crashes or side-effects from corrupting other tests. The execute_cand_spiders function in ctxexec/pipeline.py orchestrates this process, launching each spider against recorded or live URLs and capturing both the extracted data and any runtime errors.

The CandSpiderExecutor class manages these sandboxed instances, ensuring that resource leaks or infinite loops in one candidate cannot affect the validation of others. This isolation is critical when generating multiple spider variants from the same DOM representation.

Stage 2: Deriving Verification Criteria from Planning Data

Before scoring, the system extracts specific validation requirements from the structured planning JSON. The get_verification_criteria function in pipeline/verification_pipeline.py parses the verify field from the planning output (defined in pipeline/xpath_builder_planning.py), which contains the user’s description of what content should be extracted and how it should be structured.

These criteria serve as the ground truth against which the LLM compares actual spider output, ensuring that validation aligns with the original extraction intent rather than arbitrary heuristics.

Stage 3: LLM-Based Scoring and Selection

The final stage sends the spider code, its execution results, the original extraction intent, and the verification criteria to an LLM judge. The run_verification_on_cand_spider_exec_results function in pipeline/verify_sp_execution.py handles this orchestration, using the XPATH_EXECUTION_VERIFICATION_PROMPT to request a quantitative assessment.

The LLM returns a score between 0-100 plus a detailed explanation of relevance, completeness, and correctness. The sort_spider_eval_results utility then orders candidates by this score, allowing the system to automatically select the highest-performing spider for production use.

Implementation: Running the Verification Pipeline

To test and validate spiders in your own workflow, integrate the three stages as shown below. This example loads recordings, generates candidate spiders, executes them in isolation, and selects the best performer based on LLM scoring.

from utils.recordings import load_recordings
from pipeline.verification_pipeline import (
    get_verification_criteria,
    run_verification_on_cand_spider_exec_results,
)
from ctxexec.pipeline import execute_cand_spiders
from pipeline.sp_combination import get_spider_combination
from pipeline.xpath_builder_planning import make_structured_planning
from planning.rec_filtering import RecordingInterpreter
from pipeline.make_dom_repr import make_dom_representation
from betterhtmlchunking import DomRepresentation
from pipeline.roiclf_spcandmkr import classify_roi_html_create_cand_spider

# 1️⃣ Load recordings for a given task

task_id = "example123"
recordings = load_recordings(f"recordings/{task_id}")

# 2️⃣ Interpret recordings and build a mind-map

interp = RecordingInterpreter(recordings)
interp.start()
mindmap = make_mermaid_mindmap(interp.get_filtered_recordings_list())

# 3️⃣ Create a structured plan (contains `verify` strings)

plan = make_structured_planning(mindmap, "placeholder spider draft")
plan_json = plan.model_dump()

# 4️⃣ For each URL in the plan, generate & run candidate spiders

for plan_item in plan.in_url_list:
    # a) Build DOM representation

    dom: DomRepresentation = make_dom_representation(
        website_html=plan_item.website_html, MAX_NODE_REPR_LENGTH=32768
    )
    # b) Generate candidate spiders

    cand = classify_roi_html_create_cand_spider(
        dom_repr=dom,
        extracted_content_on_rec=plan_item.extracted_content,
        planning=plan_json,
        max_exec_amt=30,
    )
    # c) Execute them in isolated environments

    exec_results = execute_cand_spiders(cand, recordings)
    # d) Verify against criteria

    criteria = get_verification_criteria(plan_item.model_dump())
    eval_results = run_verification_on_cand_spider_exec_results(
        exec_results, plan_item.extracted_content, criteria
    )
    # e) Keep the best spider (highest LLM score)

    best_key = list(eval_results.keys())[0]
    best_spider = exec_results[best_key].spider_code_runnable
    print("✅ Best spider for", plan_item.url, ":\n", best_spider)

Adding Custom Unit Tests for Deterministic Validation

While the library relies on LLM scoring for selection, you can supplement this with deterministic unit tests that verify specific field extraction. The following example uses the same execute_cand_spiders helper to run a generated spider against a known recording and assert exact output values.

import pytest
from utils.recordings import load_recordings
from ctxexec.pipeline import execute_cand_spiders

def test_generated_spider_returns_expected_fields():
    # Load a deterministic recording (recorded once on a stable page)

    recordings = load_recordings("recordings/ci_test")
    
    # Generated spider code under test

    spider_code = """
    from playwright.sync_api import sync_playwright
    from parsel import Selector
    class TestSpider:
        def fetch(self, page, url):
            page.goto(url)
            page.wait_for_load_state('networkidle')
            return page.content()
        def parse(self, html, page):
            sel = Selector(text=html)
            return {"title": sel.css('title::text').get()}
    """
    
    # Wrap code in the expected CandSpiderExecutor structure

    exec_results = execute_cand_spiders(
        {"0": type("Chunk", (), {"spider_code": spider_code, "result": True})},
        recordings,
        max_exec_instances=1,
    )
    
    # Extract and validate output

    output = next(iter(exec_results.values())).spider_code_output_with_local_addresses
    data = eval(output)  # In CI you would parse JSON instead

    assert data["title"] == "Expected Page Title"

This approach leverages the exact sandboxed environment used by the library’s internal validation, ensuring your tests match production execution conditions.

Key Source Files and Their Roles

Understanding the verification architecture requires familiarity with these specific modules:

• ctxexec/pipeline.py – Contains execute_cand_spiders and the CandSpiderExecutor class that manages isolated spider execution.
• pipeline/verification_pipeline.py – Houses get_verification_criteria and the orchestration logic for running the scoring loop.
• pipeline/verify_sp_execution.py – Defines the XPATH_EXECUTION_VERIFICATION_PROMPT and Pydantic models for LLM scoring; includes run_verification_on_cand_spider_exec_results.
• spidercreator.py – The main entry point that orchestrates the end-to-end flow from planning through verification to final spider selection.
• pipeline/xpath_builder_planning.py – Defines the structured planning JSON schema, including the verify field that supplies validation criteria.
• pipeline/sp_combination.py – Merges high-scoring spider fragments into the final runnable implementation after verification completes.

Summary

• Isolated Execution – Candidate spiders run in sandboxed environments via ctxexec/pipeline.py to prevent cross-contamination and capture accurate output.
• Criteria-Driven Validation – The system extracts verification requirements from the planning JSON’s verify field, ensuring alignment with user intent.
• Quantitative LLM Scoring – An LLM judge assigns 0-100 scores based on relevance, completeness, and correctness, with full explanations for transparency.
• Deterministic Selection – The sort_spider_eval_results function ranks candidates by score, automatically promoting the most reliable spider to production.
• Extensible Testing – Developers can layer custom unit tests using the same execute_cand_spiders infrastructure for deterministic regression checks.

Frequently Asked Questions

How does SpiderCreator isolate spider execution during testing?

SpiderCreator uses the CandSpiderExecutor class in ctxexec/pipeline.py to launch each candidate spider in its own lightweight local server environment. This sandboxing ensures that runtime failures, resource leaks, or side-effects in one spider cannot interfere with the execution or validation of other candidates.

What criteria does the LLM use to score generated spiders?

According to the XPATH_EXECUTION_VERIFICATION_PROMPT in pipeline/verify_sp_execution.py, the LLM evaluates three dimensions: relevance (does the output match the extraction intent), completeness (are all required fields present), and correctness (is the data accurately extracted). The model returns a 0-100 numeric score plus a textual justification.

Can I override the LLM scoring with custom validation logic?

Yes. While the library uses run_verification_on_cand_spider_exec_results for automated ranking, you can intercept the CAND_SPIDER_EXEC_RESULTS dictionary after execute_cand_spiders completes and apply your own scoring functions before calling sort_spider_eval_results. The example in the custom unit test section demonstrates how to validate specific fields deterministically using pytest.

Where does SpiderCreator store the verification criteria for each spider?

The criteria originate in the verify field of the structured planning JSON, defined in pipeline/xpath_builder_planning.py. During validation, get_verification_criteria in pipeline/verification_pipeline.py extracts these strings and passes them to the LLM judge alongside the spider’s actual output for comparison.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}