How the Candidate Spider Execution and Verification Pipeline Works in SpiderCreator

The candidate spider execution and verification pipeline transforms AI-generated spider drafts into runnable Parsel scripts, executes them against locally-served HTML recordings, and uses an LLM to score outputs against verification criteria, selecting the highest-scoring candidate for production.

The carlosplanchon/spidercreator repository automates web-scraping spider creation through a multi-stage pipeline. After generating multiple candidate spiders from a high-level plan, the system must determine which draft actually works. This article explains the candidate spider execution and verification pipeline that validates these drafts before selecting a winner.

Candidate-Spider Generation

Before execution begins, the planner produces several candidate spider snippets stored in CAND_SPIDER_CREATION_RESULTS. Each entry contains raw Python code generated by an LLM based on the target site's structure. Only entries containing valid spider_code proceed to the execution stage.

Execution Pipeline

The execution phase converts raw candidate code into runnable spiders and tests them against local HTML recordings. This process is orchestrated by execute_cand_spiders in ctxexec/pipeline.py and implemented by the CandSpiderExecutor class in ctxexec/cand_sp_exec.py.

Iterating Over Creation Results

The pipeline first filters CAND_SPIDER_CREATION_RESULTS to process only valid candidates:

for key, chunk in CAND_SPIDER_CREATION_RESULTS.items():
    if chunk is not False and chunk.spider_code not in [None, ""]:
        # Process valid candidate

Instantiating the Executor

For each valid candidate, the system creates a CandSpiderExecutor instance with the spider code, HTML recordings, and a port offset to avoid conflicts:

cand_spider_executor = CandSpiderExecutor(
    spider_code=chunk.spider_code,
    recordings_data=recordings_data,
    port_offset=port_offset
)
cand_spider_executor.start()

The Execution Lifecycle

Inside CandSpiderExecutor.start(), the following steps transform and execute the spider:

1. Make the spider runnable – An LLM rewrites the candidate into a pure-Parsel script via make_cand_spider_runnable:

self.spider_code_runnable = make_cand_spider_runnable(self.spider_code)

2. Extract URLs – The system identifies target URLs from the runnable code using get_urls_from_spider_code:

self.runnable_spider_urls = get_urls_from_spider_code(self.spider_code_runnable)

3. Map URLs to local recordings – Each URL is mapped to a locally-served HTML snapshot with an allocated port via map_url_to_exec_context:

self.URL_TO_HTML, self.port_offset = map_url_to_exec_context(
    recordings_data=self.recordings_data,
    spider_urls=self.runnable_spider_urls,
    INITIAL_PORT=self.INITIAL_PORT,
    port_offset=self.port_offset
)

4. Rewrite spider ports – The spider code is modified to point to local addresses using rewrite_ports_in_spider:

self.spider_code_with_local_addresses = rewrite_ports_in_spider(
    spider_code=self.spider_code_runnable,
    URL_TO_LOCAL_ADDRESSES=self.URL_TO_LOCAL_ADDRESSES
)

5. Spin up HTTP servers – Local servers serve the HTML recordings on allocated ports via build_http_server_contexts:

self.http_server_contexts = build_http_server_contexts(URL_TO_HTML=self.URL_TO_HTML)

6. Execute the spider – The modified spider runs in a PTY-process while servers are active, capturing all output via execute_spider_with_http_server_context:

self.spider_code_output_with_local_addresses = execute_spider_with_http_server_context(
    spider_code=self.spider_code_with_local_addresses,
    http_server_contexts=self.http_server_contexts
)

Results are stored in CAND_SPIDER_EXEC_RESULTS before proceeding to verification.

Verification Pipeline

After execution, the system evaluates which candidate performed best against the original requirements using run_verification_on_cand_spider_exec_results in pipeline/verification_pipeline.py.

Building Verification Criteria

The pipeline extracts verification rules from the planning JSON, specifically from "verify" fields in each action:

verification_criteria = "\n".join(
    [action["verify"] for action in plan_json["action_list"]]
)

Evaluating Candidates with LLM

For each executed candidate, the system calls verify_spider_exec_result from pipeline/verify_sp_execution.py, which sends a structured prompt to gpt-4o containing:

• The runnable spider code
• Verification criteria
• Expected content from original recordings
• Actual spider output

The LLM returns an XPathExecutionVerificationResult with a score (0-100) and explanation.

Sorting and Selecting the Best Candidate

Results are sorted by descending score:

sorted_eval_results = dict(
    sorted(spider_eval_results.items(),
           key=lambda item: item[1].score,
           reverse=True)
)

The highest-scoring candidate's runnable code and output proceed to final spider combination.

End-to-End Integration

The main driver in spidercreator.py orchestrates the complete flow:

# Execute all candidates

CAND_SPIDER_EXEC_RESULTS = execute_cand_spiders(
    CAND_SPIDER_CREATION_RESULTS=CAND_SPIDER_CREATION_RESULTS,
    recordings_data=recordings_data
)

# Run verification on every candidate's execution result

verification_criteria = get_verification_criteria(plan_json=plan_json)
CAND_SPIDER_EXEC_EVAL_RESULT = run_verification_on_cand_spider_exec_results(
    CAND_SPIDER_EXEC_RESULTS=CAND_SPIDER_EXEC_RESULTS,
    extracted_content_on_rec=extracted_content_on_rec,
    verification_criteria=verification_criteria
)

# Pick the top-scoring candidate

if CAND_SPIDER_EXEC_EVAL_RESULT:
    selected_key = list(CAND_SPIDER_EXEC_EVAL_RESULT.keys())[0]
    spider_code_runnable = CAND_SPIDER_EXEC_RESULTS[selected_key].spider_code_runnable
    spider_output = CAND_SPIDER_EXEC_RESULTS[selected_key].spider_code_output_with_local_addresses

Summary

• The candidate spider execution and verification pipeline validates AI-generated scraping code before production deployment.
• Execution transforms raw candidates into runnable Parsel scripts, maps URLs to local HTML recordings, and runs spiders in isolated HTTP-server environments via CandSpiderExecutor.
• Verification uses gpt-4o to score each candidate's output against criteria extracted from the original plan, returning a 0-100 score.
• The system automatically selects the highest-scoring candidate from CAND_SPIDER_EXEC_EVAL_RESULT for final spider combination.

Frequently Asked Questions

How does the pipeline handle multiple candidate spiders simultaneously?

The execute_cand_spiders function in ctxexec/pipeline.py iterates over CAND_SPIDER_CREATION_RESULTS and instantiates a separate CandSpiderExecutor for each valid candidate. Each executor manages its own port allocation and HTTP server context, allowing candidates to run in isolation without network conflicts.

What criteria does the LLM use to verify spider execution?

The verification criteria are extracted from the "verify" fields in plan_json["action_list"] within pipeline/verification_pipeline.py. The verify_spider_exec_result function in pipeline/verify_sp_execution.py sends these criteria along with the runnable code, expected content, and actual output to gpt-4o, which returns a numerical score and explanation.

How are URLs mapped to local HTML recordings during execution?

Inside CandSpiderExecutor.start() in ctxexec/cand_sp_exec.py, the map_url_to_exec_context function maps each URL extracted from the spider code to a corresponding HTML recording from recordings_data. It allocates unique ports starting from INITIAL_PORT and returns URL_TO_HTML and the updated port_offset for server initialization.

What happens if a candidate spider fails to execute?

If a candidate's spider_code is None, empty, or marked as False in CAND_SPIDER_CREATION_RESULTS, it is skipped during the execution phase in ctxexec/pipeline.py. For candidates that fail during actual execution (e.g., runtime errors), the CandSpiderExecutor captures the output regardless, and the verification stage scores the partial or empty output accordingly, typically resulting in a low score that prevents selection.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}