# How the Candidate Spider Execution and Verification Pipeline Works in SpiderCreator

> Discover how SpiderCreator's candidate spider execution and verification pipeline turns AI drafts into Parsel scripts, tests them locally, and uses an LLM for scoring to select the best candidate.

- Repository: [Carlos A. Planchón/spidercreator](https://github.com/carlosplanchon/spidercreator)
- Tags: internals
- Published: 2026-02-26

---

**The candidate spider execution and verification pipeline transforms AI-generated spider drafts into runnable Parsel scripts, executes them against locally-served HTML recordings, and uses an LLM to score outputs against verification criteria, selecting the highest-scoring candidate for production.**

The `carlosplanchon/spidercreator` repository automates web-scraping spider creation through a multi-stage pipeline. After generating multiple candidate spiders from a high-level plan, the system must determine which draft actually works. This article explains the candidate spider execution and verification pipeline that validates these drafts before selecting a winner.

## Candidate-Spider Generation

Before execution begins, the planner produces several candidate spider snippets stored in `CAND_SPIDER_CREATION_RESULTS`. Each entry contains raw Python code generated by an LLM based on the target site's structure. Only entries containing valid `spider_code` proceed to the execution stage.

## Execution Pipeline

The execution phase converts raw candidate code into runnable spiders and tests them against local HTML recordings. This process is orchestrated by `execute_cand_spiders` in [`ctxexec/pipeline.py`](https://github.com/carlosplanchon/spidercreator/blob/main/ctxexec/pipeline.py) and implemented by the `CandSpiderExecutor` class in [`ctxexec/cand_sp_exec.py`](https://github.com/carlosplanchon/spidercreator/blob/main/ctxexec/cand_sp_exec.py).

### Iterating Over Creation Results

The pipeline first filters `CAND_SPIDER_CREATION_RESULTS` to process only valid candidates:

```python
for key, chunk in CAND_SPIDER_CREATION_RESULTS.items():
    if chunk is not False and chunk.spider_code not in [None, ""]:
        # Process valid candidate

```

### Instantiating the Executor

For each valid candidate, the system creates a `CandSpiderExecutor` instance with the spider code, HTML recordings, and a port offset to avoid conflicts:

```python
cand_spider_executor = CandSpiderExecutor(
    spider_code=chunk.spider_code,
    recordings_data=recordings_data,
    port_offset=port_offset
)
cand_spider_executor.start()

```

### The Execution Lifecycle

Inside `CandSpiderExecutor.start()`, the following steps transform and execute the spider:

1. **Make the spider runnable** – An LLM rewrites the candidate into a pure-Parsel script via `make_cand_spider_runnable`:

```python
self.spider_code_runnable = make_cand_spider_runnable(self.spider_code)

```

2. **Extract URLs** – The system identifies target URLs from the runnable code using `get_urls_from_spider_code`:

```python
self.runnable_spider_urls = get_urls_from_spider_code(self.spider_code_runnable)

```

3. **Map URLs to local recordings** – Each URL is mapped to a locally-served HTML snapshot with an allocated port via `map_url_to_exec_context`:

```python
self.URL_TO_HTML, self.port_offset = map_url_to_exec_context(
    recordings_data=self.recordings_data,
    spider_urls=self.runnable_spider_urls,
    INITIAL_PORT=self.INITIAL_PORT,
    port_offset=self.port_offset
)

```

4. **Rewrite spider ports** – The spider code is modified to point to local addresses using `rewrite_ports_in_spider`:

```python
self.spider_code_with_local_addresses = rewrite_ports_in_spider(
    spider_code=self.spider_code_runnable,
    URL_TO_LOCAL_ADDRESSES=self.URL_TO_LOCAL_ADDRESSES
)

```

5. **Spin up HTTP servers** – Local servers serve the HTML recordings on allocated ports via `build_http_server_contexts`:

```python
self.http_server_contexts = build_http_server_contexts(URL_TO_HTML=self.URL_TO_HTML)

```

6. **Execute the spider** – The modified spider runs in a PTY-process while servers are active, capturing all output via `execute_spider_with_http_server_context`:

```python
self.spider_code_output_with_local_addresses = execute_spider_with_http_server_context(
    spider_code=self.spider_code_with_local_addresses,
    http_server_contexts=self.http_server_contexts
)

```

Results are stored in `CAND_SPIDER_EXEC_RESULTS` before proceeding to verification.

## Verification Pipeline

After execution, the system evaluates which candidate performed best against the original requirements using `run_verification_on_cand_spider_exec_results` in [`pipeline/verification_pipeline.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/verification_pipeline.py).

### Building Verification Criteria

The pipeline extracts verification rules from the planning JSON, specifically from `"verify"` fields in each action:

```python
verification_criteria = "\n".join(
    [action["verify"] for action in plan_json["action_list"]]
)

```

### Evaluating Candidates with LLM

For each executed candidate, the system calls `verify_spider_exec_result` from [`pipeline/verify_sp_execution.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/verify_sp_execution.py), which sends a structured prompt to `gpt-4o` containing:

- The runnable spider code
- Verification criteria
- Expected content from original recordings
- Actual spider output

The LLM returns an `XPathExecutionVerificationResult` with a **score (0-100)** and explanation.

### Sorting and Selecting the Best Candidate

Results are sorted by descending score:

```python
sorted_eval_results = dict(
    sorted(spider_eval_results.items(),
           key=lambda item: item[1].score,
           reverse=True)
)

```

The highest-scoring candidate's runnable code and output proceed to final spider combination.

## End-to-End Integration

The main driver in [`spidercreator.py`](https://github.com/carlosplanchon/spidercreator/blob/main/spidercreator.py) orchestrates the complete flow:

```python

# Execute all candidates

CAND_SPIDER_EXEC_RESULTS = execute_cand_spiders(
    CAND_SPIDER_CREATION_RESULTS=CAND_SPIDER_CREATION_RESULTS,
    recordings_data=recordings_data
)

# Run verification on every candidate's execution result

verification_criteria = get_verification_criteria(plan_json=plan_json)
CAND_SPIDER_EXEC_EVAL_RESULT = run_verification_on_cand_spider_exec_results(
    CAND_SPIDER_EXEC_RESULTS=CAND_SPIDER_EXEC_RESULTS,
    extracted_content_on_rec=extracted_content_on_rec,
    verification_criteria=verification_criteria
)

# Pick the top-scoring candidate

if CAND_SPIDER_EXEC_EVAL_RESULT:
    selected_key = list(CAND_SPIDER_EXEC_EVAL_RESULT.keys())[0]
    spider_code_runnable = CAND_SPIDER_EXEC_RESULTS[selected_key].spider_code_runnable
    spider_output = CAND_SPIDER_EXEC_RESULTS[selected_key].spider_code_output_with_local_addresses

```

## Summary

- The **candidate spider execution and verification pipeline** validates AI-generated scraping code before production deployment.
- **Execution** transforms raw candidates into runnable Parsel scripts, maps URLs to local HTML recordings, and runs spiders in isolated HTTP-server environments via `CandSpiderExecutor`.
- **Verification** uses `gpt-4o` to score each candidate's output against criteria extracted from the original plan, returning a 0-100 score.
- The system automatically selects the highest-scoring candidate from `CAND_SPIDER_EXEC_EVAL_RESULT` for final spider combination.

## Frequently Asked Questions

### How does the pipeline handle multiple candidate spiders simultaneously?

The `execute_cand_spiders` function in [`ctxexec/pipeline.py`](https://github.com/carlosplanchon/spidercreator/blob/main/ctxexec/pipeline.py) iterates over `CAND_SPIDER_CREATION_RESULTS` and instantiates a separate `CandSpiderExecutor` for each valid candidate. Each executor manages its own port allocation and HTTP server context, allowing candidates to run in isolation without network conflicts.

### What criteria does the LLM use to verify spider execution?

The verification criteria are extracted from the `"verify"` fields in `plan_json["action_list"]` within [`pipeline/verification_pipeline.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/verification_pipeline.py). The `verify_spider_exec_result` function in [`pipeline/verify_sp_execution.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/verify_sp_execution.py) sends these criteria along with the runnable code, expected content, and actual output to `gpt-4o`, which returns a numerical score and explanation.

### How are URLs mapped to local HTML recordings during execution?

Inside `CandSpiderExecutor.start()` in [`ctxexec/cand_sp_exec.py`](https://github.com/carlosplanchon/spidercreator/blob/main/ctxexec/cand_sp_exec.py), the `map_url_to_exec_context` function maps each URL extracted from the spider code to a corresponding HTML recording from `recordings_data`. It allocates unique ports starting from `INITIAL_PORT` and returns `URL_TO_HTML` and the updated `port_offset` for server initialization.

### What happens if a candidate spider fails to execute?

If a candidate's `spider_code` is `None`, empty, or marked as `False` in `CAND_SPIDER_CREATION_RESULTS`, it is skipped during the execution phase in [`ctxexec/pipeline.py`](https://github.com/carlosplanchon/spidercreator/blob/main/ctxexec/pipeline.py). For candidates that fail during actual execution (e.g., runtime errors), the `CandSpiderExecutor` captures the output regardless, and the verification stage scores the partial or empty output accordingly, typically resulting in a low score that prevents selection.