# How to Export and Run Generated Spider Code as a Standalone Script

> Export and run generated spider code as a standalone script easily. Learn how to convert LLM code into a self-contained Python script using make_cand_spider_runnable.

- Repository: [Carlos A. Planchón/spidercreator](https://github.com/carlosplanchon/spidercreator)
- Tags: how-to-guide
- Published: 2026-02-26

---

**Use the `make_cand_spider_runnable` function in [`pipeline/make_candsp_runnable.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/make_candsp_runnable.py) to convert LLM-generated spider code into a self-contained Python script using only Parsel and the standard library, then execute it with `python your_script.py`.**

SpiderCreator transforms high-level web scraping descriptions into executable Python code through LLM generation. When you need to deploy these spiders independently of the framework, you must export and run the generated spider code as a standalone script. The repository provides a dedicated rewriting pipeline in [`pipeline/make_candsp_runnable.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/make_candsp_runnable.py) that strips Scrapy dependencies and injects a runnable entry point.

## The Core Rewriting Mechanism

The conversion process centers on the **`make_cand_spider_runnable`** function located in **[`pipeline/make_candsp_runnable.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/make_candsp_runnable.py)**. This utility takes raw LLM-generated spider code and rewrites it through a structured-output LLM (`gpt4o_llm`) using the **`SPIDER_REWRITTING_PROMPT`**.

The rewrite performs four critical transformations:

- **Removes Scrapy dependencies**: Replaces Scrapy selectors with `parsel.Selector` and eliminates framework-specific imports.
- **Strips `urljoin` calls**: Ensures URLs are handled without external utilities.
- **Adds realistic User-Agent headers**: Injects headers to prevent blocking during execution.
- **Appends a `main` guard**: Adds `if __name__ == "__main__":` followed by a call to the spider's `run()` function.

After the LLM returns markdown-formatted code, the helper **`extract_first_python_code`** from **[`utils/utils.py`](https://github.com/carlosplanchon/spidercreator/blob/main/utils/utils.py)** parses the response and extracts the first Python code block, returning a clean string ready for execution.

## Step-by-Step Export Process

Follow these steps to convert your generated spider into a standalone `.py` file.

### 1. Generate the Raw Spider Code

Obtain your initial spider code through the SpiderCreator UI or programmatic API. This raw code typically contains high-level extraction logic but depends on the full framework ecosystem.

### 2. Rewrite to Standalone Format

Pass the raw code to the conversion function. The following example demonstrates the minimal programmatic export:

```python
from pipeline.make_candsp_runnable import make_cand_spider_runnable

# `raw_spider_code` is the LLM-generated snippet obtained earlier

raw_spider_code = """

# ... your original spider description ...

"""

# Convert to a runnable script

runnable_code = make_cand_spider_runnable(raw_spider_code)

# Persist to a file

output_path = "generated_spider.py"
with open(output_path, "w", encoding="utf-8") as f:
    f.write(runnable_code)

print(f"Spider saved to {output_path}. Run with `python {output_path}`")

```

### 3. Save and Execute

The returned string is a complete, runnable script. Save it to any location and execute:

```bash
python generated_spider.py

```

The script will crawl the target URLs and print extracted fields to STDOUT. No additional framework code is required because the rewrite guarantees **zero external Scrapy dependencies**—only the standard library plus `parsel`.

## Alternative: High-Level Execution with CandSpiderExecutor

For scenarios requiring local HTTP server provisioning or automated URL mapping, use the **`CandSpiderExecutor`** class in **[`ctxexec/cand_sp_exec.py`](https://github.com/carlosplanchon/spidercreator/blob/main/ctxexec/cand_sp_exec.py)**. This high-level orchestrator handles the rewriting, sets up temporary servers for recorded HTML, and manages execution context.

```python
from ctxexec.cand_sp_exec import CandSpiderExecutor

# Load recordings (captured website HTML) – see `utils/recordings.py` for format

recordings = [{"url": "http://example.com", "website_html": "<html>...</html>"}]

# Raw spider generated by the LLM

raw_spider = """

# ... original spider description ...

"""

executor = CandSpiderExecutor(
    spider_code=raw_spider,
    recordings_data=recordings,
)

# Execute the full pipeline: rewrite, serve, crawl

executor.start()

# Access the runnable script directly if you only need the export

standalone_script = executor.spider_code_runnable
with open("standalone_spider.py", "w") as f:
    f.write(standalone_script)

```

## What Makes the Script Standalone

The output of `make_cand_spider_runnable` is engineered to run in any standard Python environment:

- **Pure Parsel architecture**: Uses `parsel.Selector` for HTML parsing and `requests` (via Parsel) for network calls instead of Scrapy’s asynchronous engine.
- **Self-contained entry point**: The script ends with an `if __name__ == "__main__":` block that instantiates and runs the spider class.
- **No framework coupling**: The code does not import from `spidercreator` internals, making it portable to other projects.
- **Console output**: Each extracted field prints directly to STDOUT for immediate visibility or piping to other tools.

## Summary

- **Primary conversion function**: `make_cand_spider_runnable` in [`pipeline/make_candsp_runnable.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/make_candsp_runnable.py) rewrites LLM-generated code into executable scripts.
- **Code extraction**: `extract_first_python_code` in [`utils/utils.py`](https://github.com/carlosplanchon/spidercreator/blob/main/utils/utils.py) parses markdown responses to isolate the Python block.
- **Zero Scrapy dependency**: The final script uses only `parsel`, standard library modules, and includes its own `main` guard.
- **Execution**: Save the output to a `.py` file and run with `python filename.py` without installing the full SpiderCreator framework.

## Frequently Asked Questions

### Do standalone exported scripts require Scrapy?

No. The `make_cand_spider_runnable` function specifically removes all Scrapy dependencies, replacing them with `parsel.Selector` and standard library HTTP handling. The resulting script runs independently of any web scraping framework.

### Where is the `if __name__ == "__main__":` entry point added?

According to the source code in [`pipeline/make_candsp_runnable.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/make_candsp_runnable.py), the LLM rewrite prompt instructs the model to append this guard block at the end of the script, followed by a call to the spider’s `run()` method. This creates a self-contained entry point for direct execution.

### Can I export spiders without using CandSpiderExecutor?

Yes. While `CandSpiderExecutor` in [`ctxexec/cand_sp_exec.py`](https://github.com/carlosplanchon/spidercreator/blob/main/ctxexec/cand_sp_exec.py) provides convenient orchestration for testing against local servers, you can call `make_cand_spider_runnable` directly to obtain the runnable code string and save it manually to disk.

### What dependencies are required to run the exported script?

The standalone script requires only the packages listed in the project’s [`requirements.txt`](https://github.com/carlosplanchon/spidercreator/blob/main/requirements.txt): primarily `parsel` for HTML parsing, `attrs` and `pydantic` for data modeling, and standard library modules. No installation of the full SpiderCreator package is necessary for execution.