how-to-guide

How to Export and Run Generated Spider Code as a Standalone Script

February 26, 2026 carlosplanchon/spidercreator ↗

Use the make_cand_spider_runnable function in pipeline/make_candsp_runnable.py to convert LLM-generated spider code into a self-contained Python script using only Parsel and the standard library, then execute it with python your_script.py.

SpiderCreator transforms high-level web scraping descriptions into executable Python code through LLM generation. When you need to deploy these spiders independently of the framework, you must export and run the generated spider code as a standalone script. The repository provides a dedicated rewriting pipeline in pipeline/make_candsp_runnable.py that strips Scrapy dependencies and injects a runnable entry point.

The Core Rewriting Mechanism

The conversion process centers on the make_cand_spider_runnable function located in pipeline/make_candsp_runnable.py. This utility takes raw LLM-generated spider code and rewrites it through a structured-output LLM (gpt4o_llm) using the SPIDER_REWRITTING_PROMPT.

The rewrite performs four critical transformations:

  • Removes Scrapy dependencies: Replaces Scrapy selectors with parsel.Selector and eliminates framework-specific imports.
  • Strips urljoin calls: Ensures URLs are handled without external utilities.
  • Adds realistic User-Agent headers: Injects headers to prevent blocking during execution.
  • Appends a main guard: Adds if __name__ == "__main__": followed by a call to the spider's run() function.

After the LLM returns markdown-formatted code, the helper extract_first_python_code from utils/utils.py parses the response and extracts the first Python code block, returning a clean string ready for execution.

Step-by-Step Export Process

Follow these steps to convert your generated spider into a standalone .py file.

1. Generate the Raw Spider Code

Obtain your initial spider code through the SpiderCreator UI or programmatic API. This raw code typically contains high-level extraction logic but depends on the full framework ecosystem.

2. Rewrite to Standalone Format

Pass the raw code to the conversion function. The following example demonstrates the minimal programmatic export:

from pipeline.make_candsp_runnable import make_cand_spider_runnable

# `raw_spider_code` is the LLM-generated snippet obtained earlier

raw_spider_code = """

# ... your original spider description ...

"""

# Convert to a runnable script

runnable_code = make_cand_spider_runnable(raw_spider_code)

# Persist to a file

output_path = "generated_spider.py"
with open(output_path, "w", encoding="utf-8") as f:
    f.write(runnable_code)

print(f"Spider saved to {output_path}. Run with `python {output_path}`")

3. Save and Execute

The returned string is a complete, runnable script. Save it to any location and execute:

python generated_spider.py

The script will crawl the target URLs and print extracted fields to STDOUT. No additional framework code is required because the rewrite guarantees zero external Scrapy dependencies—only the standard library plus parsel.

Alternative: High-Level Execution with CandSpiderExecutor

For scenarios requiring local HTTP server provisioning or automated URL mapping, use the CandSpiderExecutor class in ctxexec/cand_sp_exec.py. This high-level orchestrator handles the rewriting, sets up temporary servers for recorded HTML, and manages execution context.

from ctxexec.cand_sp_exec import CandSpiderExecutor

# Load recordings (captured website HTML) – see `utils/recordings.py` for format

recordings = [{"url": "http://example.com", "website_html": "<html>...</html>"}]

# Raw spider generated by the LLM

raw_spider = """

# ... original spider description ...

"""

executor = CandSpiderExecutor(
    spider_code=raw_spider,
    recordings_data=recordings,
)

# Execute the full pipeline: rewrite, serve, crawl

executor.start()

# Access the runnable script directly if you only need the export

standalone_script = executor.spider_code_runnable
with open("standalone_spider.py", "w") as f:
    f.write(standalone_script)

What Makes the Script Standalone

The output of make_cand_spider_runnable is engineered to run in any standard Python environment:

  • Pure Parsel architecture: Uses parsel.Selector for HTML parsing and requests (via Parsel) for network calls instead of Scrapy’s asynchronous engine.
  • Self-contained entry point: The script ends with an if __name__ == "__main__": block that instantiates and runs the spider class.
  • No framework coupling: The code does not import from spidercreator internals, making it portable to other projects.
  • Console output: Each extracted field prints directly to STDOUT for immediate visibility or piping to other tools.

Summary

  • Primary conversion function: make_cand_spider_runnable in pipeline/make_candsp_runnable.py rewrites LLM-generated code into executable scripts.
  • Code extraction: extract_first_python_code in utils/utils.py parses markdown responses to isolate the Python block.
  • Zero Scrapy dependency: The final script uses only parsel, standard library modules, and includes its own main guard.
  • Execution: Save the output to a .py file and run with python filename.py without installing the full SpiderCreator framework.

Frequently Asked Questions

Do standalone exported scripts require Scrapy?

No. The make_cand_spider_runnable function specifically removes all Scrapy dependencies, replacing them with parsel.Selector and standard library HTTP handling. The resulting script runs independently of any web scraping framework.

Where is the if __name__ == "__main__": entry point added?

According to the source code in pipeline/make_candsp_runnable.py, the LLM rewrite prompt instructs the model to append this guard block at the end of the script, followed by a call to the spider’s run() method. This creates a self-contained entry point for direct execution.

Can I export spiders without using CandSpiderExecutor?

Yes. While CandSpiderExecutor in ctxexec/cand_sp_exec.py provides convenient orchestration for testing against local servers, you can call make_cand_spider_runnable directly to obtain the runnable code string and save it manually to disk.

What dependencies are required to run the exported script?

The standalone script requires only the packages listed in the project’s requirements.txt: primarily parsel for HTML parsing, attrs and pydantic for data modeling, and standard library modules. No installation of the full SpiderCreator package is necessary for execution.

Have a question about this repo?

These articles cover the highlights, but your codebase questions are specific. Give your agent direct access to the source. Share this with your agent to get started:

Share the following with your agent to get started:
curl -s "https://instagit.com/install.md"
Works with
Claude Codex Cursor VS Code OpenClaw Any MCP Client

Maintain an open-source project? Get it listed too →