How to Integrate Generated Spiders with the Scrapy Framework: A Complete Guide

You can integrate generated spiders with the Scrapy framework by recording browser navigation actions with SpiderCreator, generating a Scrapy spider draft that inherits from scrapy.Spider and uses Parsel for extraction, and then copying the generated Python class into your Scrapy project's spiders directory.

SpiderCreator is an open-source automation tool that transforms browser recordings into executable web scraping spiders. Understanding the process for integrating generated spiders with the Scrapy framework enables you to leverage AI-generated extraction logic while maintaining compatibility with Scrapy's robust ecosystem of middleware, item pipelines, and extensions.

The SpiderCreator Workflow

SpiderCreator builds spiders in three logical phases. You can halt the process after phase two to obtain a pure Scrapy spider, or continue to phase three for a Playwright-based implementation.

Phase 1: Recording Browser Navigation

The workflow begins by capturing user interactions through Browser-Use. The create_spider() function in main/main.py launches a temporary recorder, executes your defined task, and stores the resulting actions as JSON recordings under recordings/<task_id>/. The utils/recordings.py module handles the serialization and retrieval of these navigation logs.

Phase 2: Generating the Scrapy Draft

The pipeline invokes pipeline/spider_draft.py and its make_scrapy_spider_draft() function to transform recordings into code. This function sends the JSON recordings plus a Mermaid mind-map to an LLM using the DRAFT_SCRAPY_SPIDER_CREATION_PROMPT, which explicitly instructs the model to "Use only Scrapy."

The generated draft is a plain Python class that inherits from scrapy.Spider, defines name and start_urls, and implements parse() methods. It uses Parsel (Selector(text=html)) for XPath and CSS extraction, ensuring compatibility with Scrapy's built-in selector engine.

Phase 3: Optional Playwright Conversion

If you allow the pipeline to continue, pipeline/xpath_builder_planning.py consumes the Scrapy draft to plan XPath extractions, verify execution, and ultimately produce a Playwright-only spider written to results/<task_id>/spider_code.py. To retain the Scrapy version, simply capture the draft output from phase two and halt execution.

Step-by-Step Integration Process

Step 1: Create a Task and Capture Recordings

Invoke the create_spider() function from main/main.py with a descriptive browser task. The function handles recording automatically and returns a task_id for reference.

from main import create_spider

TASK_PROMPT = """
Navigate to {url} homepage.
Extract all product cards showing name, price and link.
Visit each product link and capture the description.
"""

url = "https://tiendainglesa.com.uy/"
browser_use_task = TASK_PROMPT.format(url=url)

# This prints the Scrapy draft and the final Playwright spider.

task_id = create_spider(browser_use_task=browser_use_task)

Recordings are stored in recordings/<task_id>/ as JSON files describing each browser action.

Step 2: Extract the Scrapy Spider Draft

The create_spider() call prints the generated code to stdout. Capture the block labeled --- SCRAPY SPIDER DRAFT ---, which contains the Python class definition. Alternatively, modify spidercreator.py to write this string to a file for easier retrieval.

The draft will resemble this structure:

class TiendaInglesaSpider(scrapy.Spider):
    name = "tienda_inglesa"
    start_urls = ["https://tiendainglesa.com.uy/"]
    
    def parse(self, response):
        for card in response.xpath("//div[contains(@class,'card-product-container')]"):
            yield {
                "name": card.xpath(".//span[contains(@class,'card-product-name')]/text()").get(),
                "price": card.xpath(".//span[contains(@class,'ProductPrice')]/text()").get(),
                "link": card.xpath(".//a/@href").get(),
            }

Step 3: Integrate the Generated Code into Your Scrapy Project

Create a new Scrapy project and insert the generated class into the spiders directory.

# Create a fresh Scrapy project

scrapy startproject myproject

# Save the draft into the spiders directory

cat > myproject/myproject/spiders/generated_spider.py <<'PY'
import scrapy

class TiendaInglesaSpider(scrapy.Spider):
    name = "tienda_inglesa"
    start_urls = ["https://tiendainglesa.com.uy/"]

    def parse(self, response):
        for card in response.xpath("//div[contains(@class,'card-product-container')]"):
            yield {
                "name": card.xpath(".//span[contains(@class,'card-product-name')]/text()").get(),
                "price": card.xpath(".//span[contains(@class,'ProductPrice')]/text()").get(),
                "link": response.urljoin(card.xpath(".//a/@href").get()),
            }
PY

Ensure all imports are present. The generated draft already uses scrapy and parsel, so standard Scrapy installations satisfy these dependencies.

Step 4: Execute the Spider

Run the integrated spider using Scrapy's CLI.

cd myproject
scrapy crawl tienda_inglesa -o output.json

The spider executes using Scrapy's engine, respecting settings in settings.py and processing items through any configured item pipelines.

Key Technical Components

Understanding these core files helps you customize the integration:

• main/main.py: Contains the create_spider() entry point that coordinates recording, draft generation, and optional Playwright conversion.
• pipeline/spider_draft.py: Implements make_scrapy_spider_draft(), which constructs the LLM prompt and parses the generated Scrapy source code.
• pipeline/xpath_builder_planning.py: Processes the Scrapy draft to build and verify XPaths; continuing past this stage produces the Playwright implementation.
• utils/recordings.py: Manages JSON serialization of browser actions captured during the recording phase.

Summary

• SpiderCreator automates spider generation by recording browser actions and using LLMs to produce Scrapy-compatible code.
• The create_spider() function in main/main.py orchestrates the workflow, storing recordings in recordings/<task_id>/.
• The make_scrapy_spider_draft() function in pipeline/spider_draft.py generates a Python class inheriting from scrapy.Spider that uses Parsel for data extraction.
• You integrate the generated draft by copying the Python class into your Scrapy project's spiders directory and running scrapy crawl.
• The pipeline can optionally continue to produce a Playwright-based spider via pipeline/xpath_builder_planning.py, but the Scrapy draft remains available for immediate use.

Frequently Asked Questions

What makes the generated spider compatible with Scrapy?

The generated spider inherits from scrapy.Spider and implements the standard parse() method signature. It uses Parsel selectors (via Selector(text=html)) for XPath and CSS extraction, which is the same engine Scrapy uses internally. The DRAFT_SCRAPY_SPIDER_CREATION_PROMPT explicitly instructs the LLM to generate code compatible with the Scrapy framework only.

Can I use the generated spider without Playwright?

Yes. The Scrapy spider draft is generated in phase 2 of the pipeline before any Playwright-specific code is introduced. You can capture the draft output from make_scrapy_spider_draft() in pipeline/spider_draft.py and use it immediately in any Scrapy project without installing Playwright or running the subsequent pipeline stages.

Where are the navigation recordings stored?

Browser navigation recordings are stored as JSON files in the recordings/<task_id>/ directory, where <task_id> is the unique identifier returned by the create_spider() function. The utils/recordings.py module handles the serialization and deserialization of these action logs, which serve as the input for the LLM-based spider generation.

How do I customize the spider after generation?

After copying the generated draft into your Scrapy project, you can modify the Python class like any standard Scrapy spider. Common customizations include adding custom_settings for project-specific configurations, implementing additional parse methods for pagination, adding item loaders in items.py, or connecting the spider to item pipelines for data validation and storage.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}