# How to Integrate Generated Spiders with the Scrapy Framework: A Complete Guide

> Learn to integrate generated spiders with Scrapy. Use SpiderCreator to record navigation, generate a Scrapy spider draft, and add it to your project for efficient web scraping.

- Repository: [Carlos A. Planchón/spidercreator](https://github.com/carlosplanchon/spidercreator)
- Tags: how-to-guide
- Published: 2026-02-26

---

**You can integrate generated spiders with the Scrapy framework by recording browser navigation actions with SpiderCreator, generating a Scrapy spider draft that inherits from `scrapy.Spider` and uses Parsel for extraction, and then copying the generated Python class into your Scrapy project's spiders directory.**

SpiderCreator is an open-source automation tool that transforms browser recordings into executable web scraping spiders. Understanding the process for integrating generated spiders with the Scrapy framework enables you to leverage AI-generated extraction logic while maintaining compatibility with Scrapy's robust ecosystem of middleware, item pipelines, and extensions.

## The SpiderCreator Workflow

SpiderCreator builds spiders in three logical phases. You can halt the process after phase two to obtain a pure Scrapy spider, or continue to phase three for a Playwright-based implementation.

### Phase 1: Recording Browser Navigation

The workflow begins by capturing user interactions through Browser-Use. The `create_spider()` function in **[`main/main.py`](https://github.com/carlosplanchon/spidercreator/blob/main/main/main.py)** launches a temporary recorder, executes your defined task, and stores the resulting actions as JSON recordings under `recordings/<task_id>/`. The **[`utils/recordings.py`](https://github.com/carlosplanchon/spidercreator/blob/main/utils/recordings.py)** module handles the serialization and retrieval of these navigation logs.

### Phase 2: Generating the Scrapy Draft

The pipeline invokes **[`pipeline/spider_draft.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/spider_draft.py)** and its `make_scrapy_spider_draft()` function to transform recordings into code. This function sends the JSON recordings plus a Mermaid mind-map to an LLM using the `DRAFT_SCRAPY_SPIDER_CREATION_PROMPT`, which explicitly instructs the model to "Use only Scrapy."

The generated draft is a plain Python class that inherits from `scrapy.Spider`, defines `name` and `start_urls`, and implements `parse()` methods. It uses **Parsel** (`Selector(text=html)`) for XPath and CSS extraction, ensuring compatibility with Scrapy's built-in selector engine.

### Phase 3: Optional Playwright Conversion

If you allow the pipeline to continue, **[`pipeline/xpath_builder_planning.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/xpath_builder_planning.py)** consumes the Scrapy draft to plan XPath extractions, verify execution, and ultimately produce a Playwright-only spider written to `results/<task_id>/spider_code.py`. To retain the Scrapy version, simply capture the draft output from phase two and halt execution.

## Step-by-Step Integration Process

### Step 1: Create a Task and Capture Recordings

Invoke the `create_spider()` function from **[`main/main.py`](https://github.com/carlosplanchon/spidercreator/blob/main/main/main.py)** with a descriptive browser task. The function handles recording automatically and returns a `task_id` for reference.

```python
from main import create_spider

TASK_PROMPT = """
Navigate to {url} homepage.
Extract all product cards showing name, price and link.
Visit each product link and capture the description.
"""

url = "https://tiendainglesa.com.uy/"
browser_use_task = TASK_PROMPT.format(url=url)

# This prints the Scrapy draft and the final Playwright spider.

task_id = create_spider(browser_use_task=browser_use_task)

```

Recordings are stored in `recordings/<task_id>/` as JSON files describing each browser action.

### Step 2: Extract the Scrapy Spider Draft

The `create_spider()` call prints the generated code to stdout. Capture the block labeled `--- SCRAPY SPIDER DRAFT ---`, which contains the Python class definition. Alternatively, modify [`spidercreator.py`](https://github.com/carlosplanchon/spidercreator/blob/main/spidercreator.py) to write this string to a file for easier retrieval.

The draft will resemble this structure:

```python
class TiendaInglesaSpider(scrapy.Spider):
    name = "tienda_inglesa"
    start_urls = ["https://tiendainglesa.com.uy/"]
    
    def parse(self, response):
        for card in response.xpath("//div[contains(@class,'card-product-container')]"):
            yield {
                "name": card.xpath(".//span[contains(@class,'card-product-name')]/text()").get(),
                "price": card.xpath(".//span[contains(@class,'ProductPrice')]/text()").get(),
                "link": card.xpath(".//a/@href").get(),
            }

```

### Step 3: Integrate the Generated Code into Your Scrapy Project

Create a new Scrapy project and insert the generated class into the spiders directory.

```bash

# Create a fresh Scrapy project

scrapy startproject myproject

# Save the draft into the spiders directory

cat > myproject/myproject/spiders/generated_spider.py <<'PY'
import scrapy

class TiendaInglesaSpider(scrapy.Spider):
    name = "tienda_inglesa"
    start_urls = ["https://tiendainglesa.com.uy/"]

    def parse(self, response):
        for card in response.xpath("//div[contains(@class,'card-product-container')]"):
            yield {
                "name": card.xpath(".//span[contains(@class,'card-product-name')]/text()").get(),
                "price": card.xpath(".//span[contains(@class,'ProductPrice')]/text()").get(),
                "link": response.urljoin(card.xpath(".//a/@href").get()),
            }
PY

```

Ensure all imports are present. The generated draft already uses `scrapy` and `parsel`, so standard Scrapy installations satisfy these dependencies.

### Step 4: Execute the Spider

Run the integrated spider using Scrapy's CLI.

```bash
cd myproject
scrapy crawl tienda_inglesa -o output.json

```

The spider executes using Scrapy's engine, respecting settings in [`settings.py`](https://github.com/carlosplanchon/spidercreator/blob/main/settings.py) and processing items through any configured item pipelines.

## Key Technical Components

Understanding these core files helps you customize the integration:

- **[`main/main.py`](https://github.com/carlosplanchon/spidercreator/blob/main/main/main.py)**: Contains the `create_spider()` entry point that coordinates recording, draft generation, and optional Playwright conversion.
- **[`pipeline/spider_draft.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/spider_draft.py)**: Implements `make_scrapy_spider_draft()`, which constructs the LLM prompt and parses the generated Scrapy source code.
- **[`pipeline/xpath_builder_planning.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/xpath_builder_planning.py)**: Processes the Scrapy draft to build and verify XPaths; continuing past this stage produces the Playwright implementation.
- **[`utils/recordings.py`](https://github.com/carlosplanchon/spidercreator/blob/main/utils/recordings.py)**: Manages JSON serialization of browser actions captured during the recording phase.

## Summary

- **SpiderCreator** automates spider generation by recording browser actions and using LLMs to produce Scrapy-compatible code.
- The **`create_spider()`** function in [`main/main.py`](https://github.com/carlosplanchon/spidercreator/blob/main/main/main.py) orchestrates the workflow, storing recordings in `recordings/<task_id>/`.
- The **`make_scrapy_spider_draft()`** function in [`pipeline/spider_draft.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/spider_draft.py) generates a Python class inheriting from `scrapy.Spider` that uses **Parsel** for data extraction.
- You integrate the generated draft by copying the Python class into your Scrapy project's spiders directory and running `scrapy crawl`.
- The pipeline can optionally continue to produce a Playwright-based spider via [`pipeline/xpath_builder_planning.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/xpath_builder_planning.py), but the Scrapy draft remains available for immediate use.

## Frequently Asked Questions

### What makes the generated spider compatible with Scrapy?

The generated spider inherits from `scrapy.Spider` and implements the standard `parse()` method signature. It uses **Parsel** selectors (via `Selector(text=html)`) for XPath and CSS extraction, which is the same engine Scrapy uses internally. The `DRAFT_SCRAPY_SPIDER_CREATION_PROMPT` explicitly instructs the LLM to generate code compatible with the Scrapy framework only.

### Can I use the generated spider without Playwright?

Yes. The Scrapy spider draft is generated in **phase 2** of the pipeline before any Playwright-specific code is introduced. You can capture the draft output from `make_scrapy_spider_draft()` in [`pipeline/spider_draft.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/spider_draft.py) and use it immediately in any Scrapy project without installing Playwright or running the subsequent pipeline stages.

### Where are the navigation recordings stored?

Browser navigation recordings are stored as JSON files in the `recordings/<task_id>/` directory, where `<task_id>` is the unique identifier returned by the `create_spider()` function. The **[`utils/recordings.py`](https://github.com/carlosplanchon/spidercreator/blob/main/utils/recordings.py)** module handles the serialization and deserialization of these action logs, which serve as the input for the LLM-based spider generation.

### How do I customize the spider after generation?

After copying the generated draft into your Scrapy project, you can modify the Python class like any standard Scrapy spider. Common customizations include adding `custom_settings` for project-specific configurations, implementing additional `parse` methods for pagination, adding item loaders in [`items.py`](https://github.com/carlosplanchon/spidercreator/blob/main/items.py), or connecting the spider to item pipelines for data validation and storage.