# How to Configure Generated Spiders to Handle Rate Limiting and Captchas in SpiderCreator

> Configure generated spiders in SpiderCreator to handle rate limiting and captchas by modifying LLM prompts or post-processing Scrapy output. Learn how to integrate custom settings and middleware.

- Repository: [Carlos A. Planchón/spidercreator](https://github.com/carlosplanchon/spidercreator)
- Tags: how-to-guide
- Published: 2026-02-26

---

**You can configure generated spiders for rate limiting and captcha handling by modifying the LLM prompts in [`pipeline/spider_draft.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/spider_draft.py) to request Scrapy settings and middleware, or by post-processing the combined output in [`pipeline/sp_combination.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/sp_combination.py) to inject `custom_settings` and downloader middleware.**

`spidercreator` is an open-source tool that generates Scrapy spider code by feeding recorded browser interactions to a large language model. While the core generation pipeline does not embed built-in throttling or captcha-solving logic, the architecture provides multiple hook points—prompt engineering and code post-processing—to configure generated spiders for production-grade rate limiting and anti-bot handling.

## Understanding the SpiderCreator Architecture

The generation flow moves through several discrete pipeline stages. Knowing where each file lives allows you to intervene at the right moment to inject rate-limiting configurations or captcha middleware.

| File | Role | Intervention Point |
|------|------|-------------------|
| [`spidercreator.py`](https://github.com/carlosplanchon/spidercreator/blob/main/spidercreator.py) | Orchestrates recording → draft → candidates → execution → verification → final combination. | After `make_scrapy_spider_draft` (line 89) you can programmatically append settings to the draft. |
| [`pipeline/spider_draft.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/spider_draft.py) | Sends recordings and a mermaid mind-map to the LLM; returns raw Scrapy code. | Modify `DRAFT_SCRAPY_SPIDER_CREATION_PROMPT` to request throttling and captcha logic. |
| [`pipeline/sp_combination.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/sp_combination.py) | Merges multiple candidate spiders into one runnable script. | Wrap the combined spider in a `custom_settings` block before returning the final code (lines 81‑108). |
| [`pipeline/sp_addr_remapping.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/sp_addr_remapping.py) | Rewrites URLs to local addresses. | Inject middleware imports here if you maintain a separate `middlewares` package. |

## Implementing Rate Limiting in Generated Spiders

Scrapy provides two complementary mechanisms for rate limiting: a fixed `DOWNLOAD_DELAY` and the adaptive **AutoThrottle** extension.

### Scrapy Settings for Throttling

Insert these keys into the spider’s `custom_settings` dictionary:

- `DOWNLOAD_DELAY`: Base delay in seconds between requests to the same domain.
- `AUTOTHROTTLE_ENABLED`: Enable dynamic adjustment based on server latency.
- `AUTOTHROTTLE_START_DELAY`: Initial download delay for AutoThrottle.
- `AUTOTHROTTLE_MAX_DELAY`: Upper bound for the adaptive delay.
- `CONCURRENT_REQUESTS` and `CONCURRENT_REQUESTS_PER_DOMAIN`: Cap parallelism to reduce load.

### Injecting Settings via LLM Prompts

In [`pipeline/spider_draft.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/spider_draft.py), locate the `DRAFT_SCRAPY_SPIDER_CREATION_PROMPT` variable (around lines 20‑28). Append instructions so the LLM emits the settings automatically:

```python
DRAFT_SCRAPY_SPIDER_CREATION_PROMPT = """
...
- Respect the target website’s rate limits by adding Scrapy settings:
  DOWNLOAD_DELAY = 2
  AUTOTHROTTLE_ENABLED = True
  AUTOTHROTTLE_START_DELAY = 2
  AUTOTHROTTLE_MAX_DELAY = 60
"""

```

Because the prompt is fed directly to the model in [`pipeline/spider_draft.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/spider_draft.py) (lines 35‑44), the generated spider will already contain the `custom_settings` block.

### Post-Processing Settings in sp_combination.py

If you need to guarantee that every final spider includes throttling regardless of LLM output, modify [`pipeline/sp_combination.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/sp_combination.py). After the `extract_first_python_code` call (line 81), programmatically inject the settings:

```python

# Inside pipeline/sp_combination.py after extracting code

settings_block = '''
    custom_settings = {
        "DOWNLOAD_DELAY": 2,
        "AUTOTHROTTLE_ENABLED": True,
        "AUTOTHROTTLE_START_DELAY": 2,
        "AUTOTHROTTLE_MAX_DELAY": 60,
    }
'''

# Insert before the first method definition or after the class line

```

## Handling Captchas in Generated Spiders

Captchas require site-specific detection and solving logic. The most robust approach within the Scrapy ecosystem is a **downloader middleware** that intercepts responses, identifies challenge pages, and coordinates with a third-party solving service.

### Downloader Middleware Approach

Create [`middlewares/captcha.py`](https://github.com/carlosplanchon/spidercreator/blob/main/middlewares/captcha.py) in your project root. The middleware implements `process_response` to detect captcha markers (e.g., specific DOM elements or URL patterns) and `process_exception` for retry logic.

### Example CaptchaMiddleware Implementation

```python

# middlewares/captcha.py

import base64
import requests
from scrapy import signals
from scrapy.exceptions import IgnoreRequest

class CaptchaMiddleware:
    @classmethod
    def from_crawler(cls, crawler):
        # Read API key from Scrapy settings (never hard-code secrets)

        return cls(crawler.settings.get("CAPTCHA_API_KEY"))

    def __init__(self, api_key):
        self.api_key = api_key
        self.solver_url = "http://2captcha.com/in.php"  # example endpoint

    def process_response(self, request, response, spider):
        # Detect captcha by checking for specific image or input

        if response.css("img.captcha-img") or "captcha" in response.url.lower():
            spider.logger.info("Captcha detected, sending to solving service...")
            
            # Extract image URL or base64 data

            img_url = response.css("img.captcha-img::attr(src)").get()
            # Solve logic (pseudo-code; adapt to your provider's API)

            solution = self._solve_captcha(img_url)
            
            # Retry request with solved token

            return request.replace(
                meta={**request.meta, "captcha_solution": solution},
                dont_filter=True
            )
        return response

    def _solve_captcha(self, image_url):
        # Implementation depends on 2Captcha, Anti-Captcha, etc.

        # Return the solved token string

        pass

```

### Configuring Middleware in custom_settings

Reference the middleware in the spider’s `custom_settings` dictionary so Scrapy loads it into the downloader chain:

```python
custom_settings = {
    "DOWNLOADER_MIDDLEWARES": {
        "myproject.middlewares.captcha.CaptchaMiddleware": 543,
    },
    "CAPTCHA_API_KEY": "YOUR_2CAPTCHA_KEY",  # Inject via environment variable

}

```

To ensure the LLM generates this configuration, add the middleware requirements to the prompt in [`pipeline/spider_draft.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/spider_draft.py) as described in the rate-limiting section.

## End-to-End Configuration Example

Below is a complete, runnable spider that `spidercreator` could emit after applying the customisations outlined above. This example demonstrates the integration of both rate-limiting and captcha handling in a single `custom_settings` block:

```python
import scrapy
from myproject.middlewares.captcha import CaptchaMiddleware

class RealEstateSpider(scrapy.Spider):
    name = "real_estate"
    start_urls = ["https://example.com/listings"]
    
    custom_settings = {
        # Rate limiting

        "DOWNLOAD_DELAY": 2,
        "AUTOTHROTTLE_ENABLED": True,
        "AUTOTHROTTLE_START_DELAY": 2,
        "AUTOTHROTTLE_MAX_DELAY": 60,
        "CONCURRENT_REQUESTS": 8,
        "CONCURRENT_REQUESTS_PER_DOMAIN": 4,
        
        # Captcha handling

        "DOWNLOADER_MIDDLEWARES": {
            "myproject.middlewares.captcha.CaptchaMiddleware": 543,
        },
        "CAPTCHA_API_KEY": "YOUR_2CAPTCHA_KEY",
    }

    def parse(self, response):
        # Extraction logic remains unchanged

        yield {"title": response.css("h1::text").get()}

```

The spider now respects server load through adaptive throttling and can automatically solve captcha challenges via the injected middleware.

## Summary

- **`spidercreator`** generates spiders via LLM prompts in [`pipeline/spider_draft.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/spider_draft.py), but does not include built-in rate limiting or captcha logic.
- **Rate limiting** is implemented through Scrapy’s `custom_settings` using `DOWNLOAD_DELAY` and `AUTOTHROTTLE_ENABLED`, either by prompting the LLM to include them or by post-processing in [`pipeline/sp_combination.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/sp_combination.py).
- **Captcha handling** requires a custom downloader middleware (e.g., `CaptchaMiddleware`) that detects challenge pages and interfaces with solving services; configure it via `DOWNLOADER_MIDDLEWARES` in `custom_settings`.
- **Prompt engineering** in [`pipeline/spider_draft.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/spider_draft.py) is the least invasive method—simply instruct the LLM to emit the required settings and middleware references.
- **Post-processing** in [`pipeline/sp_combination.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/sp_combination.py) guarantees compliance when you cannot rely on LLM output consistency.

## Frequently Asked Questions

### How do I add rate limiting to a spider generated by spidercreator?

You can add rate limiting by modifying the LLM prompt in [`pipeline/spider_draft.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/spider_draft.py) to request Scrapy settings such as `DOWNLOAD_DELAY` and `AUTOTHROTTLE_ENABLED` inside a `custom_settings` dictionary. Alternatively, post-process the generated code in [`pipeline/sp_combination.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/sp_combination.py) to programmatically inject these settings after the spider class definition.

### Can spidercreator automatically solve captchas during scraping?

No, `spidercreator` does not include built-in captcha solving. However, you can configure generated spiders to handle captchas by adding a custom downloader middleware (such as `CaptchaMiddleware`) that detects captcha pages and sends them to a third-party solving service like 2Captcha. Include the middleware in `DOWNLOADER_MIDDLEWARES` within the spider’s `custom_settings`.

### Where should I configure the captcha API key for a generated spider?

Store the captcha API key in the spider’s `custom_settings` dictionary under the key `CAPTCHA_API_KEY`, and read it via the `from_crawler` class method in your middleware. Never hard-code secrets directly into the generated spider code; instead, inject the value via environment variables or Scrapy’s settings mechanism when deploying the spider.

### What is the difference between using DOWNLOAD_DELAY and AUTOTHROTTLE in Scrapy?

`DOWNLOAD_DELAY` enforces a fixed pause (in seconds) between consecutive requests to the same domain, providing predictable, static throttling. `AUTOTHROTTLE_ENABLED` activates the AutoThrottle extension, which dynamically adjusts the delay based on the target server’s response time and load, making it ideal for sites with variable performance or strict anti-bot measures.