How to Configure Generated Spiders to Handle Rate Limiting and Captchas in SpiderCreator

You can configure generated spiders for rate limiting and captcha handling by modifying the LLM prompts in pipeline/spider_draft.py to request Scrapy settings and middleware, or by post-processing the combined output in pipeline/sp_combination.py to inject custom_settings and downloader middleware.

spidercreator is an open-source tool that generates Scrapy spider code by feeding recorded browser interactions to a large language model. While the core generation pipeline does not embed built-in throttling or captcha-solving logic, the architecture provides multiple hook points—prompt engineering and code post-processing—to configure generated spiders for production-grade rate limiting and anti-bot handling.

Understanding the SpiderCreator Architecture

The generation flow moves through several discrete pipeline stages. Knowing where each file lives allows you to intervene at the right moment to inject rate-limiting configurations or captcha middleware.

File	Role	Intervention Point
spidercreator.py	Orchestrates recording → draft → candidates → execution → verification → final combination.	After make_scrapy_spider_draft (line 89) you can programmatically append settings to the draft.
pipeline/spider_draft.py	Sends recordings and a mermaid mind-map to the LLM; returns raw Scrapy code.	Modify DRAFT_SCRAPY_SPIDER_CREATION_PROMPT to request throttling and captcha logic.
pipeline/sp_combination.py	Merges multiple candidate spiders into one runnable script.	Wrap the combined spider in a custom_settings block before returning the final code (lines 81‑108).
pipeline/sp_addr_remapping.py	Rewrites URLs to local addresses.	Inject middleware imports here if you maintain a separate middlewares package.

Implementing Rate Limiting in Generated Spiders

Scrapy provides two complementary mechanisms for rate limiting: a fixed DOWNLOAD_DELAY and the adaptive AutoThrottle extension.

Scrapy Settings for Throttling

Insert these keys into the spider’s custom_settings dictionary:

• DOWNLOAD_DELAY: Base delay in seconds between requests to the same domain.
• AUTOTHROTTLE_ENABLED: Enable dynamic adjustment based on server latency.
• AUTOTHROTTLE_START_DELAY: Initial download delay for AutoThrottle.
• AUTOTHROTTLE_MAX_DELAY: Upper bound for the adaptive delay.
• CONCURRENT_REQUESTS and CONCURRENT_REQUESTS_PER_DOMAIN: Cap parallelism to reduce load.

Injecting Settings via LLM Prompts

In pipeline/spider_draft.py, locate the DRAFT_SCRAPY_SPIDER_CREATION_PROMPT variable (around lines 20‑28). Append instructions so the LLM emits the settings automatically:

DRAFT_SCRAPY_SPIDER_CREATION_PROMPT = """
...
- Respect the target website’s rate limits by adding Scrapy settings:
  DOWNLOAD_DELAY = 2
  AUTOTHROTTLE_ENABLED = True
  AUTOTHROTTLE_START_DELAY = 2
  AUTOTHROTTLE_MAX_DELAY = 60
"""

Because the prompt is fed directly to the model in pipeline/spider_draft.py (lines 35‑44), the generated spider will already contain the custom_settings block.

Post-Processing Settings in sp_combination.py

If you need to guarantee that every final spider includes throttling regardless of LLM output, modify pipeline/sp_combination.py. After the extract_first_python_code call (line 81), programmatically inject the settings:

# Inside pipeline/sp_combination.py after extracting code

settings_block = '''
    custom_settings = {
        "DOWNLOAD_DELAY": 2,
        "AUTOTHROTTLE_ENABLED": True,
        "AUTOTHROTTLE_START_DELAY": 2,
        "AUTOTHROTTLE_MAX_DELAY": 60,
    }
'''

# Insert before the first method definition or after the class line

Handling Captchas in Generated Spiders

Captchas require site-specific detection and solving logic. The most robust approach within the Scrapy ecosystem is a downloader middleware that intercepts responses, identifies challenge pages, and coordinates with a third-party solving service.

Downloader Middleware Approach

Create middlewares/captcha.py in your project root. The middleware implements process_response to detect captcha markers (e.g., specific DOM elements or URL patterns) and process_exception for retry logic.

Example CaptchaMiddleware Implementation

# middlewares/captcha.py

import base64
import requests
from scrapy import signals
from scrapy.exceptions import IgnoreRequest

class CaptchaMiddleware:
    @classmethod
    def from_crawler(cls, crawler):
        # Read API key from Scrapy settings (never hard-code secrets)

        return cls(crawler.settings.get("CAPTCHA_API_KEY"))

    def __init__(self, api_key):
        self.api_key = api_key
        self.solver_url = "http://2captcha.com/in.php"  # example endpoint

    def process_response(self, request, response, spider):
        # Detect captcha by checking for specific image or input

        if response.css("img.captcha-img") or "captcha" in response.url.lower():
            spider.logger.info("Captcha detected, sending to solving service...")
            
            # Extract image URL or base64 data

            img_url = response.css("img.captcha-img::attr(src)").get()
            # Solve logic (pseudo-code; adapt to your provider's API)

            solution = self._solve_captcha(img_url)
            
            # Retry request with solved token

            return request.replace(
                meta={**request.meta, "captcha_solution": solution},
                dont_filter=True
            )
        return response

    def _solve_captcha(self, image_url):
        # Implementation depends on 2Captcha, Anti-Captcha, etc.

        # Return the solved token string

        pass

Configuring Middleware in custom_settings

Reference the middleware in the spider’s custom_settings dictionary so Scrapy loads it into the downloader chain:

custom_settings = {
    "DOWNLOADER_MIDDLEWARES": {
        "myproject.middlewares.captcha.CaptchaMiddleware": 543,
    },
    "CAPTCHA_API_KEY": "YOUR_2CAPTCHA_KEY",  # Inject via environment variable

}

To ensure the LLM generates this configuration, add the middleware requirements to the prompt in pipeline/spider_draft.py as described in the rate-limiting section.

End-to-End Configuration Example

Below is a complete, runnable spider that spidercreator could emit after applying the customisations outlined above. This example demonstrates the integration of both rate-limiting and captcha handling in a single custom_settings block:

import scrapy
from myproject.middlewares.captcha import CaptchaMiddleware

class RealEstateSpider(scrapy.Spider):
    name = "real_estate"
    start_urls = ["https://example.com/listings"]
    
    custom_settings = {
        # Rate limiting

        "DOWNLOAD_DELAY": 2,
        "AUTOTHROTTLE_ENABLED": True,
        "AUTOTHROTTLE_START_DELAY": 2,
        "AUTOTHROTTLE_MAX_DELAY": 60,
        "CONCURRENT_REQUESTS": 8,
        "CONCURRENT_REQUESTS_PER_DOMAIN": 4,
        
        # Captcha handling

        "DOWNLOADER_MIDDLEWARES": {
            "myproject.middlewares.captcha.CaptchaMiddleware": 543,
        },
        "CAPTCHA_API_KEY": "YOUR_2CAPTCHA_KEY",
    }

    def parse(self, response):
        # Extraction logic remains unchanged

        yield {"title": response.css("h1::text").get()}

The spider now respects server load through adaptive throttling and can automatically solve captcha challenges via the injected middleware.

Summary

• spidercreator generates spiders via LLM prompts in pipeline/spider_draft.py, but does not include built-in rate limiting or captcha logic.
• Rate limiting is implemented through Scrapy’s custom_settings using DOWNLOAD_DELAY and AUTOTHROTTLE_ENABLED, either by prompting the LLM to include them or by post-processing in pipeline/sp_combination.py.
• Captcha handling requires a custom downloader middleware (e.g., CaptchaMiddleware) that detects challenge pages and interfaces with solving services; configure it via DOWNLOADER_MIDDLEWARES in custom_settings.
• Prompt engineering in pipeline/spider_draft.py is the least invasive method—simply instruct the LLM to emit the required settings and middleware references.
• Post-processing in pipeline/sp_combination.py guarantees compliance when you cannot rely on LLM output consistency.

Frequently Asked Questions

How do I add rate limiting to a spider generated by spidercreator?

You can add rate limiting by modifying the LLM prompt in pipeline/spider_draft.py to request Scrapy settings such as DOWNLOAD_DELAY and AUTOTHROTTLE_ENABLED inside a custom_settings dictionary. Alternatively, post-process the generated code in pipeline/sp_combination.py to programmatically inject these settings after the spider class definition.

Can spidercreator automatically solve captchas during scraping?

No, spidercreator does not include built-in captcha solving. However, you can configure generated spiders to handle captchas by adding a custom downloader middleware (such as CaptchaMiddleware) that detects captcha pages and sends them to a third-party solving service like 2Captcha. Include the middleware in DOWNLOADER_MIDDLEWARES within the spider’s custom_settings.

Where should I configure the captcha API key for a generated spider?

Store the captcha API key in the spider’s custom_settings dictionary under the key CAPTCHA_API_KEY, and read it via the from_crawler class method in your middleware. Never hard-code secrets directly into the generated spider code; instead, inject the value via environment variables or Scrapy’s settings mechanism when deploying the spider.

What is the difference between using DOWNLOAD_DELAY and AUTOTHROTTLE in Scrapy?

DOWNLOAD_DELAY enforces a fixed pause (in seconds) between consecutive requests to the same domain, providing predictable, static throttling. AUTOTHROTTLE_ENABLED activates the AutoThrottle extension, which dynamically adjusts the delay based on the target server’s response time and load, making it ideal for sites with variable performance or strict anti-bot measures.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}