Configuring Per-Domain Download Delays and Request Throttling in Scrapling

Set download_delay to pause between requests and concurrent_requests_per_domain to limit simultaneous connections per host, letting Scrapling's CrawlerEngine enforce throttling automatically via anyio.CapacityLimiter and sleep intervals.

Scrapling provides fine-grained control over request throttling through two orthogonal settings defined on the spider class. Whether you need to crawl politely with delays or cap connections to specific domains, the framework's asynchronous engine handles the enforcement transparently.

Understanding Scrapling's Throttling Architecture

Scrapling implements request throttling through two independent mechanisms that work together: temporal spacing via download delays and parallelism constraints via per-domain concurrency limits.

The Two Core Settings

The base Spider class in scrapling/spiders/spider.py defines the throttling interface:

• download_delay (default: 0.0): Seconds to pause before each request to the same spider, enforced globally for that spider instance. Defined at line 78.
• concurrent_requests_per_domain (default: 0): Maximum simultaneous requests to a single domain. When set to 0, the engine falls back to the global concurrent_requests limit. Defined at lines 75-78.

Where These Settings Live

These class attributes reside in the spider definition, making throttling configuration declarative and reusable across different crawling strategies.

from scrapling.spiders import Spider

class ThrottledSpider(Spider):
    name = "throttled"
    download_delay = 1.0
    concurrent_requests_per_domain = 2

How the Engine Enforces Throttling

The CrawlerEngine class in scrapling/spiders/engine.py implements the actual throttling logic using anyio primitives for asynchronous concurrency control.

Per-Domain Concurrency Control

When concurrent_requests_per_domain is non-zero, the engine creates a dedicated anyio.CapacityLimiter for each domain. This limiter acts as a semaphore, ensuring that at most the configured number of requests run simultaneously against that specific host.

The _rate_limiter() method handles this allocation at lines 71-77:

# From engine.py - conceptual implementation

if self.spider.concurrent_requests_per_domain:
    limiter = anyio.CapacityLimiter(self.spider.concurrent_requests_per_domain)
    # Stored per-domain in self._domain_limiters dict

If the attribute is 0, the engine falls back to self._global_limiter, which respects the spider's concurrent_requests setting.

Request Pacing with Download Delays

The _process_request() coroutine enforces download_delay immediately before executing the HTTP fetch. At lines 90-93, the engine checks for a non-zero delay and awaits anyio.sleep():

# From engine.py

if self.spider.download_delay:
    await anyio.sleep(self.spider.download_delay)

This sleep occurs per request, independent of the concurrency limiter. The combination of delay and concurrency cap creates a "leaky bucket" throttling pattern: requests drip out at fixed intervals while respecting domain-specific parallelism constraints.

Statistics Tracking

The engine records the effective throttling configuration in the crawl statistics for post-run analysis. At lines 236-239, the CrawlStats object captures download_delay and concurrent_requests_per_domain.

These values are accessible via the result object returned by spider.start():

result = MySpider().start()
print(result.stats.download_delay)  # 1.0

Practical Configuration Examples

Basic Global Download Delay

Use download_delay to implement polite crawling with a fixed pause between requests:

from scrapling.spiders import Spider, Response

class PoliteSpider(Spider):
    name = "polite"
    start_urls = ["https://example.com/"]
    download_delay = 2.0  # Wait 2 seconds between requests

    async def parse(self, response: Response):
        yield {"title": response.css("title::text").get()}

Running this spider issues a request, waits 2 seconds, then proceeds to the next URL. The delay is reflected in result.stats.download_delay.

Per-Domain Concurrency Limits

Restrict simultaneous connections to specific domains while allowing aggressive crawling of others:

from scrapling.spiders import Spider, Response

class DomainThrottledSpider(Spider):
    name = "domain_throttled"
    start_urls = [
        "https://site-a.com/",
        "https://site-b.com/",
        "https://site-a.com/about",
    ]
    concurrent_requests_per_domain = 1  # Only 1 request at a time per domain

    download_delay = 0.7  # Additional 700ms pause

    async def parse(self, response: Response):
        for link in response.css("a::attr(href)").getall():
            if link.startswith("http"):
                yield {"url": link}

Result: site-a.com requests are serialized via the per-domain CapacityLimiter, while site-b.com runs concurrently on its own limiter. Both domains respect the 0.7-second download delay.

Combining Global and Per-Domain Settings

Mix global concurrency pools with domain-specific caps for complex crawling scenarios:

class MixedSpider(Spider):
    name = "mixed"
    start_urls = ["https://bigsite.com/"] * 10
    concurrent_requests = 8               # Global pool: 8 total concurrent requests

    concurrent_requests_per_domain = 2    # Per-domain cap: max 2 to bigsite.com

    download_delay = 0.3                  # 300ms pause before each request

The engine enforces two constraints: no more than 8 requests globally, and no more than 2 simultaneous connections to bigsite.com. Each request waits 0.3 seconds before execution, creating a controlled crawl rate.

Inspecting Throttling Statistics

Access the effective configuration after the crawl completes:

result = MixedSpider().start()
print("Download delay used:", result.stats.download_delay)
print("Per-domain concurrency:", result.stats.concurrent_requests_per_domain)
print("Total requests:", result.stats.requests_count)

The CrawlStats object stores these values at [lines 55-60 in result.py](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/spiders/result.py#L55-L60), allowing you to verify that your throttling settings were applied correctly.

Summary

• Set download_delay on your spider class to introduce a fixed pause (in seconds) before each HTTP request, enforced by anyio.sleep() in the engine's _process_request() method.
• Set concurrent_requests_per_domain to limit simultaneous connections to individual domains; the engine creates dedicated anyio.CapacityLimiter instances per domain in _rate_limiter().
• Combine both settings to implement "leaky bucket" throttling: control parallelism with the limiter and pacing with the sleep interval.
• Inspect result.stats after crawling to verify the effective download_delay and concurrent_requests_per_domain values recorded by the engine.

Frequently Asked Questions

How does Scrapling handle throttling when concurrent_requests_per_domain is set to 0?

When concurrent_requests_per_domain is 0 (the default), Scrapling disables per-domain limiting and falls back to the global concurrent_requests limit using a single anyio.CapacityLimiter shared across all domains. This allows the spider to utilize its full concurrency pool regardless of domain distribution.

Can I set different download delays for different domains in the same spider?

No, the download_delay attribute applies globally to the spider instance. If you need domain-specific pacing, you must implement custom logic in your parse() method using anyio.sleep() conditionally based on response.url, or subclass the engine to override _process_request() with domain-aware delay logic.

What happens if I set both download_delay and concurrent_requests_per_domain?

The engine applies both constraints independently. The CapacityLimiter controls how many requests can be active simultaneously per domain, while anyio.sleep() enforces the temporal gap before each individual request. This creates a robust throttling mechanism that respects both connection limits and request rates.

Where can I verify that my throttling settings are actually being applied?

After calling spider.start(), inspect the result.stats object which contains download_delay and concurrent_requests_per_domain attributes. These values are captured by the CrawlerEngine during initialization and stored in the CrawlStats class defined in scrapling/spiders/result.py, confirming your configuration was loaded correctly.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}