# Configuring Per-Domain Download Delays and Request Throttling in Scrapling

> Learn to configure per-domain download delays and request throttling in Scrapling using download_delay and concurrent_requests_per_domain for efficient web scraping.

- Repository: [Karim shoair/Scrapling](https://github.com/D4Vinci/Scrapling)
- Tags: how-to-guide
- Published: 2026-03-08

---

**Set `download_delay` to pause between requests and `concurrent_requests_per_domain` to limit simultaneous connections per host, letting Scrapling's `CrawlerEngine` enforce throttling automatically via `anyio.CapacityLimiter` and sleep intervals.**

Scrapling provides fine-grained control over request throttling through two orthogonal settings defined on the spider class. Whether you need to crawl politely with delays or cap connections to specific domains, the framework's asynchronous engine handles the enforcement transparently.

## Understanding Scrapling's Throttling Architecture

Scrapling implements request throttling through two independent mechanisms that work together: temporal spacing via download delays and parallelism constraints via per-domain concurrency limits.

### The Two Core Settings

The base `Spider` class in [`scrapling/spiders/spider.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/spiders/spider.py) defines the throttling interface:

- **`download_delay`** (default: `0.0`): Seconds to pause before each request to the same spider, enforced globally for that spider instance. Defined at [line 78](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/spiders/spider.py#L78).
- **`concurrent_requests_per_domain`** (default: `0`): Maximum simultaneous requests to a single domain. When set to `0`, the engine falls back to the global `concurrent_requests` limit. Defined at [lines 75-78](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/spiders/spider.py#L75-L78).

### Where These Settings Live

These class attributes reside in the spider definition, making throttling configuration declarative and reusable across different crawling strategies.

```python
from scrapling.spiders import Spider

class ThrottledSpider(Spider):
    name = "throttled"
    download_delay = 1.0
    concurrent_requests_per_domain = 2

```

## How the Engine Enforces Throttling

The `CrawlerEngine` class in [`scrapling/spiders/engine.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/spiders/engine.py) implements the actual throttling logic using `anyio` primitives for asynchronous concurrency control.

### Per-Domain Concurrency Control

When `concurrent_requests_per_domain` is non-zero, the engine creates a dedicated `anyio.CapacityLimiter` for each domain. This limiter acts as a semaphore, ensuring that at most the configured number of requests run simultaneously against that specific host.

The `_rate_limiter()` method handles this allocation at [lines 71-77](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/spiders/engine.py#L71-L77):

```python

# From engine.py - conceptual implementation

if self.spider.concurrent_requests_per_domain:
    limiter = anyio.CapacityLimiter(self.spider.concurrent_requests_per_domain)
    # Stored per-domain in self._domain_limiters dict

```

If the attribute is `0`, the engine falls back to `self._global_limiter`, which respects the spider's `concurrent_requests` setting.

### Request Pacing with Download Delays

The `_process_request()` coroutine enforces `download_delay` immediately before executing the HTTP fetch. At [lines 90-93](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/spiders/engine.py#L90-L93), the engine checks for a non-zero delay and awaits `anyio.sleep()`:

```python

# From engine.py

if self.spider.download_delay:
    await anyio.sleep(self.spider.download_delay)

```

This sleep occurs **per request**, independent of the concurrency limiter. The combination of delay and concurrency cap creates a "leaky bucket" throttling pattern: requests drip out at fixed intervals while respecting domain-specific parallelism constraints.

### Statistics Tracking

The engine records the effective throttling configuration in the crawl statistics for post-run analysis. At [lines 236-239](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/spiders/engine.py#L236-L239), the `CrawlStats` object captures `download_delay` and `concurrent_requests_per_domain`.

These values are accessible via the result object returned by `spider.start()`:

```python
result = MySpider().start()
print(result.stats.download_delay)  # 1.0

```

## Practical Configuration Examples

### Basic Global Download Delay

Use `download_delay` to implement polite crawling with a fixed pause between requests:

```python
from scrapling.spiders import Spider, Response

class PoliteSpider(Spider):
    name = "polite"
    start_urls = ["https://example.com/"]
    download_delay = 2.0  # Wait 2 seconds between requests

    async def parse(self, response: Response):
        yield {"title": response.css("title::text").get()}

```

Running this spider issues a request, waits 2 seconds, then proceeds to the next URL. The delay is reflected in `result.stats.download_delay`.

### Per-Domain Concurrency Limits

Restrict simultaneous connections to specific domains while allowing aggressive crawling of others:

```python
from scrapling.spiders import Spider, Response

class DomainThrottledSpider(Spider):
    name = "domain_throttled"
    start_urls = [
        "https://site-a.com/",
        "https://site-b.com/",
        "https://site-a.com/about",
    ]
    concurrent_requests_per_domain = 1  # Only 1 request at a time per domain

    download_delay = 0.7  # Additional 700ms pause

    async def parse(self, response: Response):
        for link in response.css("a::attr(href)").getall():
            if link.startswith("http"):
                yield {"url": link}

```

**Result**: `site-a.com` requests are serialized via the per-domain `CapacityLimiter`, while `site-b.com` runs concurrently on its own limiter. Both domains respect the 0.7-second download delay.

### Combining Global and Per-Domain Settings

Mix global concurrency pools with domain-specific caps for complex crawling scenarios:

```python
class MixedSpider(Spider):
    name = "mixed"
    start_urls = ["https://bigsite.com/"] * 10
    concurrent_requests = 8               # Global pool: 8 total concurrent requests

    concurrent_requests_per_domain = 2    # Per-domain cap: max 2 to bigsite.com

    download_delay = 0.3                  # 300ms pause before each request

```

The engine enforces two constraints: no more than 8 requests globally, and no more than 2 simultaneous connections to `bigsite.com`. Each request waits 0.3 seconds before execution, creating a controlled crawl rate.

### Inspecting Throttling Statistics

Access the effective configuration after the crawl completes:

```python
result = MixedSpider().start()
print("Download delay used:", result.stats.download_delay)
print("Per-domain concurrency:", result.stats.concurrent_requests_per_domain)
print("Total requests:", result.stats.requests_count)

```

The `CrawlStats` object stores these values at [lines 55-60 in [`result.py`](https://github.com/D4Vinci/Scrapling/blob/main/result.py)](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/spiders/result.py#L55-L60), allowing you to verify that your throttling settings were applied correctly.

## Summary

- **Set `download_delay`** on your spider class to introduce a fixed pause (in seconds) before each HTTP request, enforced by `anyio.sleep()` in the engine's `_process_request()` method.
- **Set `concurrent_requests_per_domain`** to limit simultaneous connections to individual domains; the engine creates dedicated `anyio.CapacityLimiter` instances per domain in `_rate_limiter()`.
- **Combine both settings** to implement "leaky bucket" throttling: control parallelism with the limiter and pacing with the sleep interval.
- **Inspect `result.stats`** after crawling to verify the effective `download_delay` and `concurrent_requests_per_domain` values recorded by the engine.

## Frequently Asked Questions

### How does Scrapling handle throttling when `concurrent_requests_per_domain` is set to 0?

When `concurrent_requests_per_domain` is `0` (the default), Scrapling disables per-domain limiting and falls back to the global `concurrent_requests` limit using a single `anyio.CapacityLimiter` shared across all domains. This allows the spider to utilize its full concurrency pool regardless of domain distribution.

### Can I set different download delays for different domains in the same spider?

No, the `download_delay` attribute applies globally to the spider instance. If you need domain-specific pacing, you must implement custom logic in your `parse()` method using `anyio.sleep()` conditionally based on `response.url`, or subclass the engine to override `_process_request()` with domain-aware delay logic.

### What happens if I set both `download_delay` and `concurrent_requests_per_domain`?

The engine applies both constraints independently. The `CapacityLimiter` controls how many requests can be active simultaneously per domain, while `anyio.sleep()` enforces the temporal gap before each individual request. This creates a robust throttling mechanism that respects both connection limits and request rates.

### Where can I verify that my throttling settings are actually being applied?

After calling `spider.start()`, inspect the `result.stats` object which contains `download_delay` and `concurrent_requests_per_domain` attributes. These values are captured by the `CrawlerEngine` during initialization and stored in the `CrawlStats` class defined in [`scrapling/spiders/result.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/spiders/result.py), confirming your configuration was loaded correctly.