Scrapling Blocked Request Detection and Retry Logic Customization: A Complete Guide

Scrapling detects blocked requests through the is_blocked() hook in the base Spider class and automatically retries failed requests up to three times by default, with full support for custom detection logic and retry mutations via the retry_blocked_request() hook.

Scrapling is an open-source web scraping framework that handles blocked request detection and retry logic through a hook-oriented architecture in its Spider and CrawlerEngine components. Understanding how Scrapling identifies blocked requests and how you can customize both the detection criteria and retry behavior is essential for building resilient scrapers that handle anti-bot measures gracefully.

How Scrapling Detects Blocked Requests by Default

Scrapling’s default blocked request detection lives in the base Spider class at scrapling/spiders/spider.py. The framework uses a status-code-based approach that covers the most common anti-bot and server error responses.

The Default Blocked Status Codes

In scrapling/spiders/spider.py, the base Spider defines a constant set of status codes that trigger the blocked request logic:

# scrapling/spiders/spider.py

BLOCKED_CODES = {401, 403, 407, 429, 444, 500, 502, 503, 504}

These codes cover authentication failures, forbidden responses, rate limiting, and various server errors that typically indicate blocking or temporary unavailability.

The is_blocked Hook

The detection mechanism centers on the is_blocked() method, which the CrawlerEngine invokes after every response. The default implementation in the base Spider performs a simple membership check:

# scrapling/spiders/spider.py (simplified)

async def is_blocked(self, response) -> bool:
    return response.status in self.BLOCKED_CODES

This method returns True if the response status indicates blocking, triggering the retry logic pipeline in the engine.

Customizing Blocked Request Detection

While status codes provide a solid baseline, modern anti-bot systems often return 200 OK with challenge pages or CAPTCHAs. Scrapling allows you to override is_blocked() to implement content-based detection, header inspection, or JavaScript challenge detection.

Content-Based Detection Implementation

To customize detection, subclass Spider and implement your own is_blocked() logic. You can combine status code checks with body content analysis:

# examples/spiders/custom_block_spider.py

from scrapling.spiders import Spider, Response

class MySpider(Spider):
    name = "custom_block"
    start_urls = ["https://example.com"]
    
    async def is_blocked(self, response: Response) -> bool:
        # Keep the default status-code check

        if response.status in {403, 429, 503}:
            return True
        
        # Additional content-based detection

        body = response.body.decode(errors="ignore").lower()
        if "access denied" in body or "rate limit" in body:
            return True
        
        return False

This approach lets you detect soft blocks that return successful status codes but contain blocking messages in the HTML body.

Understanding the Retry Logic Pipeline

When is_blocked() returns True, the CrawlerEngine in scrapling/spiders/engine.py orchestrates the retry process. Understanding this pipeline helps you customize retry behavior effectively.

How the CrawlerEngine Handles Retries

The engine implements the retry logic in its main processing loop. After detecting a blocked response, it performs the following actions:

1. Increments the stats.blocked_requests_count counter
2. Checks the request's _retry_count against spider.max_blocked_retries (default 3)
3. If under the limit, creates a copy of the request with incremented retry count and lowered priority
4. Clears any proxy or proxies kwargs to allow fresh proxy assignment
5. Calls await spider.retry_blocked_request(retry_request, response) to allow final mutation
6. Re-queues the request with dont_filter=True to bypass deduplication

If the retry limit is exceeded, the engine logs a warning and drops the request.

The Retry Counter and max_blocked_retries

The retry state lives on the request object itself via the _retry_count attribute. This counter is separate from any HTTP-level retry mechanisms and is specific to blocked request handling. You can adjust the retry limit by setting the max_blocked_retries class attribute:

class MySpider(Spider):
    max_blocked_retries = 5  # Increase from default 3

The engine consults this attribute once per blocked response to determine whether to schedule another attempt.

Customizing Retry Behavior

Beyond adjusting the retry count, you can override retry_blocked_request() to mutate the retry request before it returns to the queue. This hook receives the copied request and the original blocked response, allowing you to implement sophisticated retry strategies.

Overriding retry_blocked_request

The base implementation simply returns the request unchanged, but you can override it to switch sessions, modify headers, or implement exponential backoff:

# examples/spiders/custom_retry_spider.py

from scrapling.spiders import Spider, Request, Response, SessionManager
from scrapling.fetchers import AsyncStealthySession, FetcherSession

class MySpider(Spider):
    name = "custom_retry"
    start_urls = ["https://example.com"]
    max_blocked_retries = 5

    def configure_sessions(self, manager: SessionManager) -> None:
        manager.add("default", FetcherSession())
        manager.add("stealth", AsyncStealthySession(block_webrtc=True), lazy=True)

    async def retry_blocked_request(self, request: Request, response: Response) -> Request:
        # After a block, use the stealth session for the retry

        request.sid = "stealth"
        self.logger.info(f"Retrying blocked request {request.url} via stealth session")
        return request

Session Switching on Retry

The example above demonstrates switching from a standard FetcherSession to an AsyncStealthySession after detection. Because the engine clears proxy arguments before calling this hook, the new session's proxy rotator (if configured) will supply a fresh proxy for the retry attempt.

Proxy Handling During Retries

According to the implementation in scrapling/spiders/engine.py, the engine automatically strips proxy and proxies kwargs from the request before re-queuing. This ensures that the ProxyRotator attached to the session provides a fresh proxy for the next attempt, preventing reuse of a potentially flagged proxy:

“On retry, the previous proxy/proxies kwargs are cleared from the request automatically, so the rotator assigns a fresh proxy.” – docs, Blocked Request Handling

Summary

• Scrapling detects blocked requests through the is_blocked() hook in the base Spider class at scrapling/spiders/spider.py, which by default checks against BLOCKED_CODES = {401, 403, 407, 429, 444, 500, 502, 503, 504}.
• The CrawlerEngine in scrapling/spiders/engine.py orchestrates retries, incrementing counters and re-queuing requests up to max_blocked_retries (default 3).
• You can customize detection by overriding is_blocked() to inspect response bodies, headers, or implement CAPTCHA detection.
• You can customize retry behavior by overriding retry_blocked_request() to switch sessions, modify headers, or implement exponential backoff.
• The engine automatically clears proxy arguments on retry, ensuring fresh proxy assignment from the session’s ProxyRotator.

Frequently Asked Questions

How does Scrapling determine if a request is blocked?

Scrapling uses the is_blocked() method defined in scrapling/spiders/spider.py. The default implementation checks if the response status code is in the BLOCKED_CODES set, which includes 401, 403, 407, 429, 444, 500, 502, 503, and 504. You can override this method to add content-based detection or inspect response headers for additional signals.

Can I customize which HTTP status codes trigger a retry?

Yes. While you can override is_blocked() entirely to implement arbitrary logic, you can also reference your own set of status codes within a custom is_blocked() implementation. The default BLOCKED_CODES constant in the base Spider class provides the baseline set, but you are not limited to these values when implementing custom detection logic.

What happens to proxy settings when a request is retried?

According to the implementation in scrapling/spiders/engine.py, the engine automatically clears any proxy or proxies kwargs from the request before re-queuing it for retry. This ensures that the session’s ProxyRotator assigns a fresh proxy for the next attempt, preventing reuse of a proxy that may have been flagged by the target site.

How can I increase the number of retry attempts for blocked requests?

Set the max_blocked_retries class attribute in your Spider subclass. The default value is 3, but you can increase it to any integer value. For example: max_blocked_retries = 5. The CrawlerEngine checks this attribute against the request’s internal _retry_count attribute to determine whether to schedule another attempt or drop the request.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}