How Scrapling StealthyFetcher Achieves Browser Fingerprint Spoofing

Scrapling's StealthyFetcher masks automated browser traffic by launching real Chromium instances via Playwright, injecting authentic HTTP headers through browserforge, spoofing Google-search referers, and applying anti-detection patches like canvas noise and WebRTC blocking.

Scrapling is an open-source web scraping framework by D4Vinci that provides high-level abstractions over browser automation tools. The Scrapling StealthyFetcher serves as the primary interface for bypassing bot detection systems, wrapping Playwright (via the patchright wrapper) in a simple API that handles fingerprint randomization, proxy rotation, and challenge solving automatically.

Architecture of the StealthyFetcher System

The High-Level Facade

The StealthyFetcher class in scrapling/fetchers/stealth_chrome.py (lines 13-16) acts as a static utility that creates either a StealthySession (synchronous) or AsyncStealthySession (asynchronous) based on your execution context. This design hides the complexity of browser lifecycle management behind a single method signature:

StealthyFetcher.fetch(url, headless=True, stealthy_headers=True, ...)

When invoked, the fetcher builds a selector configuration and enters a managed context with the appropriate session type. The session validates options through StealthConfig (defined in scrapling/engines/_browsers/_validators.py) before launching any browser processes, ensuring mutually exclusive flags are resolved before execution.

Session Management and Browser Context

The core engine resides in scrapling/engines/_browsers/_stealth.py (lines 27-38), where StealthySession and AsyncStealthySession handle the actual browser orchestration. These classes:

• Launch Chromium via sync_playwright or async_playwright
• Connect via Chrome DevTools Protocol (CDP) or launch persistent contexts
• Maintain a pool of reusable pages (max_pages) to avoid context-creation overhead
• Merge generated headers with user-supplied extra_headers
• Execute optional page_action callbacks after navigation

Browser Fingerprint Spoofing Mechanisms

Realistic Header Generation with browserforge

In scrapling/engines/toolbelt/fingerprints.py (lines 66-86), the generate_headers function creates complete browser header sets—including User-Agent, Accept-Encoding, Accept-Language, and Sec-CH-UA headers—using the browserforge library. When stealthy_headers=True (the default), these headers replace Playwright's default fingerprint, which detection systems often flag as automation traffic.

The fetcher merges these generated headers with any provided via extra_headers, with user values taking precedence. This prevents the "default-requests" signature that basic HTTP clients exhibit.

Referer Spoofing and Google Search Simulation

The generate_convincing_referer function in scrapling/engines/toolbelt/fingerprints.py (lines 21-46) extracts the target domain using tld.get_tld and constructs a realistic Google search URL (https://www.google.com/search?q=example). By default, this makes requests appear to originate from organic search results, a pattern common in legitimate browser behavior.

You can disable this behavior by setting google_search=False or override it completely by supplying a custom referer in extra_headers. The logic check occurs in StealthySession.fetch (lines 215-220 in _stealth.py).

Canvas Noise and WebRTC Leak Prevention

The session applies several runtime patches to the Chromium context:

• Canvas noise: When hide_canvas=True, the engine injects JavaScript that adds subtle noise to HTML5 canvas operations, preventing fingerprinting based on rendering consistency
• WebRTC blocking: Setting block_webrtc=True disables WebRTC to prevent local IP address leaks that reveal non-residential networks
• WebGL toggling: Controls WebGL vendor and renderer strings to match the generated User-Agent profile
• Resource blocking: disable_resources=True drops images, fonts, and media files to speed up scraping while maintaining the request fingerprint

Performance and Reliability Features

Page Pooling for Concurrent Requests

Rather than creating a new browser context for every request, StealthySession maintains a page_pool (defined in the __slots__ block) that reuses pages up to the max_pages limit. This pool resets state (cookies, localStorage, sessionStorage) between fetches while preserving the warm browser instance, significantly reducing per-request latency.

Automatic Cloudflare Challenge Solving

When solve_cloudflare=True, the engine detects Cloudflare challenge pages and executes the _cloudflare_solver method (lines 111-186 in _stealth.py). This method:

1. Detects the "Just a moment..." interstitial page
2. Clicks the turnstile verification canvas element
3. Waits for the challenge to clear and the target content to load
4. Proceeds with extraction only after validation passes

This eliminates the need for external CAPTCHA-solving services for basic Cloudflare protections.

Proxy Rotation and Retry Logic

The fetcher accepts a proxy_rotator parameter that supplies fresh proxy configurations for each attempt. If a request fails due to proxy errors, the logic in StealthySession.fetch (lines 22-29 and 69-82) automatically retries with a new proxy from the rotator up to the specified retries limit. This maintains consistent browser fingerprints across different IP addresses, reducing blocking due to IP reputation.

Implementation Examples

Synchronous Stealth Fetching

Fetch a page with full fingerprint spoofing and resource blocking:

from scrapling.fetchers.stealth_chrome import StealthyFetcher

resp = StealthyFetcher.fetch(
    "https://example.com",
    headless=True,
    disable_resources=True,
    hide_canvas=True,
    block_webrtc=True,
    solve_cloudflare=True,
    extra_flags=["--no-sandbox"]
)

print(resp.status)       # 200

print(resp.headers)      # Realistic browser headers

Source: StealthyFetcher.fetch implementation in stealth_chrome.py (lines 13-20).

Asynchronous Usage with Custom Actions

Perform custom JavaScript execution after page load using the async interface:

import asyncio
from scrapling.fetchers.stealth_chrome import AsyncStealthySession

async def scroll_to_bottom(page):
    await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")

async def main():
    async with AsyncStealthySession(
        headless=False,
        extra_headers={"X-Custom": "value"},
        page_action=scroll_to_bottom,
    ) as engine:
        response = await engine.fetch("https://news.ycombinator.com")
        print(response.status, len(response.text))

asyncio.run(main())

Source: Async fetch method in _stealth.py (lines 443-463).

Disabling Automatic Referer Generation

Override the Google-search referer for API endpoints or direct access:

resp = StealthyFetcher.fetch(
    "https://myapi.com/endpoint",
    stealthy_headers=True,
    extra_headers={"referer": "https://myapp.example.com/"},
    google_search=False   # Disable automatic referer generation

)

Source: Referer handling logic in _stealth.py (lines 215-220).

Proxy Rotation with Retries

Route requests through rotating proxies with automatic failover:

resp = StealthyFetcher.fetch(
    "https://blocked-site.com",
    proxy_rotator=my_rotator,   # Scrapling ProxyRotator instance

    retries=5,
    retry_delay=2,
    stealthy_headers=True
)

Source: Proxy handling in StealthySession.fetch (lines 22-29).

Summary

• Scrapling StealthyFetcher wraps Playwright in scrapling/fetchers/stealth_chrome.py to provide a simple, high-level API for stealth browsing.
• Fingerprint spoofing relies on browserforge-generated headers in scrapling/engines/toolbelt/fingerprints.py, Google-search referer simulation, and runtime patches for canvas and WebRTC.
• Page pooling in scrapling/engines/_browsers/_stealth.py reuses browser pages across requests to improve performance while isolating session state.
• Cloudflare bypass is handled natively via _cloudflare_solver (lines 111-186), which automates turnstile clicking without external services.
• Proxy rotation integrates directly into the retry logic, allowing fresh IP attempts while maintaining consistent browser fingerprints.

Frequently Asked Questions

How does Scrapling StealthyFetcher differ from standard Playwright?

Unlike raw Playwright, which uses detectable default flags and consistent fingerprints, Scrapling's implementation generates realistic browser headers via browserforge, spoofs referers as Google search traffic, and applies runtime patches for canvas noise and WebRTC blocking. The StealthySession class manages these configurations automatically while adding page pooling and built-in Cloudflare solving.

Can I use Scrapling StealthyFetcher without headless mode?

Yes. Setting headless=False launches a visible Chromium window, which is useful for debugging or sites that detect headless environments. The fetcher applies the same fingerprint spoofing regardless of visibility mode, though non-headless mode often encounters fewer bot checks due to the presence of a real windowing system.

What causes the "default-requests" fingerprint that StealthyFetcher prevents?

Basic HTTP clients and unpatched Playwright instances send consistent User-Agent strings, ordered headers, and missing browser-specific headers like Sec-CH-UA. Detection systems flag these patterns as automation. StealthyFetcher prevents this by using generate_headers in scrapling/engines/toolbelt/fingerprints.py to produce randomized, authentic Chromium headers that vary between requests.

How does the Cloudflare solver work without external CAPTCHA services?

The _cloudflare_solver method in scrapling/engines/_browsers/_stealth.py detects Cloudflare's "Just a moment..." interstitial and programmatically clicks the turnstile checkbox using Playwright's mouse automation. It then waits for the challenge to clear and the target content to load, handling the JavaScript-based verification entirely within the browser context.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}