How Scrapling Bypasses Cloudflare Turnstile Using StealthyFetcher

Scrapling's StealthyFetcher automatically detects and solves Cloudflare Turnstile challenges by launching a stealth Playwright browser session that analyzes challenge types, simulates human-like clicks, and waits for network stability before returning the final page response.

Scrapling is an open-source web scraping library that provides advanced browser automation capabilities. When you need to bypass Cloudflare Turnstile using StealthyFetcher, the library orchestrates a sophisticated detection and solving pipeline that handles both non-interactive waiting challenges and interactive click-based verification seamlessly.

How StealthyFetcher Detects and Solves Cloudflare Challenges

The Entry Point: StealthyFetcher.fetch and the solve_cloudflare Flag

The bypass mechanism starts in scrapling/fetchers/stealth_chrome.py, where StealthyFetcher.fetch accepts a solve_cloudflare=True parameter. This flag is forwarded to StealthySession in scrapling/engines/_browsers/_stealth.py, triggering the automated solver pipeline immediately after page navigation.

Inside StealthySession.fetch, the flag is read from validated parameters (params.solve_cloudflare). If true, _cloudflare_solver(page) is invoked after the initial navigation to handle any detected challenges.

Challenge Detection with Regex Pattern Matching

Before solving, the system must identify the challenge type. The _detect_cloudflare helper uses the __CF_PATTERN__ regex defined in _stealth.py to scan page content and classify challenges as "non-interactive", "embedded", or "interactive". This classification determines the subsequent solving strategy.

The detection logic examines the page content and iframe structure to distinguish between simple waiting pages and complex Turnstile widgets that require user interaction.

The Solving Algorithm: From Network Idle to Click Simulation

The _cloudflare_solver method implements a deterministic resolution flow:

1. Network Stabilization: Waits for network idle using _wait_for_networkidle(page, timeout=5000) to ensure the challenge fully loads.
2. Type Determination: Calls _detect_cloudflare to identify the specific challenge variant.
3. Non-Interactive Handling: For "Just a moment..." pages, loops with page.wait_for_timeout(1000) until the challenge title disappears.
4. Interactive/Embedded Resolution: Locates the Turnstile iframe using page.frame(url=__CF_PATTERN__), calculates a random offset within its bounding box, and simulates a human-like click.
5. Retry Logic: If the challenge persists after maximum attempts, the solver recurses via return self._cloudflare_solver(page).
6. Final Validation: Ensures page stability with _wait_for_page_stability before returning the response.

All steps are mirrored in the asynchronous version (AsyncStealthySession._cloudflare_solver) with await-ed Playwright calls, ensuring identical solving capability for async fetches.

Implementation Details of the Cloudflare Solver

The solver operates through specific method calls in scrapling/engines/_browsers/_stealth.py:

• _wait_for_networkidle: Ensures all network activity ceases before interaction, preventing premature clicks on loading elements.
• _detect_cloudflare: Analyzes the DOM for Cloudflare-specific markers and iframe patterns.
• Iframe Targeting: Uses page.frame(url=__CF_PATTERN__) to isolate the Turnstile widget from the parent page context.
• Human Simulation: Calculates random coordinates within the challenge bounding box to avoid detection as automated input.

Code Examples

Synchronous Fetch with Cloudflare Bypass

from scrapling.fetchers.stealth_chrome import StealthyFetcher

# Automatically solves Cloudflare Turnstile challenges

response = StealthyFetcher.fetch(
    "https://protected-site.example.com",
    solve_cloudflare=True,          # Enable the automated solver

    headless=False,                 # Optional: visualize the solving process

    timeout=30000,                  # Request timeout in milliseconds

)

print(response.status)   # 200

print(response.text)     # HTML content after challenge resolution

Asynchronous Fetch with Cloudflare Bypass

import asyncio
from scrapling.fetchers.stealth_chrome import StealthyFetcher

async def main():
    response = await StealthyFetcher.async_fetch(
        "https://protected-site.example.com",
        solve_cloudflare=True,
        headless=True,
        timeout=30000,
    )
    print(response.status)
    print(response.text)

asyncio.run(main())

Custom Page Actions After Challenge Resolution

def scroll_page(page):
    # Execute custom automation after Cloudflare is solved

    page.evaluate("window.scrollTo(0, document.body.scrollHeight)")

response = StealthyFetcher.fetch(
    "https://protected-site.example.com",
    solve_cloudflare=True,
    page_action=scroll_page,  # Runs after challenge resolution

)

Key Source Files and Architecture

File	Role in Cloudflare Bypass
scrapling/fetchers/stealth_chrome.py	Defines StealthyFetcher and forwards the solve_cloudflare flag to the session layer.
scrapling/engines/_browsers/_stealth.py	Implements StealthySession and AsyncStealthySession, containing the core _cloudflare_solver, _detect_cloudflare, and challenge interaction logic.
scrapling/engines/_browsers/_validators.py	Validates the solve_cloudflare parameter and other session configuration options.
scrapling/engines/toolbelt/convertor.py	Constructs the final Response object returned after successful challenge resolution.

These files together give Scrapling the ability to detect, interact with, and solve Cloudflare Turnstile challenges automatically, providing a transparent, "stealthy" fetch experience for end users.

Summary

• Scrapling bypasses Cloudflare Turnstile through the StealthyFetcher class by launching a stealth Playwright browser session.
• The solve_cloudflare=True parameter triggers an automated pipeline that detects challenge types using regex pattern matching against __CF_PATTERN__.
• The solver handles non-interactive challenges by waiting for page redirects and interactive/embedded challenges by simulating human-like clicks within Turnstile iframes.
• Implementation resides primarily in scrapling/engines/_browsers/_stealth.py, with entry points in scrapling/fetchers/stealth_chrome.py.
• Both synchronous and asynchronous fetch methods support Cloudflare bypass with identical configuration parameters.

Frequently Asked Questions

What types of Cloudflare challenges can StealthyFetcher solve?

Scrapling's StealthyFetcher can solve three types of Cloudflare challenges: non-interactive (the "Just a moment..." waiting page), embedded Turnstile widgets, and interactive challenges requiring explicit user verification. The solver automatically detects the challenge type using the __CF_PATTERN__ regex and applies the appropriate resolution strategy for each variant.

Is headless mode supported when bypassing Cloudflare Turnstile?

Yes, headless mode is fully supported, though running with headless=False can be useful for debugging. The Cloudflare solver operates entirely within the Playwright browser context, so it functions regardless of whether the browser window is visible. However, some highly protected sites may employ additional bot detection that correlates with headless indicators, so visible mode occasionally provides better success rates.

How does the retry mechanism work if the challenge fails initially?

The _cloudflare_solver method implements recursive retry logic. If the challenge persists after the maximum number of attempts (clicks or waits), the solver calls itself recursively via return self._cloudflare_solver(page). This creates a robust retry loop that continues until the challenge clears or the overall request timeout is reached, ensuring high reliability against intermittent network delays or slow challenge rotations.

Can I execute custom actions after Cloudflare is solved but before returning the response?

Yes, the page_action parameter accepts a callable function that executes immediately after the Cloudflare challenge is resolved. This function receives the Playwright page object as its argument, allowing you to scroll, click additional elements, or execute JavaScript before Scrapling constructs and returns the final Response object. This is implemented in the session's fetch logic after the _cloudflare_solver completes successfully.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}