How to Bypass Anti-Bot Detection with Stealth Settings in Crawl4AI

Enable enable_stealth=True in your BrowserConfig to activate Crawl4AI's built-in stealth mode, which automatically injects playwright-stealth scripts to mask browser fingerprints and evade common bot detection checks.

Crawl4AI provides a robust framework to bypass anti-bot detection mechanisms employed by modern websites and WAFs. By integrating the playwright-stealth library, Crawl4AI can override automated browser signatures—such as navigator.webdriver flags and WebGL fingerprints—allowing your crawlers to mimic genuine user traffic. This capability is essential for accessing sites protected by Cloudflare, DataDome, or proprietary bot mitigation systems.

How Stealth Mode Works in Crawl4AI

When you enable stealth settings, Crawl4AI executes a precise sequence of operations across three architectural layers to mask detection vectors before any navigation occurs.

Configuration Definition

In crawl4ai/async_configs.py, the BrowserConfig class defines the enable_stealth parameter (lines 433-440). Setting this flag to True triggers the stealth pipeline, though it cannot be combined with browser_mode='builtin' due to architectural incompatibility.

Adapter Instantiation

The BrowserManager class in crawl4ai/browser_manager.py evaluates this configuration flag during initialization (lines 622-625). When enabled, it instantiates a StealthAdapter rather than the standard browser adapter, preparing the framework to inject anti-detection scripts prior to page interactions.

Fingerprint Masking Execution

The StealthAdapter in crawl4ai/browser_adapter.py orchestrates the actual obfuscation. It dynamically imports either stealth_async or stealth_sync based on your execution context (lines 58-70), then invokes apply_stealth(page) (lines 73-84) before any console capture or DOM manipulation. This method overwrites critical detection vectors including:

• navigator.webdriver — Set to undefined to eliminate the primary automation flag
• window.chrome — Populated with realistic runtime properties
• Screen dimensions — Randomized to match common desktop viewports
• WebGL vendor and renderer — Masked to appear as standard GPU hardware
• Plugins and mime types — Standardized to mimic genuine Chrome installations

Basic Configuration to Enable Stealth

To activate stealth mode, instantiate BrowserConfig with enable_stealth=True. This configuration works with both headless and headed browser instances.

from crawl4ai import AsyncWebCrawler, BrowserConfig

config = BrowserConfig(
    headless=True,
    enable_stealth=True,
    viewport_width=1920,
    viewport_height=1080
)

async with AsyncWebCrawler(config=config) as crawler:
    result = await crawler.arun("https://example.com")

Verifying Stealth Effectiveness Against Detection Scripts

You can verify that stealth mode actively masks browser fingerprints by injecting JavaScript that inspects detection vectors and returning the results via console capture. The following example demonstrates the behavioral differences between standard and stealth-enabled sessions.

import asyncio, json
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig

async def run_test(use_stealth: bool):
    cfg = BrowserConfig(
        headless=False,
        enable_stealth=use_stealth,
        viewport_width=1280,
        viewport_height=800,
    )
    
    detection_js = """
    (() => {
        const r = {
            webdriver: navigator.webdriver,
            chrome: !!window.chrome,
            pluginsLength: navigator.plugins.length,
            userAgent: navigator.userAgent,
            languages: navigator.languages,
            webglVendor: (() => {
                try {
                    const c = document.createElement('canvas');
                    const gl = c.getContext('webgl') || c.getContext('experimental-webgl');
                    const ext = gl.getExtension('WEBGL_debug_renderer_info');
                    return gl.getParameter(ext.UNMASKED_VENDOR_WEBGL);
                } catch (e) { return 'Error'; }
            })(),
        };
        console.log('DETECTION_RESULTS:', JSON.stringify(r, null, 2));
        return r;
    })();
    """
    
    run_cfg = CrawlerRunConfig(
        js_code=detection_js,
        capture_console_messages=True,
        wait_until="networkidle",
        delay_before_return_html=2.0,
    )
    
    async with AsyncWebCrawler(config=cfg) as crawler:
        result = await crawler.arun("https://bot.sannysoft.com", config=run_cfg)
        
        for msg in result.console_messages or []:
            if "DETECTION_RESULTS:" in msg.get("text", ""):
                json_str = msg["text"].replace("DETECTION_RESULTS:", "").strip()
                data = json.loads(json_str)
                print(f"Stealth={use_stealth}:", json.dumps(data, indent=2))

# Execute comparison

asyncio.run(run_test(False))
asyncio.run(run_test(True))

With enable_stealth=False, the output typically reveals webdriver: true and incomplete chrome object properties. When enable_stealth=True, these automation indicators are masked, presenting the fingerprint of a standard manual Chrome installation.

Bypassing Cloudflare with Stealth Settings

For sites protected by Cloudflare or similar JavaScript challenges, combine stealth mode with strategic delays to allow challenge resolution and script initialization.

from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig

async def fetch_cloudflare():
    cfg = BrowserConfig(
        headless=True,
        enable_stealth=True,
        viewport_width=1920,
        viewport_height=1080,
    )
    
    run_cfg = CrawlerRunConfig(
        wait_until="networkidle",
        delay_before_return_html=3.0,
    )
    
    async with AsyncWebCrawler(config=cfg) as crawler:
        result = await crawler.arun("https://nowsecure.nl", config=run_cfg)
        
        # Detect challenge pages by checking for challenge indicators

        challenge_phrases = ["Checking your browser", "Just a moment", "cf-browser-verification"]
        blocked = any(phrase in (result.html or "") for phrase in challenge_phrases)
        
        return {"blocked": blocked, "status": result.status_code}

Using Stealth with Managed Browser Mode

For persistent sessions requiring cookie retention or profile-based crawling, combine stealth settings with use_managed_browser. This configuration maintains anti-detection capabilities while running a separate Chromium instance managed by Crawl4AI.

from crawl4ai import AsyncWebCrawler, BrowserConfig

cfg = BrowserConfig(
    headless=False,
    enable_stealth=True,
    use_managed_browser=True,
    browser_type="chromium",
)

async with AsyncWebCrawler(config=cfg) as crawler:
    result = await crawler.arun("https://example.com")

Summary

• Enable stealth mode by setting enable_stealth=True in BrowserConfig to mask automated browser fingerprints before page navigation.
• Core architecture involves crawl4ai/async_configs.py for configuration validation, crawl4ai/browser_manager.py for StealthAdapter instantiation, and crawl4ai/browser_adapter.py for script injection.
• Fingerprint masking overwrites navigator.webdriver, populates window.chrome, and randomizes WebGL vendor strings using the playwright-stealth library.
• Best practices include combining stealth with realistic viewport dimensions, delay_before_return_html for script settling, and wait_until="networkidle" for challenge-based protections.
• Compatibility requirement that stealth mode requires standard Chromium/Chrome instances and cannot be used with browser_mode='builtin'.

Frequently Asked Questions

Does Crawl4AI's stealth mode guarantee bypass of all bot detection systems?

While the stealth settings effectively mask common detection vectors like navigator.webdriver and WebGL fingerprints, sophisticated detection services may employ behavioral analysis, IP reputation scoring, or advanced fingerprinting techniques. Stealth mode provides the technical foundation for appearing as genuine traffic, but production deployments should supplement it with human-like interaction patterns and quality proxy rotation.

Can I use stealth mode with the built-in browser mode?

No, according to the validation logic in crawl4ai/async_configs.py, the enable_stealth flag is strictly incompatible with browser_mode='builtin'. You must use standard Chromium or Chrome instances launched by Playwright for the StealthAdapter to successfully inject anti-detection scripts.

What specific browser properties does Crawl4AI modify when stealth is enabled?

The StealthAdapter.apply_stealth method in crawl4ai/browser_adapter.py (lines 73-84) modifies multiple fingerprint vectors: it undefines navigator.webdriver, instantiates a realistic window.chrome object with runtime properties, standardizes the navigator.plugins array length, and masks WebGL vendor and renderer strings to appear as common consumer GPU hardware.

Do I need to install playwright-stealth separately?

Yes, Crawl4AI detects the presence of the playwright-stealth Python package at runtime. If the package is not installed in your environment, the stealth functionality will be unavailable. Install it via pip install playwright-stealth to utilize the enable_stealth configuration option.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}