What is stealth.js and How MediaCrawler Uses It for Anti-Detection Scraping

stealth.js is a minified JavaScript bundle that masks browser automation fingerprints by injecting anti-detection patches before page navigation, enabling MediaCrawler to scrape Chinese social platforms without triggering bot protection.

MediaCrawler is an open-source scraping framework designed to extract data from platforms like Zhihu, XiaoHongShu, Weibo, Douyin, and Bilibili. At its core lies libs/stealth.min.js, a stealth script derived from the puppeteer-extra stealth plugin that eliminates tell-tale signs of automated browsing such as navigator.webdriver flags and Chrome runtime properties.

What is stealth.js in MediaCrawler?

stealth.js (distributed as libs/stealth.min.js) is a pre-generated JavaScript bundle created by the puppeteer-extra stealth ecosystem. according to the MediaCrawler source code, it contains a collection of "evasions"—code patches that hide the typical fingerprints of an automated Chromium or Playwright session.

Origin and Purpose

The script is not hand-written for MediaCrawler but is a compiled artifact from the widely-used puppeteer-extra-plugin-stealth package. Its primary purpose is to make the browser instance appear as a standard human-driven Chrome installation rather than an automation tool. This is critical because modern anti-bot systems check for properties like navigator.webdriver, window.chrome, and plugin inconsistencies to detect automation.

Key Evasions Covered

When injected, stealth.js patches several detection vectors:

• navigator.webdriver – Removes the webdriver property that Playwright and Selenium set to true
• Chrome runtime – Mocks the window.chrome object to match genuine browser expectations
• Proxy detection – Hides proxy-related headers and WebRTC leaks that reveal non-residential connections
• User agent consistency – Ensures the user agent string aligns with other browser properties like navigator.platform

How MediaCrawler Implements stealth.js

MediaCrawler applies the stealth script using Playwright's add_init_script method, which executes the code before any page navigation begins. This ensures anti-detection patches are active the moment the crawler hits the target site.

Direct Injection in Platform Core Modules

Each platform-specific crawler integrates stealth.js directly within its core initialization logic. In media_platform/zhihu/core.py (lines 100-101), the script is loaded immediately after browser context creation:

# media_platform/zhihu/core.py (excerpt)

if config.ENABLE_CDP_MODE:
    # CDP launch logic omitted...

else:
    chromium = playwright.chromium
    self.browser_context = await self.launch_browser(
        chromium, None, self.user_agent, headless=config.HEADLESS
    )
    # Inject stealth script to hide automation fingerprints

    await self.browser_context.add_init_script(path="libs/stealth.min.js")

This identical pattern appears across other platform modules including media_platform/xhs/core.py (XiaoHongShu), media_platform/weibo/core.py, and implementations for Douyin, Bilibili, and Kuaishou.

CDP Helper Integration

For Chrome DevTools Protocol (CDP) mode, MediaCrawler provides a generic helper in tools/cdp_browser.py (lines 400-407) that wraps the stealth injection:

# tools/cdp_browser.py (excerpt)

async def add_stealth_script(self, script_path: str = "libs/stealth.min.js"):
    """Add anti-detection script to CDP browser context."""
    if self.browser_context and os.path.exists(script_path):
        try:
            await self.browser_context.add_init_script(path=script_path)
            utils.logger.info(
                f"[CDPBrowserManager] Added anti-detection script: {script_path}"
            )
        except Exception as e:
            utils.logger.warning(
                f"[CDPBrowserManager] Failed to add anti-detection script: {e}"
            )

This method provides centralized error handling and logging while achieving the same pre-navigation injection as the platform-specific implementations.

Code Implementation Examples

Standalone Playwright Usage

You can replicate MediaCrawler's anti-detection approach in any Playwright script:

from playwright.async_api import async_playwright

async def launch_with_stealth():
    async with async_playwright() as pw:
        browser = await pw.chromium.launch(headless=False)
        context = await browser.new_context()
        
        # Load the stealth bundle from the repository

        await context.add_init_script(path="libs/stealth.min.js")
        
        page = await context.new_page()
        await page.goto("https://www.zhihu.com")
        
        # The site now sees the browser as a normal user agent

        await page.screenshot(path="zhihu_home.png")
        await browser.close()

# asyncio.run(launch_with_stealth())

CDP Mode Initialization

When operating in CDP mode, the script is applied after establishing the Chrome DevTools connection:

#tools/cdp_browser.py workflow
browser_manager = CDPBrowserManager()
await browser_manager.init()
await browser_manager.add_stealth_script("libs/stealth.min.js")
page = await browser_manager.new_page()

Why Anti-Detection Matters for MediaCrawler

Chinese social platforms employ sophisticated bot detection that examines browser fingerprints, behavioral patterns, and automation flags. Without stealth.js, MediaCrawler sessions would immediately expose navigator.webdriver === true, triggering CAPTCHA challenges or outright IP bans. By injecting the stealth bundle before navigation, MediaCrawler maintains the appearance of legitimate human traffic across all supported platforms, ensuring reliable data extraction from JavaScript-heavy single-page applications.

Summary

• libs/stealth.min.js is the minified anti-detection script derived from puppeteer-extra stealth, located in the repository root.
• Injection method uses add_init_script to execute patches before page loads, implemented in both standard Playwright and CDP modes.
• Platform coverage includes Zhihu, XiaoHongShu, Weibo, Douyin, Bilibili, and Kuaishou, each using the same stealth mechanism.
• Key evasions remove navigator.webdriver, mock Chrome runtime properties, and hide proxy indicators.
• File locations for implementation details are media_platform/zhihu/core.py (lines 100-101) and tools/cdp_browser.py (lines 400-407).

Frequently Asked Questions

What does stealth.js actually modify in the browser?

stealth.js modifies the JavaScript execution environment before any page code runs. It patches the navigator object to remove webdriver properties, simulates realistic chrome runtime objects, and overrides methods that leak automation status (like Navigator.plugins length checks). These modifications happen at the context level, meaning they affect all pages within that browser context.

Is stealth.js compatible with both Playwright and Puppeteer?

While MediaCrawler uses Playwright exclusively, stealth.js originates from the puppeteer-extra ecosystem and is compatible with both engines. The add_init_script method exists in both Playwright and Puppeteer, making the integration pattern transferable. However, MediaCrawler's specific implementation in media_platform/zhihu/core.py uses Playwright's async API.

Where is stealth.js located in the MediaCrawler repository?

The minified bundle resides at libs/stealth.min.js in the repository root. This single file is referenced across all platform implementations and the CDP helper (tools/cdp_browser.py) via relative paths. The unminified source or build scripts are not included in the repository; only the production-ready minified version is distributed.

Can I use stealth.js for other web scraping projects?

Yes, libs/stealth.min.js is a standalone JavaScript file that can be injected into any Playwright or Puppeteer project. Simply copy the file to your project and use browser_context.add_init_script(path="libs/stealth.min.js") before navigating to target sites. Note that as anti-detection measures evolve, you may need to update the bundle to maintain effectiveness against newer bot detection systems.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}