How MediaCrawler Handles Platform-Specific API Signatures and JavaScript Injection

MediaCrawler uses dedicated signature helper modules per platform plus stealth JavaScript injection to bypass anti-bot detection and generate valid request signatures for Zhihu, Douyin, Xiaohongshu, and other platforms.

Each social media platform in the MediaCrawler project—Zhihu, Douyin, Xiaohongshu (XHS), Bilibili, Kuaishou, Tieba, and Weibo—protects its APIs with proprietary signing algorithms implemented in JavaScript. This article examines exactly how the codebase handles platform-specific API signatures and JavaScript injection to emulate legitimate browser sessions.

---

Platform-Specific Signature Architecture

MediaCrawler isolates signing logic into dedicated help.py modules or client-level implementations. This modular design lets each platform evolve independently without breaking others.

Zhihu: JavaScript Engine Delegation

In media_platform/zhihu/help.py, the sign(url, cookies) function forwards parameters to a bundled JavaScript engine (ZHIHU_SGIN_JS) and extracts two critical headers:

• x-zst-81 – timestamp-based token
• x-zse-96 – request signature

# media_platform/zhihu/help.py (conceptual flow)

def sign(url: str, cookies: dict) -> dict:
    # ZHIHU_SGIN_JS contains the obfuscated signing algorithm

    result = js_engine.eval(ZHIHU_SGIN_JS, url, cookies)
    return {
        "x-zst-81": result["zst81"],
        "x-zse-96": result["zse96"]
    }

The core module in media_platform/zhihu/core.py injects these headers before every authenticated request.

Douyin: execjs with External JS Library

Douyin's implementation in media_platform/douyin/help.py uses the execjs library to load libs/douyin.js. This handwritten JavaScript file exposes functions like sign_datail and sign_reply that generate the a_bogus parameter required by Douyin's API:

from media_platform.douyin.client import DouyinClient

client = DouyinClient()
params = {"aweme_id": "1234567890", "device_platform": "android"}

# Internally:

# 1. execjs loads libs/douyin.js

# 2. Calls sign_datail(params, user_agent) → returns a_bogus

resp = await client.get("/aweme/v1/web/aweme/detail/", params=params)

The a_bogus parameter is platform-specific to Douyin and changes frequently, requiring the external JS file to stay synchronized with upstream changes.

Xiaohongshu: Dual Implementation Strategy

Xiaohongshu employs two complementary approaches in media_platform/xhs/:

Implementation	File	Method	Use Case
Pure Python	xhs_sign.py	Algorithmic reproduction	Fast, no browser overhead
Playwright	playwright_sign.py	Delegates to xhshow library	When Python implementation breaks

# media_platform/xhs/xhs_sign.py — Python-native signing

from media_platform.xhs.xhs_sign import sign_request

headers = sign_request(endpoint, params)  # No JS runtime needed

# media_platform/xhs/playwright_sign.py — Browser-based fallback

from media_platform.xhs.playwright_sign import PlaywrightSigner

signer = PlaywrightSigner()
headers = await signer.sign_with_xhshow(endpoint, params)  # Uses Playwright page

This redundancy ensures crawler resilience when XHS rotates their signing algorithm.

Bilibili: WBI Specification Compliance

Bilibili follows a documented standard. In media_platform/bilibili/help.py, the BiliWbiSigner class implements the WBI (Web Binary Interface) specification:

from media_platform.bilibili.help import BiliWbiSigner

signer = BiliWbiSigner()
query = {"mid": "12345", "platform": "web"}
signed_query = signer.sign(query)

# signed_query now includes: {"w_rid": "md5_hash_of_query_plus_salt", ...}

The algorithm concatenates sorted query parameters with a dynamic salt (rotated periodically by Bilibili), then computes an MD5 hash to produce w_rid.

Kuaishou: Full Playwright Evaluation

Unlike other platforms, Kuaishou's signing logic in media_platform/kuaishou/help.py runs entirely within a headless browser. The get_ks_sign_from_playwright function executes the platform's native JavaScript in a controlled Chromium context:

# media_platform/kuaishou/help.py

async def get_ks_sign_from_playwright(url: str) -> dict:
    page = await browser_context.new_page()
    # Evaluate KS's own signing functions in their runtime environment

    signature = await page.evaluate("""() => {
        return window._getSign(JSON.parse(arguments[0]));
    }""", json.dumps(parse_url(url)))
    return signature

This approach sacrifices speed for accuracy when the signing algorithm resists static analysis.

Tieba: Deterministic MD5 with Secret

Baidu Tieba uses a simpler but effective scheme in media_platform/tieba/client.py. The client builds an MD5 hash over alphabetically sorted parameters concatenated with PC_SIGN_SECRET:

# media_platform/tieba/client.py

def _generate_sign(params: dict) -> str:
    sorted_params = "&".join(f"{k}={v}" for k, v in sorted(params.items()))
    return md5(f"{sorted_params}{PC_SIGN_SECRET}".encode()).hexdigest()

No JavaScript injection is required—Tieba's signing is fully implemented in Python.

Weibo: Core-Integrated Signing

Weibo handles signatures directly in media_platform/weibo/core.py, adding headers based on the platform's login flow and session state without a separate helper module.

---

JavaScript Injection for Stealth and Anti-Detection

Generating valid signatures is necessary but insufficient. Modern platforms detect automation through JavaScript environment fingerprinting. MediaCrawler neutralizes these checks via systematic stealth script injection.

The Stealth Script: libs/stealth.min.js

MediaCrawler bundles libs/stealth.min.js, a minified anti-fingerprinting script derived from the puppeteer-extra-plugin-stealth ecosystem. This script:

• Deletes navigator.webdriver
• Spoofs navigator.plugins, navigator.languages, and screen properties
• Overrides WebDriver property getters to return undefined
• Patches chrome.runtime and other extension APIs to appear absent

Per-Platform Injection Pattern

Every Playwright-based core module injects the stealth script before any navigation:

# media_platform/zhihu/core.py (representative of all platforms)

class ZhihuCrawler:
    async def initialize(self):
        self.browser = await chromium.launch()
        self.browser_context = await self.browser.new_context()
        
        # Critical: inject before any page creation

        await self.browser_context.add_init_script(path="libs/stealth.min.js")
        
        self.page = await self.browser_context.new_page()

The same pattern appears in:

• media_platform/douyin/core.py
• media_platform/xhs/core.py
• media_platform/kuaishou/core.py

CDP Browser Manager

For advanced use cases, tools/cdp_browser.py provides CDPBrowserManager with a dedicated add_stealth_script() method:

from tools.cdp_browser import CDPBrowserManager

manager = CDPBrowserManager()
await manager.add_stealth_script()  # Injects libs/stealth.min.js via CDP

This enables stealth injection when connecting to existing Chrome instances via the Chrome DevTools Protocol (CDP), bypassing Playwright's standard launch flow.

---

Signature Generation Flow: Complete Example

Here's how a Zhihu request flows from caller to signed HTTP request:

from media_platform.zhihu.client import ZhihuClient

async def fetch_answers():
    client = ZhihuClient()
    url = "https://www.zhihu.com/api/v4/questions/123456/answers"
    
    # Step 1: core.py initializes browser with stealth script injected

    await client.initialize()  # add_init_script("libs/stealth.min.js")

    
    # Step 2: client.get() calls help.sign() with URL and cookies

    # Step 3: help.py evaluates ZHIHU_SGIN_JS, extracts x-zst-81 / x-zse-96

    # Step 4: headers applied, request executed in stealth context

    response = await client.get(url)
    
    return response.json()

The JavaScript signing algorithm runs in the stealth-injected context, producing signatures indistinguishable from genuine browser sessions.

---

Implementation Comparison by Platform

Platform	Signature Source	Injection Method	Stealth Required
Zhihu	ZHIHU_SGIN_JS via help.py	add_init_script	Yes
Douyin	libs/douyin.js via execjs	add_init_script	Yes
Xiaohongshu	xhs_sign.py or playwright_sign.py	add_init_script	Yes
Bilibili	Pure Python WBI in help.py	None	Optional
Kuaishou	Native JS via Playwright evaluation	add_init_script	Yes
Tieba	Pure Python MD5 in client.py	None	No
Weibo	Integrated in core.py	add_init_script	Yes

---

Summary

• Modular signature helpers (help.py modules) isolate platform-specific signing logic per service
• Dual strategies (Python-native vs. JavaScript evaluation) balance speed and accuracy across platforms
• Stealth script injection via add_init_script(path="libs/stealth.min.js") masks automation fingerprints before any page loads
• CDP support through tools/cdp_browser.py extends stealth capabilities to externally managed browser instances
• No one-size-fits-all: Each platform requires tailored analysis—Bilibili uses documented specs while Kuaishou demands full browser evaluation

---

Frequently Asked Questions

How does MediaCrawler avoid detection when signing requests?

MediaCrawler injects libs/stealth.min.js into every Playwright context before navigation. This script patches navigator.webdriver and other automation indicators, ensuring the JavaScript signing environment matches a genuine browser. The injection happens in media_platform/{platform}/core.py via browser_context.add_init_script().

Why does Xiaohongshu need two different signing implementations?

Xiaohongshu's signing algorithm changes periodically. The pure-Python xhs_sign.py offers speed when functional, while playwright_sign.py delegates to the official xhshow library as a fallback. This dual approach minimizes downtime when the platform rotates its anti-scraping measures.

What is the a_bogus parameter in Douyin requests?

a_bogus is Douyin's request signature parameter. MediaCrawler generates it by loading libs/douyin.js through execjs and calling platform-specific functions (sign_datail, sign_reply) with request parameters and user-agent strings. The JavaScript file requires manual updates when Douyin changes their algorithm.

When should I use the CDP browser manager instead of standard Playwright?

Use tools/cdp_browser.py when connecting to existing Chrome instances rather than launching new browsers. The add_stealth_script() method ensures stealth injection works via CDP, useful for distributed crawling or reusing already-warmed browser sessions with established platform cookies.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}