MediaCrawler Platform API Signature Handling: XHS and Douyin Implementation Guide

MediaCrawler generates platform-specific API signatures by implementing a pure-Python algorithm for Xiaohongshu (XHS) that recreates obfuscated JavaScript logic, while executing the original JavaScript via execjs for Douyin's a_bogus parameter.

MediaCrawler is an open-source scraping framework maintained by NanmiCoder that automates data collection from Chinese short-video platforms. Understanding its MediaCrawler platform API signature handling is essential for developers building custom crawlers, as the project implements distinct cryptographic strategies for XHS and Douyin to bypass anti-bot protections.

Xiaohongshu (XHS) Pure-Python Signature Algorithm

The XHS signature mechanism relies on a custom Base64-encoded string containing a CRC32-derived field (x-s-common). Rather than executing external JavaScript, MediaCrawler fully recreates this algorithm in Python within media_platform/xhs/xhs_sign.py.

Custom Base64 and CRC32 Implementation

The implementation uses a shuffled Base64 character table (BASE64_CHARS) distinct from standard Base64 encoding. The mrc() function computes a CRC32 variant using a pre-computed CRC32_TABLE combined with an unsigned right-shift helper _right_shift_unsigned. This mimics the bitwise operations found in the original obfuscated client code.

The encoding process handles data in chunks through _encode_chunk(), processing up to 16,383 bytes at a time before b64_encode() applies final padding. This chunked approach ensures compatibility with the platform's memory handling expectations.

Trace ID Generation

Every XHS request requires a unique trace identifier generated by get_trace_id(). This function produces a random 16-character hexadecimal string that serves as the foundation for subsequent signature calculations. The trace ID populates the X-S header, while its CRC32 derivative populates X-S-Common.

Building Signed Headers

The following implementation demonstrates how to construct valid request headers using the pure-Python signature functions:

from media_platform.xhs.xhs_sign import get_trace_id, mrc, b64_encode, encode_utf8

def build_xhs_headers(cookie: str) -> dict:
    """
    Generate XHS API headers with valid X-S and X-S-Common signatures.
    """
    # Generate the base trace ID

    trace_id = get_trace_id()
    
    # Compute CRC32-derived value

    crc_value = mrc(trace_id)
    
    # Encode using custom Base64 routine

    x_s_common = b64_encode(encode_utf8(str(crc_value)))
    
    return {
        "X-S": trace_id,
        "X-S-Common": x_s_common,
        "Cookie": cookie,
        # Additional headers (X-T, X-B3-TraceId) constructed similarly

    }

Douyin JavaScript Signature Execution

Unlike XHS, Douyin's a_bogus parameter relies on a heavily obfuscated JavaScript routine that changes frequently. MediaCrawler handles this through media_platform/douyin/help.py by executing the original libs/douyin.js community-provided script rather than attempting a Python port.

execjs Integration

The primary method loads the JavaScript file using execjs, which embeds a JavaScript engine (Node.js or V8) to run the signing functions. Depending on the endpoint, the code calls either sign_datail (default) or sign_reply (for comment endpoints):

import execjs
from urllib.parse import urlencode

# Load the JavaScript implementation once (cached)

douyin_js = execjs.compile(
    open('libs/douyin.js', encoding='utf-8-sig').read()
)

def get_a_bogus(params: dict, user_agent: str, is_reply: bool = False) -> str:
    """
    Generate Douyin a_bogus token using native JavaScript execution.
    """
    param_str = urlencode(params, doseq=True)
    func_name = "sign_reply" if is_reply else "sign_datail"
    
    return douyin_js.call(func_name, param_str, user_agent)

Playwright Fallback Method

When a Playwright Page object is available, MediaCrawler provides a fallback that evaluates the signature function directly within the browser context. This method targets the specific obfuscated path window.bdms.init._v[2].p[42]:

from playwright.async_api import Page

async def get_a_bogus_from_playwright(
    params: str, 
    post_data: dict, 
    user_agent: str, 
    page: Page
) -> str:
    """
    Deprecated fallback: Evaluate signature inside Playwright page.
    """
    a_bogus = await page.evaluate(
        "([params, post_data, ua]) => "
        "window.bdms.init._v[2].p[42].apply(null, [0, 1, 8, params, post_data, ua])",
        [params, post_data or "", user_agent]
    )
    return a_bogus

Implementation Strategy Comparison

MediaCrawler employs these divergent approaches based on platform-specific constraints:

• XHS (Pure Python): The algorithm is stable and published by the community, making a Python port feasible. This eliminates JavaScript runtime overhead and version-locking issues, with all constants (BASE64_CHARS, CRC32_TABLE) frozen in the repository.

• Douyin (JavaScript Execution): The signing algorithm is large, frequently updated, and resistant to deobfuscation. Using execjs with the original douyin.js allows rapid updates when Douyin changes its validation logic, while the Playwright fallback ensures correctness if the local JS engine mismatches the official client version.

Practical Integration Examples

For developers extending MediaCrawler, here are minimal implementations for obtaining valid signatures:

XHS Header Generation

from media_platform.xhs.xhs_sign import get_trace_id, mrc, b64_encode, encode_utf8

def generate_xhs_auth_headers(cookie_string: str) -> dict:
    trace = get_trace_id()
    crc_val = mrc(trace)
    encoded = b64_encode(encode_utf8(str(crc_val)))
    
    return {
        "X-S": trace,
        "X-S-Common": encoded,
        "Cookie": cookie_string,
        "User-Agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 16_6 like Mac OS X)..."
    }

Douyin a_bogus Generation

import execjs

# Initialize once at module level

_js_context = execjs.compile(
    open('libs/douyin.js', encoding='utf-8-sig').read()
)

def sign_douyin_request(endpoint: str, params: dict, ua: str) -> str:
    is_reply = "/reply" in endpoint
    func = "sign_reply" if is_reply else "sign_datail"
    query = "&".join(f"{k}={v}" for k, v in params.items())
    
    return _js_context.call(func, query, ua)

Summary

• XHS signatures are generated entirely in Python via media_platform/xhs/xhs_sign.py using custom Base64 encoding and CRC32 calculations, eliminating external dependencies.
• Douyin signatures require executing libs/douyin.js through execjs in media_platform/douyin/help.py, with a Playwright fallback for edge cases.
• The X-S and X-S-Common headers for XHS derive from a 16-character trace ID processed through mrc() and b64_encode().
• The a_bogus parameter for Douyin invokes either sign_datail or sign_reply functions from the original JavaScript implementation.
• MediaCrawler's architecture prioritizes maintainability for Douyin (external JS) and performance for XHS (pure Python).

Frequently Asked Questions

How does MediaCrawler handle XHS signature updates when the platform changes its algorithm?

When Xiaohongshu updates its signing logic, developers must manually reverse-engineer the new JavaScript algorithm and update the Python implementation in media_platform/xhs/xhs_sign.py. The project maintains a fallback in media_platform/xhs/playwright_sign.py that uses the external xhshow library via Playwright if the pure-Python implementation temporarily breaks.

Why does Douyin use JavaScript execution instead of a Python port like XHS?

Douyin's a_bogus generation involves thousands of lines of heavily obfuscated JavaScript that changes frequently. Attempting to port this to Python would create a fragile implementation requiring constant maintenance. By executing the original libs/douyin.js through execjs, MediaCrawler simply replaces the JavaScript file when the platform updates, ensuring immediate compatibility without code changes.

What are the performance implications of using execjs for Douyin signatures?

The execjs approach introduces overhead from spawning a JavaScript runtime (Node.js), typically adding 50-200ms per signature generation compared to pure Python. For high-throughput scenarios, developers can cache execjs.compile() results as shown in the examples, or use the Playwright fallback which evaluates signatures within an existing browser context.

Can MediaCrawler generate signatures without installing Node.js?

For XHS, no JavaScript runtime is required as the algorithm is pure Python. For Douyin, Node.js is mandatory when using the primary execjs method in media_platform/douyin/help.py. However, if you use the Playwright fallback method get_a_bogus_from_playwright(), the signature evaluates inside the browser context, bypassing the Node.js requirement at the cost of maintaining an active browser instance.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}