# MediaCrawler Platform API Signature Handling: XHS and Douyin Implementation Guide

> Master XHS and Douyin API signature handling with MediaCrawler. This guide details pure-Python XHS signing and original JavaScript execution for Douyin's a_bogus parameter.

- Repository: [程序员阿江-Relakkes/MediaCrawler](https://github.com/NanmiCoder/MediaCrawler)
- Tags: api-reference
- Published: 2026-07-31

---

**MediaCrawler generates platform-specific API signatures by implementing a pure-Python algorithm for Xiaohongshu (XHS) that recreates obfuscated JavaScript logic, while executing the original JavaScript via execjs for Douyin's a_bogus parameter.**

MediaCrawler is an open-source scraping framework maintained by NanmiCoder that automates data collection from Chinese short-video platforms. Understanding its **MediaCrawler platform API signature handling** is essential for developers building custom crawlers, as the project implements distinct cryptographic strategies for XHS and Douyin to bypass anti-bot protections.

## Xiaohongshu (XHS) Pure-Python Signature Algorithm

The XHS signature mechanism relies on a custom Base64-encoded string containing a CRC32-derived field (`x-s-common`). Rather than executing external JavaScript, MediaCrawler fully recreates this algorithm in Python within [`media_platform/xhs/xhs_sign.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/xhs/xhs_sign.py).

### Custom Base64 and CRC32 Implementation

The implementation uses a shuffled Base64 character table (`BASE64_CHARS`) distinct from standard Base64 encoding. The `mrc()` function computes a CRC32 variant using a pre-computed `CRC32_TABLE` combined with an unsigned right-shift helper `_right_shift_unsigned`. This mimics the bitwise operations found in the original obfuscated client code.

The encoding process handles data in chunks through `_encode_chunk()`, processing up to 16,383 bytes at a time before `b64_encode()` applies final padding. This chunked approach ensures compatibility with the platform's memory handling expectations.

### Trace ID Generation

Every XHS request requires a unique trace identifier generated by `get_trace_id()`. This function produces a random 16-character hexadecimal string that serves as the foundation for subsequent signature calculations. The trace ID populates the `X-S` header, while its CRC32 derivative populates `X-S-Common`.

### Building Signed Headers

The following implementation demonstrates how to construct valid request headers using the pure-Python signature functions:

```python
from media_platform.xhs.xhs_sign import get_trace_id, mrc, b64_encode, encode_utf8

def build_xhs_headers(cookie: str) -> dict:
    """
    Generate XHS API headers with valid X-S and X-S-Common signatures.
    """
    # Generate the base trace ID

    trace_id = get_trace_id()
    
    # Compute CRC32-derived value

    crc_value = mrc(trace_id)
    
    # Encode using custom Base64 routine

    x_s_common = b64_encode(encode_utf8(str(crc_value)))
    
    return {
        "X-S": trace_id,
        "X-S-Common": x_s_common,
        "Cookie": cookie,
        # Additional headers (X-T, X-B3-TraceId) constructed similarly

    }

```

## Douyin JavaScript Signature Execution

Unlike XHS, Douyin's `a_bogus` parameter relies on a heavily obfuscated JavaScript routine that changes frequently. MediaCrawler handles this through [`media_platform/douyin/help.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/douyin/help.py) by executing the original [`libs/douyin.js`](https://github.com/NanmiCoder/MediaCrawler/blob/main/libs/douyin.js) community-provided script rather than attempting a Python port.

### execjs Integration

The primary method loads the JavaScript file using **execjs**, which embeds a JavaScript engine (Node.js or V8) to run the signing functions. Depending on the endpoint, the code calls either `sign_datail` (default) or `sign_reply` (for comment endpoints):

```python
import execjs
from urllib.parse import urlencode

# Load the JavaScript implementation once (cached)

douyin_js = execjs.compile(
    open('libs/douyin.js', encoding='utf-8-sig').read()
)

def get_a_bogus(params: dict, user_agent: str, is_reply: bool = False) -> str:
    """
    Generate Douyin a_bogus token using native JavaScript execution.
    """
    param_str = urlencode(params, doseq=True)
    func_name = "sign_reply" if is_reply else "sign_datail"
    
    return douyin_js.call(func_name, param_str, user_agent)

```

### Playwright Fallback Method

When a Playwright `Page` object is available, MediaCrawler provides a fallback that evaluates the signature function directly within the browser context. This method targets the specific obfuscated path `window.bdms.init._v[2].p[42]`:

```python
from playwright.async_api import Page

async def get_a_bogus_from_playwright(
    params: str, 
    post_data: dict, 
    user_agent: str, 
    page: Page
) -> str:
    """
    Deprecated fallback: Evaluate signature inside Playwright page.
    """
    a_bogus = await page.evaluate(
        "([params, post_data, ua]) => "
        "window.bdms.init._v[2].p[42].apply(null, [0, 1, 8, params, post_data, ua])",
        [params, post_data or "", user_agent]
    )
    return a_bogus

```

## Implementation Strategy Comparison

MediaCrawler employs these divergent approaches based on platform-specific constraints:

- **XHS (Pure Python)**: The algorithm is stable and published by the community, making a Python port feasible. This eliminates JavaScript runtime overhead and version-locking issues, with all constants (`BASE64_CHARS`, `CRC32_TABLE`) frozen in the repository.

- **Douyin (JavaScript Execution)**: The signing algorithm is large, frequently updated, and resistant to deobfuscation. Using `execjs` with the original [`douyin.js`](https://github.com/NanmiCoder/MediaCrawler/blob/main/douyin.js) allows rapid updates when Douyin changes its validation logic, while the Playwright fallback ensures correctness if the local JS engine mismatches the official client version.

## Practical Integration Examples

For developers extending MediaCrawler, here are minimal implementations for obtaining valid signatures:

### XHS Header Generation

```python
from media_platform.xhs.xhs_sign import get_trace_id, mrc, b64_encode, encode_utf8

def generate_xhs_auth_headers(cookie_string: str) -> dict:
    trace = get_trace_id()
    crc_val = mrc(trace)
    encoded = b64_encode(encode_utf8(str(crc_val)))
    
    return {
        "X-S": trace,
        "X-S-Common": encoded,
        "Cookie": cookie_string,
        "User-Agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 16_6 like Mac OS X)..."
    }

```

### Douyin a_bogus Generation

```python
import execjs

# Initialize once at module level

_js_context = execjs.compile(
    open('libs/douyin.js', encoding='utf-8-sig').read()
)

def sign_douyin_request(endpoint: str, params: dict, ua: str) -> str:
    is_reply = "/reply" in endpoint
    func = "sign_reply" if is_reply else "sign_datail"
    query = "&".join(f"{k}={v}" for k, v in params.items())
    
    return _js_context.call(func, query, ua)

```

## Summary

- **XHS signatures** are generated entirely in Python via [`media_platform/xhs/xhs_sign.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/xhs/xhs_sign.py) using custom Base64 encoding and CRC32 calculations, eliminating external dependencies.
- **Douyin signatures** require executing [`libs/douyin.js`](https://github.com/NanmiCoder/MediaCrawler/blob/main/libs/douyin.js) through `execjs` in [`media_platform/douyin/help.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/douyin/help.py), with a Playwright fallback for edge cases.
- The **X-S** and **X-S-Common** headers for XHS derive from a 16-character trace ID processed through `mrc()` and `b64_encode()`.
- The **a_bogus** parameter for Douyin invokes either `sign_datail` or `sign_reply` functions from the original JavaScript implementation.
- MediaCrawler's architecture prioritizes **maintainability** for Douyin (external JS) and **performance** for XHS (pure Python).

## Frequently Asked Questions

### How does MediaCrawler handle XHS signature updates when the platform changes its algorithm?

When Xiaohongshu updates its signing logic, developers must manually reverse-engineer the new JavaScript algorithm and update the Python implementation in [`media_platform/xhs/xhs_sign.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/xhs/xhs_sign.py). The project maintains a fallback in [`media_platform/xhs/playwright_sign.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/xhs/playwright_sign.py) that uses the external **xhshow** library via Playwright if the pure-Python implementation temporarily breaks.

### Why does Douyin use JavaScript execution instead of a Python port like XHS?

Douyin's `a_bogus` generation involves thousands of lines of heavily obfuscated JavaScript that changes frequently. Attempting to port this to Python would create a fragile implementation requiring constant maintenance. By executing the original [`libs/douyin.js`](https://github.com/NanmiCoder/MediaCrawler/blob/main/libs/douyin.js) through `execjs`, MediaCrawler simply replaces the JavaScript file when the platform updates, ensuring immediate compatibility without code changes.

### What are the performance implications of using execjs for Douyin signatures?

The `execjs` approach introduces overhead from spawning a JavaScript runtime (Node.js), typically adding 50-200ms per signature generation compared to pure Python. For high-throughput scenarios, developers can cache `execjs.compile()` results as shown in the examples, or use the Playwright fallback which evaluates signatures within an existing browser context.

### Can MediaCrawler generate signatures without installing Node.js?

For XHS, no JavaScript runtime is required as the algorithm is pure Python. For Douyin, Node.js is mandatory when using the primary `execjs` method in [`media_platform/douyin/help.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/douyin/help.py). However, if you use the Playwright fallback method `get_a_bogus_from_playwright()`, the signature evaluates inside the browser context, bypassing the Node.js requirement at the cost of maintaining an active browser instance.