How MediaCrawler Manages API Versions and Changes for Xiaohongshu (XHS)

MediaCrawler isolates each platform in dedicated packages with centralized constants, signature abstraction layers, and factory patterns to handle API changes without modifying core crawler logic.

The NanmiCoder/MediaCrawler repository employs a modular architecture to manage different API versions and changes for platforms like Xiaohongshu (xhs). By isolating platform-specific logic into self-contained modules and abstracting version-sensitive operations, the codebase remains resilient against frequent third-party API churn.

Modular Platform Isolation

MediaCrawler organizes each social media platform into its own subpackage under media_platform/. For Xiaohongshu, the media_platform/xhs/ directory contains granular, single-purpose modules that separate concerns:

• login.py – Manages authentication flows and cookie refresh mechanisms.
• field.py – Stores static field names and API endpoint URLs as module-level constants.
• extractor.py – Transforms raw HTTP responses into Pydantic models defined in model/m_xiaohongshu.py.
• exception.py – Defines platform-specific exceptions like RateLimitError and InvalidSignatureError for version-related error handling.

This isolation ensures that changes to the XHS API never leak into other platform implementations such as Douyin or Bilibili.

Centralized Constants for Version Control

All version-sensitive strings reside in centralized locations. In media_platform/xhs/field.py, API field names and JSON keys are defined as constants. When Xiaohongshu renames a field in a newer API version, updating the constant in this single file propagates the change throughout the entire application.

Similarly, media_platform/xhs/extractor.py defines endpoint URLs as constants. This design allows developers to point the crawler at a new API version by modifying only the constant values, eliminating the need to hunt through business logic for hardcoded strings.

Signature Abstraction and Library Patching

Request signing logic is abstracted into media_platform/xhs/playwright_sign.py and media_platform/xhs/xhs_sign.py. These modules wrap the third-party xhshow library, which handles the cryptographic heavy lifting required by Xiaohongshu's API.

When the upstream API changed its a3_hash calculation algorithm, MediaCrawler implemented a compatibility fix without forking the library. The xhs_sign.py module contains a _patch_xhshow_a3_hash function that monkey-patches the library at runtime:

# Example: Generating a signed request for a Xiaohongshu endpoint

from media_platform.xhs.playwright_sign import sign_with_xhshow

cookie_str = "xsec_user_id=12345; webId=abcde..."
payload = {
    "note_id": "63f0c8e2c3c2c70001c5e6d2",
    "page": 1,
}

headers = sign_with_xhshow(
    cookie_str=cookie_str,
    api_path="/api/sns/web/v1/note/detail",
    payload=payload,
    method="POST",
)

The sign_with_xhshow function internally applies the _patch_xhshow_a3_hash patch to maintain compatibility with the latest API version, keeping the core crawler implementation unchanged.

Factory Pattern for Version Switching

The data persistence layer uses a factory pattern to support multiple API versions simultaneously. Located in store/xhs/__init__.py and implemented in store/xhs/_store_impl.py, the XhsStoreFactory reads configuration settings and instantiates the appropriate store implementation:

# Example: Switching API version via the store factory

from store.xhs import XhsStoreFactory

# The factory reads the configured version (default = "v1")

store = XhsStoreFactory.get_store(version="v2")
await store.update_xhs_note({"note_id": "...", "content": "new content"})

This approach allows the crawler to switch between API versions by changing a configuration flag, with the factory handling the instantiation of version-specific logic.

Configuration-Driven Feature Toggles

Runtime configuration management supports version switching without code changes. The config module—imported throughout the XHS package—reads environment variables such as SAVE_DATA_PATH and version identifiers. Adding support for a new XHS API version typically requires only toggling a configuration flag, leaving the crawling logic untouched.

Summary

• Modular isolation keeps platform-specific code in media_platform/xhs/, preventing version changes from affecting other crawlers.
• Centralized constants in field.py and extractor.py allow single-file updates when API endpoints or field names change.
• Signature abstraction via xhs_sign.py and playwright_sign.py uses runtime patching (_patch_xhshow_a3_hash) to adapt to algorithm changes without modifying core logic.
• Factory pattern implementation in store/xhs/ enables runtime version selection through XhsStoreFactory.get_store().
• Configuration-driven design allows API version switching via environment variables rather than code modifications.

Frequently Asked Questions

How does MediaCrawler handle breaking changes in XHS API signatures?

When Xiaohongshu modifies its signing algorithm, MediaCrawler updates the abstraction layer in media_platform/xhs/xhs_sign.py. The repository uses a monkey-patch approach via _patch_xhshow_a3_hash to fix upstream library incompatibilities without forking dependencies, ensuring the core crawler remains stable.

What is the purpose of the XhsStoreFactory?

The XhsStoreFactory defined in store/xhs/__init__.py acts as a version-aware instantiation mechanism. It reads configuration to determine which API version to use and returns the appropriate implementation from store/xhs/_store_impl.py, enabling seamless switching between different API versions at runtime.

How do I switch between different API versions in MediaCrawler?

Switch versions by passing a version parameter to the factory method or by setting the appropriate environment variable in your configuration. The factory in store/xhs/__init__.py handles the instantiation logic, while centralized constants in field.py and extractor.py ensure the correct endpoints and field mappings are used.

Where are API endpoint URLs defined in the codebase?

All endpoint URLs are defined as constants in media_platform/xhs/extractor.py. This centralization means that when Xiaohongshu releases a new API version, you only need to update the URL constants in this single file rather than searching through multiple modules for hardcoded strings.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}