How NanmiCoder/MediaCrawler Handles XiaoHongShu (XHS) Login Authentication

MediaCrawler authenticates to XiaoHongShu through a pluggable login abstraction supporting QR-code scanning, mobile phone SMS verification, and raw cookie injection, with each method implemented in media_platform/xhs/login.py and secured by cryptographic signature generation in playwright_sign.py.

The NanmiCoder/MediaCrawler repository provides a robust framework for crawling Chinese social media platforms. For XiaoHongShu (XHS, also known as Little Red Book), it implements a sophisticated authentication system that handles session management through multiple login strategies while automatically generating required API signatures.

Login Architecture and Abstraction Layer

The authentication system follows a layered architecture that separates platform-agnostic interfaces from XiaoHongShu-specific implementations.

At the base layer, base/base_crawler.py defines the AbstractLogin class. This abstract interface requires concrete implementations to provide four key methods: begin, login_by_qrcode, login_by_mobile, and login_by_cookies. The XiaoHongShuLogin class in media_platform/xhs/login.py extends this abstraction, using Playwright's BrowserContext and Page objects to drive the browser automation.

The entry point begin() method selects the concrete implementation based on config.LOGIN_TYPE:

async def begin(self):
    if config.LOGIN_TYPE == "qrcode":
        await self.login_by_qrcode()
    elif config.LOGIN_TYPE == "phone":
        await self.login_by_mobile()
    elif config.LOGIN_TYPE == "cookie":
        await self.login_by_cookies()
    else:
        raise ValueError("Invalid login type")

User-facing configuration flows through cmd_arg/arg.py, which exposes the LoginTypeEnum to the CLI via the --lt argument. Valid options include qrcode, phone, and cookie, with the selected value stored in config.LOGIN_TYPE. Platform-specific defaults are managed in config/xhs_config.py.

Authentication Methods for XiaoHongShu

The XiaoHongShuLogin class implements three distinct authentication strategies, selectable at runtime through configuration.

QR Code Login

The QR-code method automates the visual login flow used by XiaoHongShu's web interface. When config.LOGIN_TYPE is set to "qrcode", the login_by_qrcode() method executes the following sequence:

1. Navigates to the XiaoHongShu login page using the Playwright page instance
2. Captures the QR-code image element via utils.find_login_qrcode from tools/utils.py
3. Displays the QR-code in the terminal using utils.show_qrcode for user scanning
4. Polls check_login_state() for up to 600 attempts to detect successful authentication

The check_login_state() method first attempts to locate the "Me" sidebar element as a UI indicator of login status, then falls back to detecting CAPTCHA challenges, and finally validates the web_session cookie value against the pre-login baseline using utils.convert_cookies.

Mobile Phone and SMS Login

For automated scenarios requiring credential-based entry, the phone login method interacts with XiaoHongShu's SMS verification system. When using --lt phone, the login_by_mobile() method:

• Opens the login dialog and fills the phone number supplied via --login_phone
• Triggers the SMS request through the web interface
• Polls a Redis cache (created by CacheFactory.create_cache) for the verification code under the key xhs_{phone}
• Enters the retrieved code, clicks "Agree", and submits the form
• Calls check_login_state() to confirm session establishment

This method requires a running Redis instance to bridge the gap between the SMS receiver and the crawler process.

Cookie Injection Login

The cookie method provides the fastest authentication path for pre-validated sessions, ideal for CI/CD pipelines or scheduled jobs. When config.LOGIN_TYPE equals "cookie", the login_by_cookies() method:

• Parses the raw cookie string from config.COOKIES using utils.convert_str_cookie_to_dict
• Extracts the web_session cookie, which uniquely identifies the logged-in session
• Injects the cookie into the browser context via self.browser_context.add_cookies

The crawler then immediately proceeds to check_login_state() to verify the session validity before making API requests.

API Signature Generation and Security

Every request to XiaoHongShu's private API requires cryptographic headers (x-s, x-t, x-s-common, x-b3-traceid) to authenticate the client. MediaCrawler delegates this computation to sign_with_xhshow() in media_platform/xhs/playwright_sign.py, with low-level trace-id extraction handled by media_platform/xhs/xhs_sign.py.

The signature generation process involves:

1. Content string construction via _build_sign_string, which respects GET versus POST semantics when building the payload
2. Hash computation using the patched xhshow library (_patch_xhshow_a3_hash), which fixes a bug in the original library that stripped query parameters from GET requests
3. Header assembly returning a dictionary with the four required authentication headers

These headers attach automatically to all outgoing requests in api/main.py before data extraction occurs.

Configuration and Command-Line Usage

MediaCrawler exposes authentication parameters through standard CLI arguments defined in cmd_arg/arg.py.

To run the crawler with different authentication methods:

# QR-code login (most common)

python main.py --platform xhs --lt qrcode

# Mobile phone login (requires a reachable phone number)

python main.py --platform xhs --lt phone --login_phone 13800138000

# Cookie login (useful for CI/CD or when you already have a valid web_session)

python main.py --platform xhs --lt cookie --cookies "web_session=abcdef123456; other=..."

For programmatic usage, instantiate the login class directly:

import asyncio
from playwright.async_api import async_playwright
from media_platform.xhs.login import XiaoHongShuLogin
from config import LOGIN_TYPE, XHS_SPECIFIED_NOTE_URL_LIST

async def run_login():
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=False)
        context = await browser.new_context()
        page = await context.new_page()
        # Example: QR-code login

        login = XiaoHongShuLogin(
            login_type="qrcode",
            browser_context=context,
            context_page=page,
        )
        await login.begin()          # performs the full QR-code flow

        # After successful login, you can use the stored cookies:

        cookies = await context.cookies()
        print("Logged-in cookies:", cookies)

asyncio.run(run_login())

To generate request signatures manually:

from media_platform.xhs.playwright_sign import sign_with_xhshow

# Example for a GET request

headers = sign_with_xhshow(
    uri="/api/sns/web/v1/feed",
    data={"page": 1, "page_size": 20},
    cookie_str="web_session=abcdef123456; a1=xyz",
    method="GET"
)

print(headers)

# → {'x-s': '...', 'x-t': '...', 'x-s-common': '...', 'x-b3-traceid': '...'}

Summary

• MediaCrawler implements a pluggable authentication architecture via AbstractLogin in base/base_crawler.py and XiaoHongShuLogin in media_platform/xhs/login.py
• Three authentication methods are supported: QR-code scanning, mobile phone SMS verification (using Redis for code retrieval), and direct cookie injection
• The check_login_state() method validates login success by detecting UI elements or monitoring the web_session cookie for up to 600 polling attempts
• All API requests require cryptographic headers (x-s, x-t, x-s-common, x-b3-traceid) generated by sign_with_xhshow() in playwright_sign.py, which patches the xhshow library to handle GET requests correctly
• Configuration is controlled via CLI arguments (--lt, --login_phone, --cookies) stored in the global config module

Frequently Asked Questions

How does MediaCrawler detect successful login on XiaoHongShu?

The check_login_state() method in media_platform/xhs/login.py uses a multi-layered approach. It first checks for the visibility of the "Me" sidebar element using Playwright's is_visible() method. If that fails, it detects CAPTCHA challenges. Finally, it compares the current web_session cookie value against the pre-login value using utils.convert_cookies(). This process repeats for up to 600 attempts with configurable delays.

What is the purpose of the web_session cookie in XiaoHongShu authentication?

The web_session cookie serves as the unique session identifier that proves authentication status to XiaoHongShu's servers. When using cookie login, only this specific cookie is injected into the browser context. During QR and phone login methods, the check_login_state() function specifically monitors this cookie's value to determine when authentication has succeeded.

Why does MediaCrawler patch the xhshow library for signature generation?

The original xhshow library contained a bug that stripped query parameters when computing signatures for GET requests, causing authentication failures. The _patch_xhshow_a3_hash function in media_platform/xhs/playwright_sign.py corrects this by ensuring URL parameters are included in the hash computation. This patched algorithm generates the x-s, x-t, x-s-common, and x-b3-traceid headers required by XiaoHongShu's API.

Can I use MediaCrawler with XiaoHongShu without manual interaction?

Yes, but only with the cookie or phone login methods. The QR-code method requires manual scanning of the displayed code. For fully automated workflows, supply a valid web_session cookie via --cookies (cookie method), or configure the phone method with a Redis-backed SMS receiver to automatically retrieve verification codes without human intervention.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}