How to Use MediaCrawler for Xiaohongshu Data: Complete Setup Guide

MediaCrawler extracts Xiaohongshu (XHS) data through a modular Python API that handles QR-code authentication, URL parsing, and async data retrieval, supporting both individual notes and creator profile scraping.

MediaCrawler is an open-source scraping framework maintained at NanmiCoder/MediaCrawler that supports multiple Chinese social platforms. When using MediaCrawler for Xiaohongshu data collection, you leverage a dedicated XHS module consisting of specialized login handlers, URL parsers, and high-level crawler classes that interface with the platform's internal API endpoints.

Architecture Overview

The Xiaohongshu implementation follows a layered architecture where each component handles a specific phase of the data extraction pipeline.

Core Source Files

Understanding the repository structure helps you customize the scraping behavior:

• media_platform/xhs/login.py – Contains QR-code, mobile-phone, and cookie-based authentication flows using Selenium and Chrome DevTools Protocol
• media_platform/xhs/help.py – Provides URL parsing utilities that convert public Xiaohongshu links into structured data models
• model/m_xiaohongshu.py – Defines Pydantic models NoteUrlInfo and CreatorUrlInfo that validate and store URL metadata
• media_platform/xhs/client.py – Implements XhsClient, the low-level HTTP wrapper targeting https://edith.xiaohongshu.com
• media_platform/xhs/core.py – Houses the XhsCrawler class with high-level methods like get_note_detail() and get_creator_notes()
• store/xhs/__init__.py – Handles persistence layer for JSON, MongoDB, or Excel output formats
• main.py – CLI entry point for command-line execution

Configuration and Authentication

Before extracting data, configure your credentials. The system supports three authentication methods implemented in media_platform/xhs/login.py: QR-code scanning, mobile SMS login, or cookie reuse.

Create a .env file in the project root (copy from .env.example):

XHS_COOKIE="your_xiaohongshu_cookie_here"

If this value is empty, the crawler will automatically prompt for QR-code login at runtime using the Selenium-based handler.

Parsing Xiaohongshu URLs

The framework requires structured URL objects before making API calls. Use the helper functions in media_platform/xhs/help.py to transform public URLs into the internal representation defined in model/m_xiaohongshu.py.

Key parsing functions:

• parse_note_info_from_note_url(url) – Returns a NoteUrlInfo model containing the note ID and extraction metadata
• parse_creator_info_from_profile_url(url) – Returns a CreatorUrlInfo model with the user ID and profile parameters

These parsers validate URL patterns and extract the necessary identifiers for the XHS API endpoints.

Implementing the Crawler

The XhsCrawler class in media_platform/xhs/core.py serves as the primary interface. It automatically manages authentication state and rate limiting when calling the internal API at https://edith.xiaohongshu.com.

Fetching Individual Notes

This example demonstrates parsing a note URL and extracting title and image data:

import asyncio
from media_platform.xhs.core import XhsCrawler
from media_platform.xhs.help import parse_note_info_from_note_url

# Public Xiaohongshu note URL

NOTE_URL = "https://www.xiaohongshu.com/explore/64b95d01000000000c034587"

async def fetch_single_note():
    # Parse URL into structured model

    note_info = parse_note_info_from_note_url(NOTE_URL)
    
    # Initialize crawler (handles login automatically)

    crawler = XhsCrawler()
    
    # Retrieve detailed note data

    note_detail = await crawler.get_note_detail(note_info)
    
    # Process results

    print("Title:", note_detail["title"])
    print("Images:")
    for img in note_detail.get("image_list", []):
        print(f"  - {img['url']}")

if __name__ == "__main__":
    asyncio.run(fetch_single_note())

Scraping Creator Profiles

To extract a creator's publication history, use get_creator_notes() with pagination support:

from media_platform.xhs.help import parse_creator_info_from_profile_url
from media_platform.xhs.core import XhsCrawler

CREATOR_URL = "https://www.xiaohongshu.com/user/profile/5eb8e1d400000000010075ae"

async def crawl_creator_feed():
    creator_info = parse_creator_info_from_profile_url(CREATOR_URL)
    crawler = XhsCrawler()
    
    # Fetch first page of creator's notes

    notes_page = await crawler.get_creator_notes(creator_info, page=1)
    
    for note in notes_page["notes"]:
        print(f"Note ID: {note['note_id']} - Title: {note['title']}")

# Execute with asyncio.run(crawl_creator_feed())

Using the Command Line Interface

MediaCrawler includes a CLI wrapper in main.py for quick execution without writing Python scripts:

# Fetch specific note

python main.py xhs note --url "https://www.xiaohongshu.com/explore/64b95d01000000000c034587"

# Fetch creator profile

python main.py xhs creator --url "https://www.xiaohongshu.com/user/profile/5eb8e1d400000000010075ae"

The CLI parser in main.py routes these arguments to the XhsCrawler class with automatic authentication handling.

Storage and Persistence

Extracted data flows through the storage layer defined in store/xhs/__init__.py. By default, the system supports JSON file export, MongoDB insertion, and Excel generation. You can configure the storage backend in config/xhs_config.py alongside proxy settings and timeout parameters.

The crawler returns standard Python dictionaries that you can also process directly before persistence, enabling custom data pipelines or real-time processing workflows.

Summary

• Clone the NanmiCoder/MediaCrawler repository and install dependencies via requirements.txt
• Configure authentication in environment variables or use QR-code login through media_platform/xhs/login.py
• Parse target URLs using helpers in media_platform/xhs/help.py to generate NoteUrlInfo or CreatorUrlInfo objects
• Instantiate XhsCrawler from media_platform/xhs/core.py and call async methods like get_note_detail() or get_creator_notes()
• Store results using the built-in storage layer in store/xhs/__init__.py or custom backends

Frequently Asked Questions

Does MediaCrawler require official Xiaohongshu API access?

No. MediaCrawler interacts with the internal XHS API endpoints (primarily edith.xiaohongshu.com) through reverse-engineered HTTP requests rather than the official public API. This approach allows data extraction without applying for developer credentials, though you must authenticate as a standard user via cookies or QR-code login handled in media_platform/xhs/login.py.

How does the crawler handle rate limiting and anti-bot measures?

The XhsCrawler class in media_platform/xhs/core.py implements request throttling and session management. It uses realistic headers and timing intervals between requests. For additional protection, you can configure proxy settings in config/xhs_config.py to distribute requests across multiple IP addresses.

Can I scrape multiple Xiaohongshu notes simultaneously?

Yes. The async architecture using asyncio allows you to gather multiple notes concurrently by creating tasks with asyncio.gather(). However, respect the platform's rate limits by controlling concurrency levels in your event loop to avoid triggering anti-bot mechanisms.

What data fields are available for extracted notes?

According to the NoteUrlInfo model in model/m_xiaohongshu.py and the API responses handled in media_platform/xhs/client.py, available fields include note title, content text, image URLs, video URLs, like count, comment count, share count, creation timestamp, and creator metadata. The exact schema depends on the specific endpoint called (get_note_detail versus get_creator_notes).

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}