# How to Use MediaCrawler for Xiaohongshu Data: Complete Setup Guide

> Learn how to use MediaCrawler for Xiaohongshu data. This complete setup guide covers QR-code authentication, URL parsing, and async data retrieval for notes and profiles.

- Repository: [程序员阿江-Relakkes/MediaCrawler](https://github.com/NanmiCoder/MediaCrawler)
- Tags: getting-started
- Published: 2026-06-26

---

**MediaCrawler extracts Xiaohongshu (XHS) data through a modular Python API that handles QR-code authentication, URL parsing, and async data retrieval, supporting both individual notes and creator profile scraping.**

MediaCrawler is an open-source scraping framework maintained at `NanmiCoder/MediaCrawler` that supports multiple Chinese social platforms. When using MediaCrawler for Xiaohongshu data collection, you leverage a dedicated XHS module consisting of specialized login handlers, URL parsers, and high-level crawler classes that interface with the platform's internal API endpoints.

## Architecture Overview

The Xiaohongshu implementation follows a layered architecture where each component handles a specific phase of the data extraction pipeline.

### Core Source Files

Understanding the repository structure helps you customize the scraping behavior:

- **[`media_platform/xhs/login.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/xhs/login.py)** – Contains QR-code, mobile-phone, and cookie-based authentication flows using Selenium and Chrome DevTools Protocol
- **[`media_platform/xhs/help.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/xhs/help.py)** – Provides URL parsing utilities that convert public Xiaohongshu links into structured data models
- **[`model/m_xiaohongshu.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/model/m_xiaohongshu.py)** – Defines **Pydantic** models `NoteUrlInfo` and `CreatorUrlInfo` that validate and store URL metadata
- **[`media_platform/xhs/client.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/xhs/client.py)** – Implements `XhsClient`, the low-level HTTP wrapper targeting `https://edith.xiaohongshu.com`
- **[`media_platform/xhs/core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/xhs/core.py)** – Houses the `XhsCrawler` class with high-level methods like `get_note_detail()` and `get_creator_notes()`
- **[`store/xhs/__init__.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/xhs/__init__.py)** – Handles persistence layer for JSON, MongoDB, or Excel output formats
- **[`main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/main.py)** – CLI entry point for command-line execution

## Configuration and Authentication

Before extracting data, configure your credentials. The system supports three authentication methods implemented in [`media_platform/xhs/login.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/xhs/login.py): QR-code scanning, mobile SMS login, or cookie reuse.

Create a `.env` file in the project root (copy from `.env.example`):

```bash
XHS_COOKIE="your_xiaohongshu_cookie_here"

```

If this value is empty, the crawler will automatically prompt for QR-code login at runtime using the Selenium-based handler.

## Parsing Xiaohongshu URLs

The framework requires structured URL objects before making API calls. Use the helper functions in [`media_platform/xhs/help.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/xhs/help.py) to transform public URLs into the internal representation defined in [`model/m_xiaohongshu.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/model/m_xiaohongshu.py).

**Key parsing functions:**

- **`parse_note_info_from_note_url(url)`** – Returns a `NoteUrlInfo` model containing the note ID and extraction metadata
- **`parse_creator_info_from_profile_url(url)`** – Returns a `CreatorUrlInfo` model with the user ID and profile parameters

These parsers validate URL patterns and extract the necessary identifiers for the XHS API endpoints.

## Implementing the Crawler

The `XhsCrawler` class in [`media_platform/xhs/core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/xhs/core.py) serves as the primary interface. It automatically manages authentication state and rate limiting when calling the internal API at `https://edith.xiaohongshu.com`.

### Fetching Individual Notes

This example demonstrates parsing a note URL and extracting title and image data:

```python
import asyncio
from media_platform.xhs.core import XhsCrawler
from media_platform.xhs.help import parse_note_info_from_note_url

# Public Xiaohongshu note URL

NOTE_URL = "https://www.xiaohongshu.com/explore/64b95d01000000000c034587"

async def fetch_single_note():
    # Parse URL into structured model

    note_info = parse_note_info_from_note_url(NOTE_URL)
    
    # Initialize crawler (handles login automatically)

    crawler = XhsCrawler()
    
    # Retrieve detailed note data

    note_detail = await crawler.get_note_detail(note_info)
    
    # Process results

    print("Title:", note_detail["title"])
    print("Images:")
    for img in note_detail.get("image_list", []):
        print(f"  - {img['url']}")

if __name__ == "__main__":
    asyncio.run(fetch_single_note())

```

### Scraping Creator Profiles

To extract a creator's publication history, use `get_creator_notes()` with pagination support:

```python
from media_platform.xhs.help import parse_creator_info_from_profile_url
from media_platform.xhs.core import XhsCrawler

CREATOR_URL = "https://www.xiaohongshu.com/user/profile/5eb8e1d400000000010075ae"

async def crawl_creator_feed():
    creator_info = parse_creator_info_from_profile_url(CREATOR_URL)
    crawler = XhsCrawler()
    
    # Fetch first page of creator's notes

    notes_page = await crawler.get_creator_notes(creator_info, page=1)
    
    for note in notes_page["notes"]:
        print(f"Note ID: {note['note_id']} - Title: {note['title']}")

# Execute with asyncio.run(crawl_creator_feed())

```

### Using the Command Line Interface

MediaCrawler includes a CLI wrapper in [`main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/main.py) for quick execution without writing Python scripts:

```bash

# Fetch specific note

python main.py xhs note --url "https://www.xiaohongshu.com/explore/64b95d01000000000c034587"

# Fetch creator profile

python main.py xhs creator --url "https://www.xiaohongshu.com/user/profile/5eb8e1d400000000010075ae"

```

The CLI parser in [`main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/main.py) routes these arguments to the `XhsCrawler` class with automatic authentication handling.

## Storage and Persistence

Extracted data flows through the storage layer defined in [`store/xhs/__init__.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/xhs/__init__.py). By default, the system supports JSON file export, MongoDB insertion, and Excel generation. You can configure the storage backend in [`config/xhs_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/xhs_config.py) alongside proxy settings and timeout parameters.

The crawler returns standard Python dictionaries that you can also process directly before persistence, enabling custom data pipelines or real-time processing workflows.

## Summary

- **Clone** the `NanmiCoder/MediaCrawler` repository and install dependencies via [`requirements.txt`](https://github.com/NanmiCoder/MediaCrawler/blob/main/requirements.txt)
- **Configure** authentication in environment variables or use QR-code login through [`media_platform/xhs/login.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/xhs/login.py)
- **Parse** target URLs using helpers in [`media_platform/xhs/help.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/xhs/help.py) to generate `NoteUrlInfo` or `CreatorUrlInfo` objects
- **Instantiate** `XhsCrawler` from [`media_platform/xhs/core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/xhs/core.py) and call async methods like `get_note_detail()` or `get_creator_notes()`
- **Store** results using the built-in storage layer in [`store/xhs/__init__.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/xhs/__init__.py) or custom backends

## Frequently Asked Questions

### Does MediaCrawler require official Xiaohongshu API access?

No. MediaCrawler interacts with the internal XHS API endpoints (primarily `edith.xiaohongshu.com`) through reverse-engineered HTTP requests rather than the official public API. This approach allows data extraction without applying for developer credentials, though you must authenticate as a standard user via cookies or QR-code login handled in [`media_platform/xhs/login.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/xhs/login.py).

### How does the crawler handle rate limiting and anti-bot measures?

The `XhsCrawler` class in [`media_platform/xhs/core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/xhs/core.py) implements request throttling and session management. It uses realistic headers and timing intervals between requests. For additional protection, you can configure proxy settings in [`config/xhs_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/xhs_config.py) to distribute requests across multiple IP addresses.

### Can I scrape multiple Xiaohongshu notes simultaneously?

Yes. The async architecture using `asyncio` allows you to gather multiple notes concurrently by creating tasks with `asyncio.gather()`. However, respect the platform's rate limits by controlling concurrency levels in your event loop to avoid triggering anti-bot mechanisms.

### What data fields are available for extracted notes?

According to the `NoteUrlInfo` model in [`model/m_xiaohongshu.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/model/m_xiaohongshu.py) and the API responses handled in [`media_platform/xhs/client.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/xhs/client.py), available fields include note title, content text, image URLs, video URLs, like count, comment count, share count, creation timestamp, and creator metadata. The exact schema depends on the specific endpoint called (`get_note_detail` versus `get_creator_notes`).