# How MindSpider Crawls Social Media Data from Weibo, Douyin, and Xiaohongshu

> Learn how MindSpider crawls Weibo, Douyin, and Xiaohongshu. Discover its DeepSentimentCrawling pipeline using Playwright for efficient social media data extraction and storage.

- Repository: [BaiFu/bettafish](https://github.com/666ghj/bettafish)
- Tags: how-to-guide
- Published: 2026-02-23

---

**MindSpider uses a modular DeepSentimentCrawling pipeline that configures platform-specific MediaCrawler instances with Playwright automation to extract posts, comments, and metadata from Weibo, Douyin, and Xiaohongshu, storing results in dedicated SQLAlchemy ORM tables.**

MindSpider (also referred to as *Mind Spider*) is an AI-driven opinion-mining system within the `666ghj/bettafish` repository. Its **DeepSentimentCrawling** stage orchestrates the entire process of collecting raw social media data from Chinese platforms, handling everything from keyword discovery to structured database persistence.

## The DeepSentimentCrawling Architecture

The crawling workflow begins with topic discovery and ends with structured data storage. The `PlatformCrawler` class in [`MindSpider/DeepSentimentCrawling/platform_crawler.py`](https://github.com/666ghj/bettafish/blob/main/MindSpider/DeepSentimentCrawling/platform_crawler.py) serves as the central orchestrator, managing the hand-off between keyword generation, platform configuration, and execution.

### Keyword Generation and Topic Discovery

Before any browser automation starts, the `BroadTopicExtraction` module identifies trending topics. The script [`MindSpider/BroadTopicExtraction/get_today_news.py`](https://github.com/666ghj/bettafish/blob/main/MindSpider/BroadTopicExtraction/get_today_news.py) scrapes current events and populates the `daily_topics` table with keywords. These keywords are then fed into the crawling pipeline to target specific conversations across social platforms.

### Platform Selection and Configuration

The `PlatformCrawler` maintains a registry of supported platforms in the `supported_platforms` attribute (line 33 of [`platform_crawler.py`](https://github.com/666ghj/bettafish/blob/main/platform_crawler.py)). This dictionary maps platform codes to their configurations:

- `"wb"` → Weibo
- `"dy"` → Douyin
- `"xhs"` → Xiaohongshu (XHS)

When `run_multi_platform_crawl_by_keywords()` is invoked, it iterates over this list to configure and launch platform-specific crawlers sequentially.

## How MindSpider Configures the MediaCrawler

MindSpider does not reinvent the browser automation layer. Instead, it wraps the open-source **MediaCrawler** project, injecting configuration dynamically to ensure data flows into MindSpider's own database schema rather than MediaCrawler's defaults.

### Database Binding and Persistence Setup

The method `configure_mediacrawler_db()` (lines 44-64 in [`platform_crawler.py`](https://github.com/666ghj/bettafish/blob/main/platform_crawler.py)) rewrites the [`db_config.py`](https://github.com/666ghj/bettafish/blob/main/db_config.py) file inside the MediaCrawler submodule. This ensures that all scraped items—posts, comments, and metadata—are persisted to MindSpider's MySQL or PostgreSQL instance using the connection credentials defined in [`MindSpider/config.py`](https://github.com/666ghj/bettafish/blob/main/MindSpider/config.py).

### Platform-Specific Base Configuration

For each platform run, `create_base_config()` (lines 68-98) generates a temporary [`base_config.py`](https://github.com/666ghj/bettafish/blob/main/base_config.py) tailored to the target site. This configuration specifies:

- **Platform type**: `xhs`, `dy`, or `wb`
- **Crawler mode**: `search` (keyword-based)
- **Keyword list**: Injected from the daily topics
- **Storage mode**: `db` or `postgresql` (matching the persistence layer)
- **Note limits**: Maximum posts to retrieve per keyword

## Executing the Crawl with Playwright Automation

Once configuration is complete, MindSpider launches the actual scraping process via subprocess execution, utilizing Playwright to control headless Chrome instances.

### Subprocess Execution and Headless Browser Control

The `run_crawler()` method (lines 118-78 in [`platform_crawler.py`](https://github.com/666ghj/bettafish/blob/main/platform_crawler.py)) constructs a command-line invocation targeting [`MediaCrawler/main.py`](https://github.com/666ghj/bettafish/blob/main/MediaCrawler/main.py). Playwright drives the real UI of each platform:

1. Opens the target website (Weibo.com, Douyin.com, or XHS)
2. Handles authentication via QR-code scan or cookie reuse
3. Executes keyword searches
4. Scrolls through infinite-scroll feeds
5. Extracts JSON payloads from network responses or DOM parsing

### Data Mapping to ORM Models

Raw scraped data is automatically mapped to MindSpider's SQLAlchemy ORM classes defined in [`MindSpider/schema/models_bigdata.py`](https://github.com/666ghj/bettafish/blob/main/MindSpider/schema/models_bigdata.py):

- **Weibo**: `WeiboNote` (line 218) and `WeiboNoteComment` store posts and replies
- **Douyin**: `DouyinAweme` (line 111) and `DouyinAwemeComment` handle videos and comments
- **Xiaohongshu**: `XhsNote` (line 295) and `XhsNoteComment` capture notes and discussions

## Practical Code Examples

### Single Platform Crawl (Weibo)

```python
from MindSpider.DeepSentimentCrawling.platform_crawler import PlatformCrawler

# Initialize the crawler manager

crawler = PlatformCrawler()

# Define target keywords

keywords = ["AI", "机器学习", "ChatGPT"]

# Execute Weibo crawl (platform code "wb")

stats = crawler.run_crawler(
    platform="wb",
    keywords=keywords,
    login_type="qrcode",
    max_notes=100
)

print(f"Weibo crawl completed: {stats}")

```

### Multi-Platform Batch Crawl

```python
from MindSpider.DeepSentimentCrawling.platform_crawler import PlatformCrawler

crawler = PlatformCrawler()

keywords = ["新能源", "5G", "元宇宙"]
platforms = ["wb", "dy", "xhs"]  # Weibo, Douyin, Xiaohongshu

# Run sequential crawl across all platforms

results = crawler.run_multi_platform_crawl_by_keywords(
    keywords=keywords,
    platforms=platforms,
    login_type="qrcode",
    max_notes_per_keyword=50
)

print(f"Batch crawl statistics: {results}")

```

### Querying Crawled Data via SQLAlchemy

```python
from MindSpider.schema.models_bigdata import WeiboNote
from MindSpider.schema.db_manager import SessionLocal

session = SessionLocal()

# Retrieve latest Weibo posts

latest_posts = (
    session.query(WeiboNote)
    .order_by(WeiboNote.create_time.desc())
    .limit(10)
    .all()
)

for post in latest_posts:
    print(f"{post.nickname}: {post.content[:100]}...")
    
session.close()

```

## Summary

- **MindSpider** orchestrates social media crawling through the `PlatformCrawler` class in [`MindSpider/DeepSentimentCrawling/platform_crawler.py`](https://github.com/666ghj/bettafish/blob/main/MindSpider/DeepSentimentCrawling/platform_crawler.py).
- The system supports **Weibo** (`wb`), **Douyin** (`dy`), and **Xiaohongshu** (`xhs`) through a unified configuration interface.
- **Playwright** automation handles QR-code login, keyword search, and infinite-scroll extraction via the embedded MediaCrawler submodule.
- Data persists to **MySQL/PostgreSQL** through SQLAlchemy models defined in [`MindSpider/schema/models_bigdata.py`](https://github.com/666ghj/bettafish/blob/main/MindSpider/schema/models_bigdata.py), with dedicated tables for each platform's content and comments.
- The modular architecture allows batch crawling across multiple platforms using `run_multi_platform_crawl_by_keywords()`.

## Frequently Asked Questions

### What authentication methods does MindSpider support for social media platforms?

MindSpider supports **QR-code authentication** and **cookie reuse** through its Playwright-based automation layer. On first execution, the crawler launches a headless browser that displays a QR code for manual scanning via the target platform's mobile app. Subsequent runs automatically reuse stored session cookies to bypass repeated authentication, as implemented in the MediaCrawler submodule invoked by `PlatformCrawler.run_crawler()`.

### How does MindSpider handle rate limiting and anti-bot measures?

The crawling engine utilizes **Playwright** with headless Chrome to mimic genuine user behavior, including realistic scrolling patterns, random delays between actions, and proper header injection. The underlying MediaCrawler submodule manages request throttling and implements retry logic for failed extractions. While specific rate-limit parameters are configurable through the generated [`base_config.py`](https://github.com/666ghj/bettafish/blob/main/base_config.py) files, the system prioritizes platform compliance by spacing out requests and respecting robots.txt directives where applicable.

### Can I extend MindSpider to crawl additional social media platforms?

Yes, the architecture is designed for **modular extensibility**. Adding a new platform requires creating a corresponding crawler implementation in the MediaCrawler submodule under `media_platform/`, then registering the platform code in `PlatformCrawler.supported_platforms` (line 33 of [`platform_crawler.py`](https://github.com/666ghj/bettafish/blob/main/platform_crawler.py)). The database layer automatically accommodates new platforms through SQLAlchemy model definitions in [`MindSpider/schema/models_bigdata.py`](https://github.com/666ghj/bettafish/blob/main/MindSpider/schema/models_bigdata.py), while the configuration pipeline in `create_base_config()` handles platform-specific settings without requiring changes to the core orchestration logic.

### What is the difference between MindSpider and the MediaCrawler submodule?

**MindSpider** is the high-level opinion-mining framework that manages keyword generation, database persistence, and downstream analytics. The **MediaCrawler** submodule is a specialized browser-automation library focused solely on extracting data from specific social media platforms. MindSpider wraps MediaCrawler by dynamically generating configuration files ([`base_config.py`](https://github.com/666ghj/bettafish/blob/main/base_config.py), [`db_config.py`](https://github.com/666ghj/bettafish/blob/main/db_config.py)) and invoking it as a subprocess via `PlatformCrawler.run_crawler()`, then mapping the output to its own SQLAlchemy models for unified storage and analysis.