How MindSpider Crawls Social Media Data from Weibo, Douyin, and Xiaohongshu

MindSpider uses a modular DeepSentimentCrawling pipeline that configures platform-specific MediaCrawler instances with Playwright automation to extract posts, comments, and metadata from Weibo, Douyin, and Xiaohongshu, storing results in dedicated SQLAlchemy ORM tables.

MindSpider (also referred to as Mind Spider) is an AI-driven opinion-mining system within the 666ghj/bettafish repository. Its DeepSentimentCrawling stage orchestrates the entire process of collecting raw social media data from Chinese platforms, handling everything from keyword discovery to structured database persistence.

The DeepSentimentCrawling Architecture

The crawling workflow begins with topic discovery and ends with structured data storage. The PlatformCrawler class in MindSpider/DeepSentimentCrawling/platform_crawler.py serves as the central orchestrator, managing the hand-off between keyword generation, platform configuration, and execution.

Keyword Generation and Topic Discovery

Before any browser automation starts, the BroadTopicExtraction module identifies trending topics. The script MindSpider/BroadTopicExtraction/get_today_news.py scrapes current events and populates the daily_topics table with keywords. These keywords are then fed into the crawling pipeline to target specific conversations across social platforms.

Platform Selection and Configuration

The PlatformCrawler maintains a registry of supported platforms in the supported_platforms attribute (line 33 of platform_crawler.py). This dictionary maps platform codes to their configurations:

• "wb" → Weibo
• "dy" → Douyin
• "xhs" → Xiaohongshu (XHS)

When run_multi_platform_crawl_by_keywords() is invoked, it iterates over this list to configure and launch platform-specific crawlers sequentially.

How MindSpider Configures the MediaCrawler

MindSpider does not reinvent the browser automation layer. Instead, it wraps the open-source MediaCrawler project, injecting configuration dynamically to ensure data flows into MindSpider's own database schema rather than MediaCrawler's defaults.

Database Binding and Persistence Setup

The method configure_mediacrawler_db() (lines 44-64 in platform_crawler.py) rewrites the db_config.py file inside the MediaCrawler submodule. This ensures that all scraped items—posts, comments, and metadata—are persisted to MindSpider's MySQL or PostgreSQL instance using the connection credentials defined in MindSpider/config.py.

Platform-Specific Base Configuration

For each platform run, create_base_config() (lines 68-98) generates a temporary base_config.py tailored to the target site. This configuration specifies:

• Platform type: xhs, dy, or wb
• Crawler mode: search (keyword-based)
• Keyword list: Injected from the daily topics
• Storage mode: db or postgresql (matching the persistence layer)
• Note limits: Maximum posts to retrieve per keyword

Executing the Crawl with Playwright Automation

Once configuration is complete, MindSpider launches the actual scraping process via subprocess execution, utilizing Playwright to control headless Chrome instances.

Subprocess Execution and Headless Browser Control

The run_crawler() method (lines 118-78 in platform_crawler.py) constructs a command-line invocation targeting MediaCrawler/main.py. Playwright drives the real UI of each platform:

1. Opens the target website (Weibo.com, Douyin.com, or XHS)
2. Handles authentication via QR-code scan or cookie reuse
3. Executes keyword searches
4. Scrolls through infinite-scroll feeds
5. Extracts JSON payloads from network responses or DOM parsing

Data Mapping to ORM Models

Raw scraped data is automatically mapped to MindSpider's SQLAlchemy ORM classes defined in MindSpider/schema/models_bigdata.py:

• Weibo: WeiboNote (line 218) and WeiboNoteComment store posts and replies
• Douyin: DouyinAweme (line 111) and DouyinAwemeComment handle videos and comments
• Xiaohongshu: XhsNote (line 295) and XhsNoteComment capture notes and discussions

Practical Code Examples

Single Platform Crawl (Weibo)

from MindSpider.DeepSentimentCrawling.platform_crawler import PlatformCrawler

# Initialize the crawler manager

crawler = PlatformCrawler()

# Define target keywords

keywords = ["AI", "机器学习", "ChatGPT"]

# Execute Weibo crawl (platform code "wb")

stats = crawler.run_crawler(
    platform="wb",
    keywords=keywords,
    login_type="qrcode",
    max_notes=100
)

print(f"Weibo crawl completed: {stats}")

Multi-Platform Batch Crawl

from MindSpider.DeepSentimentCrawling.platform_crawler import PlatformCrawler

crawler = PlatformCrawler()

keywords = ["新能源", "5G", "元宇宙"]
platforms = ["wb", "dy", "xhs"]  # Weibo, Douyin, Xiaohongshu

# Run sequential crawl across all platforms

results = crawler.run_multi_platform_crawl_by_keywords(
    keywords=keywords,
    platforms=platforms,
    login_type="qrcode",
    max_notes_per_keyword=50
)

print(f"Batch crawl statistics: {results}")

Querying Crawled Data via SQLAlchemy

from MindSpider.schema.models_bigdata import WeiboNote
from MindSpider.schema.db_manager import SessionLocal

session = SessionLocal()

# Retrieve latest Weibo posts

latest_posts = (
    session.query(WeiboNote)
    .order_by(WeiboNote.create_time.desc())
    .limit(10)
    .all()
)

for post in latest_posts:
    print(f"{post.nickname}: {post.content[:100]}...")
    
session.close()

Summary

• MindSpider orchestrates social media crawling through the PlatformCrawler class in MindSpider/DeepSentimentCrawling/platform_crawler.py.
• The system supports Weibo (wb), Douyin (dy), and Xiaohongshu (xhs) through a unified configuration interface.
• Playwright automation handles QR-code login, keyword search, and infinite-scroll extraction via the embedded MediaCrawler submodule.
• Data persists to MySQL/PostgreSQL through SQLAlchemy models defined in MindSpider/schema/models_bigdata.py, with dedicated tables for each platform's content and comments.
• The modular architecture allows batch crawling across multiple platforms using run_multi_platform_crawl_by_keywords().

Frequently Asked Questions

What authentication methods does MindSpider support for social media platforms?

MindSpider supports QR-code authentication and cookie reuse through its Playwright-based automation layer. On first execution, the crawler launches a headless browser that displays a QR code for manual scanning via the target platform's mobile app. Subsequent runs automatically reuse stored session cookies to bypass repeated authentication, as implemented in the MediaCrawler submodule invoked by PlatformCrawler.run_crawler().

How does MindSpider handle rate limiting and anti-bot measures?

The crawling engine utilizes Playwright with headless Chrome to mimic genuine user behavior, including realistic scrolling patterns, random delays between actions, and proper header injection. The underlying MediaCrawler submodule manages request throttling and implements retry logic for failed extractions. While specific rate-limit parameters are configurable through the generated base_config.py files, the system prioritizes platform compliance by spacing out requests and respecting robots.txt directives where applicable.

Can I extend MindSpider to crawl additional social media platforms?

Yes, the architecture is designed for modular extensibility. Adding a new platform requires creating a corresponding crawler implementation in the MediaCrawler submodule under media_platform/, then registering the platform code in PlatformCrawler.supported_platforms (line 33 of platform_crawler.py). The database layer automatically accommodates new platforms through SQLAlchemy model definitions in MindSpider/schema/models_bigdata.py, while the configuration pipeline in create_base_config() handles platform-specific settings without requiring changes to the core orchestration logic.

What is the difference between MindSpider and the MediaCrawler submodule?

MindSpider is the high-level opinion-mining framework that manages keyword generation, database persistence, and downstream analytics. The MediaCrawler submodule is a specialized browser-automation library focused solely on extracting data from specific social media platforms. MindSpider wraps MediaCrawler by dynamically generating configuration files (base_config.py, db_config.py) and invoking it as a subprocess via PlatformCrawler.run_crawler(), then mapping the output to its own SQLAlchemy models for unified storage and analysis.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}