MindSpider Crawling Strategies for Social Platforms: A Technical Deep Dive

MindSpider uses three configurable crawling strategies—search, detail, and creator—applied uniformly across seven Chinese social platforms via the PlatformCrawler class in MindSpider/DeepSentimentCrawling/platform_crawler.py.

The 666ghj/bettafish repository implements MindSpider as a unified sentiment analysis system that orchestrates multi-platform data collection. Its crawling architecture leverages a platform-agnostic pipeline where the same core logic handles 小红书 (Xiaohongshu), 抖音 (Douyin), 快手 (Kuaishou), 哔哩哔哩 (Bilibili), 微博 (Weibo), 贴吧 (Tieba), and 知乎 (Zhihu) by dynamically generating configurations and invoking MediaCrawler subprocesses.

How MindSpider Structures Its Crawling Pipeline

The crawling logic centers on the PlatformCrawler class, which abstracts platform differences into a consistent execution flow. Rather than implementing separate crawlers for each social network, MindSpider injects platform-specific parameters into a standardized MediaCrawler wrapper.

Platform Discovery and Database Binding

Upon initialization, the crawler defines its supported platforms in the supported_platforms list:

self.supported_platforms = ['xhs', 'dy', 'ks', 'bili', 'wb', 'tieba', 'zhihu']

Before any crawl execution, the configure_mediacrawler_db method (lines 44-62) patches MediaCrawler's db_config.py to align with MindSpider's database instance. The system reads the dialect (mysql or postgresql) from config.settings.DB_DIALECT, ensuring all scraped data flows into the same storage backend regardless of platform.

Dynamic Configuration Generation

For each crawling operation, the create_base_config method (lines 188-197) generates a fresh base_config.py file containing:

• PLATFORM: The target platform identifier
• KEYWORDS: Comma-separated search terms
• CRAWLER_TYPE: The specific MindSpider crawling strategy (search, detail, or creator)
• SAVE_DATA_OPTION: Database backend specification ("db" for MySQL, "postgresql" for PostgreSQL)
• CRAWLER_MAX_NOTES_COUNT: Pagination limit for the crawl session

The Three Core Crawling Strategies

MindSpider distinguishes between three distinct data collection modes through the CRAWLER_TYPE parameter injected at line 192 of platform_crawler.py. These strategies determine how MediaCrawler interacts with each platform's API or frontend.

Search Strategy

The search strategy performs keyword-based content discovery across platforms. When CRAWLER_TYPE is set to "search", MediaCrawler retrieves posts, notes, or videos matching the supplied keywords. This is the default mode for broad sentiment monitoring and trend analysis, fetching up to CRAWLER_MAX_NOTES_COUNT items per keyword.

Detail Strategy

The detail strategy retrieves comprehensive metadata for specific content items. When deeper inspection is required beyond search result snippets, setting CRAWLER_TYPE to "detail" fetches full post content, comment threads, and engagement statistics for individual notes or videos identified by their unique identifiers.

Creator Strategy

The creator strategy focuses on user profile extraction rather than content. By setting CRAWLER_TYPE to "creator", the crawler targets author information, follower counts, and historical posting patterns. This mode treats keywords as creator identifiers or search terms for discovering specific user accounts across platforms.

Multi-Platform Execution and Aggregation

The PlatformCrawler class manages the lifecycle of crawling operations through subprocess orchestration and state monitoring.

Subprocess Management and Timeouts

The run_crawler method (lines 58-66) constructs and executes shell commands to invoke MediaCrawler/main.py with arguments:

• --platform: Target platform identifier
• --lt: Login type (defaults to "qrcode")
• --type: The crawling strategy (search/detail/creator)
• --save_data_option: Database target

Each subprocess receives a strict 60-minute timeout to prevent hanging operations on slow-responsive platforms.

Login Handling and QR Code Authentication

MindSpider handles authentication through the _parse_crawl_output method (lines 38-40), which monitors subprocess output for Chinese login prompts ("登录" or "扫码"). When detected, the system flags login_required for subsequent handling, allowing the QR code authentication flow to complete before data collection begins.

Result Aggregation and Statistics

After each platform crawl completes, run_crawler (lines 80-89) records performance metrics into self.crawl_stats, including:

• Execution duration
• Number of notes/posts collected
• Comment count totals
• Return code status

The run_multi_platform_crawl_by_keywords method (lines 71-84) aggregates these per-platform statistics into a global summary, enabling comparative analysis of data yields across different social networks.

Code Examples

Running Keyword Searches Across Multiple Platforms

This example demonstrates the default search strategy across Xiaohongshu and Douyin simultaneously:

from MindSpider.DeepSentimentCrawling.platform_crawler import PlatformCrawler

crawler = PlatformCrawler()

# Define target keywords

keywords = ["美妆", "旅行", "健康"]

# Execute multi-platform crawl with search strategy

stats = crawler.run_multi_platform_crawl_by_keywords(
    keywords=keywords,
    platforms=["xhs", "dy"],          # Target Xiaohongshu and Douyin

    login_type="qrcode",               # QR code authentication

    max_notes_per_keyword=30           # Limit to 30 items per keyword

)

print("Overall summary:", stats)

Extracting Creator Profiles from Specific Platforms

This example overrides the default configuration to use the creator strategy for Zhihu:

from MindSpider.DeepSentimentCrawling.platform_crawler import PlatformCrawler

crawler = PlatformCrawler()
platform = "zhihu"
keywords = ["人工智能"]   # Interpreted as creator search terms

# Generate config with creator strategy

crawler.create_base_config(
    platform=platform, 
    keywords=keywords, 
    crawler_type="creator", 
    max_notes=10
)

# Execute crawl

creator_stats = crawler.run_crawler(
    platform=platform,
    keywords=keywords,
    login_type="qrcode",
    max_notes=10
)

print("Creator crawl result:", creator_stats)

Summary

• MindSpider implements a unified crawling architecture through the PlatformCrawler class in MindSpider/DeepSentimentCrawling/platform_crawler.py.
• Three distinct crawling strategies (search, detail, creator) control data collection behavior across all supported platforms.
• The system supports seven Chinese social platforms: Xiaohongshu, Douyin, Kuaishou, Bilibili, Weibo, Tieba, and Zhihu.
• Dynamic configuration generation via create_base_config allows runtime strategy switching without code modifications.
• Database alignment ensures consistent storage across MySQL and PostgreSQL backends through runtime patching of MediaCrawler's configuration.
• Subprocess orchestration with 60-minute timeouts and QR code login handling provides robust automation for long-running crawls.

Frequently Asked Questions

What social platforms does MindSpider support?

MindSpider supports seven major Chinese social platforms: 小红书 (Xiaohongshu/xhs), 抖音 (Douyin/dy), 快手 (Kuaishou/ks), 哔哩哔哩 (Bilibili/bili), 微博 (Weibo/wb), 贴吧 (Tieba/tieba), and 知乎 (Zhihu/zhihu). These are defined in the supported_platforms list within platform_crawler.py.

How does MindSpider handle database configuration?

The configure_mediacrawler_db method dynamically patches MediaCrawler's db_config.py to point to MindSpider's database instance. It reads the dialect (MySQL or PostgreSQL) from config.settings.DB_DIALECT and ensures all platforms write to the same centralized storage, unifying data from different social networks.

What is the difference between search and detail crawling strategies?

The search strategy performs broad keyword-based discovery, fetching multiple posts or videos matching search terms. The detail strategy retrieves comprehensive metadata for specific content items identified by unique IDs, including full text and comment threads. Search is used for monitoring trends, while detail is used for deep analysis of specific content.

How does MindSpider manage login authentication?

MindSpider defaults to QR code login (login_type="qrcode") for all platforms. The _parse_crawl_output method monitors subprocess output for Chinese login prompts ("登录" or "扫码"). When detected, the system flags the requirement and pauses execution until authentication completes, preventing failed crawls due to session expiration.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}