MediaCrawler Crawler Modes: Search, Detail, and Creator Explained

MediaCrawler supports three distinct crawler modes—search, detail, and creator—that determine whether the tool scrapes keyword-based results, fetches specific content by ID, or harvests entire creator profiles.

MediaCrawler is an open-source multi-platform content scraping framework that unifies data collection across social media sites through a standardized mode system. The crawler modes in MediaCrawler define the fundamental workflow for every supported platform, from Zhihu and Weibo to XHS and Douyin. These modes are enforced through the CrawlerTypeEnum enumeration and drive which extraction logic executes at runtime.

Understanding the Three MediaCrawler Crawler Modes

Search Mode

Search mode performs keyword-based discovery across the chosen platform. It aggregates content items—such as posts, videos, or answers—that match the supplied keywords.

Activate this mode by passing --type search along with the --keywords argument. For example, to find Zhihu content about artificial intelligence:

media_crawler \
    --platform zhihu \
    --type search \
    --keywords "artificial intelligence"

Detail Mode

Detail mode retrieves complete metadata for specific content items when you already know their identifiers. This mode bypasses discovery and fetches full detail for one or more IDs supplied via --ids (or specified_ids in the API).

Activate this mode with --type detail:

media_crawler \
    --platform douyin \
    --type detail \
    --ids 7123456789,7123456790

Creator Mode

Creator mode enables profile-centric harvesting. It crawls a creator’s public profile and all associated content by accepting user identifiers via --creator_ids.

Activate this mode with --type creator:

media_crawler \
    --platform weibo \
    --type creator \
    --creator_ids 12345678

How Crawler Modes Work in the Source Code

According to the MediaCrawler source code, the mode system is type-safe and enforced through Python enumerations and context variables.

The CrawlerTypeEnum class in cmd_arg/arg.py (lines 60-66) defines the available modes:

class CrawlerTypeEnum(str, Enum):
    """Crawler type enumeration"""
    SEARCH = "search"
    DETAIL = "detail"
    CREATOR = "creator"

When the application initializes, the selected mode is stored in crawler_type_var (defined in var.py) and consumed by platform-specific implementations. For instance, in media_platform/zhihu/core.py (lines 135-144), the core logic branches based on config.CRAWLER_TYPE:

if config.CRAWLER_TYPE == "search":
    await self.batch_search(...)
elif config.CRAWLER_TYPE == "detail":
    await self.batch_get_content_detail(...)
elif config.CRAWLER_TYPE == "creator":
    await self.batch_get_creator_detail(...)

This pattern repeats across all platform modules—including media_platform/xhs/core.py and media_platform/weibo/core.py—ensuring consistent behavior whether scraping Zhihu answers or Douyin videos. The api/services/crawler_manager.py module translates CLI arguments and API requests into this runtime configuration, while storage implementations in store/*/_store_impl.py incorporate the mode into output filenames (e.g., search_comments_2024-04-01.json).

Practical Examples: Running MediaCrawler in Each Mode

CLI Examples

Search mode with JSON output:

media_crawler \
    --platform zhihu \
    --type search \
    --keywords "deep learning, neural networks" \
    --output json

Detail mode with CSV output:

media_crawler \
    --platform douyin \
    --type detail \
    --ids 1234567890123456789,1234567890123456790 \
    --output csv

Creator mode with MongoDB output:

media_crawler \
    --platform weibo \
    --type creator \
    --creator_ids 987654321 \
    --output mongodb

Programmatic API Usage

You can also trigger modes via the Python API using CrawlerStartRequest:

from api.schemas import CrawlerStartRequest, PlatformEnum, CrawlerTypeEnum

req = CrawlerStartRequest(
    platform=PlatformEnum.ZHIHU,
    crawler_type=CrawlerTypeEnum.DETAIL,
    specified_ids=["12345678"],
    output="json"
)

Summary

• Search mode performs keyword-based discovery across platforms, specified with --type search and --keywords, invoking batch_search() in platform core implementations.
• Detail mode retrieves specific content by ID using --type detail and --ids, executing batch_get_content_detail() logic defined in media_platform/<platform>/core.py.
• Creator mode harvests entire user profiles via --type creator and --creator_ids, triggering batch_get_creator_detail() methods.
• All modes are type-enforced through CrawlerTypeEnum in cmd_arg/arg.py and stored in the crawler_type_var context variable at runtime.

Frequently Asked Questions

What is the difference between search and detail modes in MediaCrawler?

Search mode discovers new content using keywords across the platform, while detail mode fetches complete metadata for already-known content IDs. Search mode calls batch_search() in platform implementations, whereas detail mode triggers batch_get_content_detail().

How do I specify which crawler mode to use in MediaCrawler?

Set the mode using the --type CLI argument (e.g., --type creator) or the crawler_type field when using the programmatic API. The value must match one of the three CrawlerTypeEnum values defined in cmd_arg/arg.py: search, detail, or creator.

Can I use creator mode to fetch all posts from a specific Weibo or Xiaohongshu account?

Yes. Creator mode is designed specifically for profile-centric harvesting. Pass the creator's user ID via --creator_ids when running from the CLI, or populate creator_ids in the API request. The crawler will execute batch_get_creator_detail() to retrieve all public content from that account.

Where is the crawler mode stored during execution?

The selected mode is stored in crawler_type_var, a context variable defined in var.py. Platform-specific core modules in media_platform/<platform>/core.py read this variable through config.CRAWLER_TYPE to determine which extraction branch to execute.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}