What Kind of Data Can Be Extracted by MediaCrawler: Content, Comments, and Creator Profiles

MediaCrawler extracts three structured entity types—content (posts, articles, and videos), comments (including nested replies), and creator profiles—from major Chinese social media platforms, normalizing them into Pydantic models for unified storage across CSV, JSON, SQL, and NoSQL backends.

MediaCrawler is an open-source multi-platform scraper designed to systematically harvest social media data using Playwright and Chrome DevTools Protocol. According to the source code in NanmiCoder/MediaCrawler, the tool implements a generic extraction architecture that normalizes heterogeneous platform data into three consistent entity types. This design allows downstream analytics to treat output as a homogeneous dataset regardless of whether the source is Zhihu, Baidu Tieba, Xiaohongshu, or Douyin.

The Three Core Data Entities

Content Data

The primary publishable material—whether a Zhihu answer, Tieba thread, Xiaohongshu note, or Douyin video—is captured in platform-specific Pydantic models. In model/m_zhihu.py, the ZhihuContent class defines fields including title, desc, publish_time, url, and voteup_count. Baidu Tieba implements equivalent functionality through TiebaNote in model/m_baidu_tieba.py, while Xiaohongshu and Douyin use NoteUrlInfo and VideoUrlInfo respectively in their model files.

Comment Data

Discussion threads and replies are extracted as structured comment objects with support for nested hierarchies. The ZhihuComment model in model/m_zhihu.py captures comment_id, content, publish_time, like_count, and sub_comment_count to track reply chains. Baidu Tieba implements TiebaComment in model/m_baidu_tieba.py with equivalent metadata fields.

Creator Information

Author metadata is extracted via creator-specific models including ZhihuCreator and TiebaCreator in their respective model files, or CreatorUrlInfo for Xiaohongshu and Douyin in model/m_xiaohongshu.py and model/m_douyin.py. These models standardize user_id, user_nickname, avatar, follows, fans, and gender demographics across platforms.

Data Extraction Architecture

Crawler implementations inherit from AbstractCrawler defined in base/base_crawler.py, with concrete classes like ZhihuCrawler and TieBaCrawler handling platform-specific authentication and page fetching via Playwright or CDP. After retrieval, raw HTTP responses are parsed and validated against the Pydantic models listed above.

The architecture enforces strict separation between extraction and persistence. Validated model instances are converted to dictionaries and passed to storage implementations via the AbstractStore interface, which requires three standardized methods: store_content, store_comment, and store_creator. This abstraction ensures that extraction logic remains independent of storage backends.

Supported Storage Backends

MediaCrawler ships with multiple storage implementations demonstrated in store/zhihu/_store_impl.py. The AbstractStore interface supports:

• CSV/Excel: Appending records via AsyncFileWriter.write_to_csv in tools/async_file_writer.py
• JSON/JSONL: Line-delimited formats for streaming data pipelines
• SQLite: Lightweight relational storage for local development
• MySQL: Production-grade relational persistence
• MongoDB: Document storage via MongoDBStoreBase.save_or_update in database/mongodb_store_base.py

Practical Code Examples

Storing Zhihu Content in MongoDB

from model.m_zhihu import ZhihuContent
from store.zhihu._store_impl import ZhihuMongoStoreImplement

# Build a content object (validation occurs automatically)

content = ZhihuContent(
    content_id="12345",
    content_type="answer",
    content_text="Python is great!",
    title="Why Python?",
    voteup_count=42,
    user_id="u6789",
    user_nickname="alice",
)

# Persist it (the store expects a plain dict)

store = ZhihuMongoStoreImplement()
await store.store_content(content.dict())

The store_content method internally calls MongoDBStoreBase.save_or_update, writing the document into the zhihu_contents collection.

Exporting Tieba Comments to CSV

from model.m_baidu_tieba import TiebaComment
from store.tieba._store_impl import TieBaCsvStoreImplement

comment = TiebaComment(
    comment_id="cmt987",
    content="Nice post!",
    publish_time="2024-09-01 12:00",
    user_nickname="bob",
    note_id="note123",
    tieba_name="Python吧"
)

csv_store = TieBaCsvStoreImplement()
await csv_store.store_comment(comment.dict())

The CSV implementation uses AsyncFileWriter.write_to_csv to append structured data to the appropriate platform-specific file.

Handling Creator Metadata

from model.m_xiaohongshu import CreatorUrlInfo

creator = CreatorUrlInfo(user_id="u001", xsec_token="abc", xsec_source="web")
print(creator.json())

The CreatorUrlInfo model stores URL ingredients required to request additional profile data from the platform API.

Summary

• MediaCrawler extracts three normalized entity types: content, comments, and creator profiles
• Platform-specific Pydantic models in model/m_zhihu.py, model/m_baidu_tieba.py, model/m_xiaohongshu.py, and model/m_douyin.py validate incoming data
• The AbstractCrawler base class in base/base_crawler.py provides the generic extraction framework
• Storage implementations support CSV, JSON, Excel, SQLite, MySQL, and MongoDB via the AbstractStore interface
• All extracted data maintains consistent field naming across platforms, enabling homogeneous downstream analysis

Frequently Asked Questions

Does MediaCrawler extract video-specific metadata?

Yes, video platforms like Douyin use the VideoUrlInfo model in model/m_douyin.py to capture playback URLs, video statistics, and engagement metrics, treating them as content entities while preserving video-specific attributes.

Can MediaCrawler handle nested comment threads?

Yes, comment models include sub_comment_count fields to track reply depth, and the extraction logic supports recursive parsing of nested comment chains as implemented in ZhihuComment and TiebaComment.

What storage formats does MediaCrawler support for extracted data?

The tool supports CSV, JSON/JSONL, Excel, SQLite, MySQL, and MongoDB through the AbstractStore interface, with concrete implementations located in store/ directories (e.g., store/zhihu/_store_impl.py).

How does MediaCrawler authenticate to extract private creator data?

Crawler classes utilize Playwright and Chrome DevTools Protocol (CDP) for browser automation, managing cookies, session tokens, and anti-detection measures required to access user profile data and comment sections that require authentication.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}