# What Kind of Data Can Be Extracted by MediaCrawler: Content, Comments, and Creator Profiles

> Discover the data MediaCrawler extracts: content, comments, and creator profiles from Chinese social media. Normalize and store data in CSV, JSON, SQL, or NoSQL.

- Repository: [程序员阿江-Relakkes/MediaCrawler](https://github.com/NanmiCoder/MediaCrawler)
- Tags: getting-started
- Published: 2026-06-28

---

**MediaCrawler extracts three structured entity types—content (posts, articles, and videos), comments (including nested replies), and creator profiles—from major Chinese social media platforms, normalizing them into Pydantic models for unified storage across CSV, JSON, SQL, and NoSQL backends.**

MediaCrawler is an open-source multi-platform scraper designed to systematically harvest social media data using Playwright and Chrome DevTools Protocol. According to the source code in `NanmiCoder/MediaCrawler`, the tool implements a generic extraction architecture that normalizes heterogeneous platform data into three consistent entity types. This design allows downstream analytics to treat output as a homogeneous dataset regardless of whether the source is Zhihu, Baidu Tieba, Xiaohongshu, or Douyin.

## The Three Core Data Entities

### Content Data

The primary publishable material—whether a Zhihu answer, Tieba thread, Xiaohongshu note, or Douyin video—is captured in platform-specific Pydantic models. In [`model/m_zhihu.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/model/m_zhihu.py), the **`ZhihuContent`** class defines fields including `title`, `desc`, `publish_time`, `url`, and `voteup_count`. Baidu Tieba implements equivalent functionality through **`TiebaNote`** in [`model/m_baidu_tieba.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/model/m_baidu_tieba.py), while Xiaohongshu and Douyin use **`NoteUrlInfo`** and **`VideoUrlInfo`** respectively in their model files.

### Comment Data

Discussion threads and replies are extracted as structured comment objects with support for nested hierarchies. The **`ZhihuComment`** model in [`model/m_zhihu.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/model/m_zhihu.py) captures `comment_id`, `content`, `publish_time`, `like_count`, and `sub_comment_count` to track reply chains. Baidu Tieba implements **`TiebaComment`** in [`model/m_baidu_tieba.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/model/m_baidu_tieba.py) with equivalent metadata fields.

### Creator Information

Author metadata is extracted via creator-specific models including **`ZhihuCreator`** and **`TiebaCreator`** in their respective model files, or **`CreatorUrlInfo`** for Xiaohongshu and Douyin in [`model/m_xiaohongshu.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/model/m_xiaohongshu.py) and [`model/m_douyin.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/model/m_douyin.py). These models standardize `user_id`, `user_nickname`, `avatar`, `follows`, `fans`, and `gender` demographics across platforms.

## Data Extraction Architecture

Crawler implementations inherit from **`AbstractCrawler`** defined in [`base/base_crawler.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/base/base_crawler.py), with concrete classes like `ZhihuCrawler` and `TieBaCrawler` handling platform-specific authentication and page fetching via Playwright or CDP. After retrieval, raw HTTP responses are parsed and validated against the Pydantic models listed above.

The architecture enforces strict separation between extraction and persistence. Validated model instances are converted to dictionaries and passed to storage implementations via the **`AbstractStore`** interface, which requires three standardized methods: `store_content`, `store_comment`, and `store_creator`. This abstraction ensures that extraction logic remains independent of storage backends.

## Supported Storage Backends

MediaCrawler ships with multiple storage implementations demonstrated in [`store/zhihu/_store_impl.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/zhihu/_store_impl.py). The **`AbstractStore`** interface supports:

- **CSV/Excel**: Appending records via `AsyncFileWriter.write_to_csv` in [`tools/async_file_writer.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/async_file_writer.py)
- **JSON/JSONL**: Line-delimited formats for streaming data pipelines
- **SQLite**: Lightweight relational storage for local development
- **MySQL**: Production-grade relational persistence
- **MongoDB**: Document storage via `MongoDBStoreBase.save_or_update` in [`database/mongodb_store_base.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/database/mongodb_store_base.py)

## Practical Code Examples

### Storing Zhihu Content in MongoDB

```python
from model.m_zhihu import ZhihuContent
from store.zhihu._store_impl import ZhihuMongoStoreImplement

# Build a content object (validation occurs automatically)

content = ZhihuContent(
    content_id="12345",
    content_type="answer",
    content_text="Python is great!",
    title="Why Python?",
    voteup_count=42,
    user_id="u6789",
    user_nickname="alice",
)

# Persist it (the store expects a plain dict)

store = ZhihuMongoStoreImplement()
await store.store_content(content.dict())

```

The `store_content` method internally calls `MongoDBStoreBase.save_or_update`, writing the document into the `zhihu_contents` collection.

### Exporting Tieba Comments to CSV

```python
from model.m_baidu_tieba import TiebaComment
from store.tieba._store_impl import TieBaCsvStoreImplement

comment = TiebaComment(
    comment_id="cmt987",
    content="Nice post!",
    publish_time="2024-09-01 12:00",
    user_nickname="bob",
    note_id="note123",
    tieba_name="Python吧"
)

csv_store = TieBaCsvStoreImplement()
await csv_store.store_comment(comment.dict())

```

The CSV implementation uses `AsyncFileWriter.write_to_csv` to append structured data to the appropriate platform-specific file.

### Handling Creator Metadata

```python
from model.m_xiaohongshu import CreatorUrlInfo

creator = CreatorUrlInfo(user_id="u001", xsec_token="abc", xsec_source="web")
print(creator.json())

```

The `CreatorUrlInfo` model stores URL ingredients required to request additional profile data from the platform API.

## Summary

- MediaCrawler extracts three normalized entity types: **content**, **comments**, and **creator profiles**
- Platform-specific Pydantic models in [`model/m_zhihu.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/model/m_zhihu.py), [`model/m_baidu_tieba.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/model/m_baidu_tieba.py), [`model/m_xiaohongshu.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/model/m_xiaohongshu.py), and [`model/m_douyin.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/model/m_douyin.py) validate incoming data
- The **`AbstractCrawler`** base class in [`base/base_crawler.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/base/base_crawler.py) provides the generic extraction framework
- Storage implementations support CSV, JSON, Excel, SQLite, MySQL, and MongoDB via the **`AbstractStore`** interface
- All extracted data maintains consistent field naming across platforms, enabling homogeneous downstream analysis

## Frequently Asked Questions

### Does MediaCrawler extract video-specific metadata?

Yes, video platforms like Douyin use the **`VideoUrlInfo`** model in [`model/m_douyin.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/model/m_douyin.py) to capture playback URLs, video statistics, and engagement metrics, treating them as content entities while preserving video-specific attributes.

### Can MediaCrawler handle nested comment threads?

Yes, comment models include **`sub_comment_count`** fields to track reply depth, and the extraction logic supports recursive parsing of nested comment chains as implemented in `ZhihuComment` and `TiebaComment`.

### What storage formats does MediaCrawler support for extracted data?

The tool supports CSV, JSON/JSONL, Excel, SQLite, MySQL, and MongoDB through the `AbstractStore` interface, with concrete implementations located in `store/` directories (e.g., [`store/zhihu/_store_impl.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/zhihu/_store_impl.py)).

### How does MediaCrawler authenticate to extract private creator data?

Crawler classes utilize Playwright and Chrome DevTools Protocol (CDP) for browser automation, managing cookies, session tokens, and anti-detection measures required to access user profile data and comment sections that require authentication.