# How to Parse the Data Extracted by MediaCrawler: A Complete Guide

> Master parsing MediaCrawler data. Learn how this Python tool transforms raw social media content into clean, typed objects with its efficient three-layer pipeline.

- Repository: [程序员阿江-Relakkes/MediaCrawler](https://github.com/NanmiCoder/MediaCrawler)
- Tags: how-to-guide
- Published: 2026-06-28

---

**MediaCrawler transforms raw HTML and JSON from Chinese social platforms into clean, typed Python objects through a three-layer pipeline of extraction utilities, platform-specific parsers, and Pydantic-style data models.**

To parse the data extracted by MediaCrawler effectively, you need to understand how the repository structures its extraction workflow. The codebase processes content from platforms like Zhihu, Tieba, and Douyin by first sanitizing raw responses with low-level utilities, then mapping fields to platform-specific extractors, and finally validating against strict data models before persistence. This architecture ensures type safety and consistent formatting across MongoDB, Excel, and other backends.

## Understanding the Three-Layer Parsing Architecture

MediaCrawler's parsing workflow is built around three distinct layers that transform raw network responses into database-ready records:

1. **Low-level extraction utilities** – Generic functions in [`tools/crawler_util.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/crawler_util.py) that strip HTML tags, decode URL parameters, and parse cookies.
2. **Platform-specific extractors** – Classes like `ZhihuHelp` and `TieBaExtractor` that understand the JSON structure of each platform's API and invoke the low-level utilities.
3. **Store/Model layer** – Typed data classes in `model/*.py` and persistence logic in `store/*/_store_impl.py` that handle the final validation and writing to configured backends.

## Step 1: Clean Raw Content with Low-Level Utilities

The foundation of MediaCrawler's parsing pipeline lives in **[`tools/crawler_util.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/crawler_util.py)**. This file contains the core `extract_text_from_html` function that removes script tags, style elements, and all remaining HTML markup to produce plain text.

```python
def extract_text_from_html(html: str) -> str:
    """Extract text from HTML, removing all tags."""
    if not html:
        return ""
    # Remove script and style elements

    clean_html = re.sub(r'<(script|style)[^>]*>.*?</\1>', '',
                        html, flags=re.DOTALL)
    # Remove all other tags

    clean_text = re.sub(r'<[^>]+>', '', clean_html).strip()
    return clean_text

```

Source: [`tools/crawler_util.py#L15-L24`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/crawler_util.py#L15-L24)

Additional helpers like `extract_url_params_to_dict` and `convert_str_cookie_to_dict` (lines 26-34) support the extractors when decoding query strings or authentication cookies.

## Step 2: Transform Platform-Specific Responses

Each supported platform implements its own extractor class that knows how to navigate the platform's API response structure and apply the low-level cleaning utilities.

### Zhihu Content Extraction

In **[`media_platform/zhihu/help.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/zhihu/help.py)**, the `ZhihuHelp` class processes JSON payloads from Zhihu's API. When extracting an answer, it sanitizes every text field using `extract_text_from_html`:

```python
res.content_text = extract_text_from_html(answer.get("content", ""))
res.title = extract_text_from_html(answer.get("title", ""))
res.desc = extract_text_from_html(answer.get("description", "")
                                   or answer.get("excerpt", ""))

```

Source: [`media_platform/zhihu/help.py#L111-L115`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/zhihu/help.py#L111-L115)

Comment extraction follows the same pattern:

```python
res.content = extract_text_from_html(comment.get("content"))

```

Source: [`media_platform/zhihu/help.py#L254-L256`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/zhihu/help.py#L254-L256)

### Tieba Search Results

The `TieBaExtractor` class (demonstrated in **[`tests/test_tieba_extractor.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tests/test_tieba_extractor.py)**) handles Baidu Tieba's HTML responses. It uses BeautifulSoup internally to parse the DOM, then relies on `extract_text_from_html` for final cleaning:

```python
notes = TieBaExtractor.extract_search_note_list(read_fixture("search_keyword_notes.html"))

```

Source: [`tests/test_tieba_extractor.py#L16-L17`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tests/test_tieba_extractor.py#L16-L17)

### Douyin Media Processing

For Douyin, the extraction logic resides in **[`store/douyin/_store_impl.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/douyin/_store_impl.py)**. Private helper functions prefixed with `_extract_` parse the raw `aweme_item` JSON to assemble media URLs and image lists:

```python
"video_download_url": _extract_video_download_url(aweme_item),
"pictures": ",".join(_extract_comment_image_list(comment_item)),

```

Source: [`store/douyin/_store_impl.py#L181-L184`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/douyin/_store_impl.py#L181-L184)

These helpers extract specific fields from the nested JSON and handle any necessary HTML cleaning before the data reaches the model layer.

## Step 3: Validate and Structure with Data Models

After extraction, data flows into strictly typed models defined in the `model/` directory. These Pydantic-style classes ensure that all fields are present and correctly typed before persistence:

- **Zhihu**: [`model/m_zhihu.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/model/m_zhihu.py) defines `ZhihuContent`, `ZhihuComment`, and `ZhihuCreator`
- **Tieba**: [`model/m_baidu_tieba.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/model/m_baidu_tieba.py) defines `TiebaNote` and `TiebaComment`
- **Douyin**: [`model/m_douyin.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/model/m_douyin.py) defines `DouyinMedia` and `DouyinComment`

For example, the `ZhihuContent` model contains fields like `content_text`, `title`, `desc`, and `author`, which the extractor populates directly after cleaning.

Source: [[`model/m_zhihu.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/model/m_zhihu.py)](https://github.com/NanmiCoder/MediaCrawler/blob/main/model/m_zhihu.py)

## Step 4: Persist Clean Data via the Store Layer

The final parsing step involves the store implementations in `store/*/_store_impl.py`. These classes take the validated model objects and write them to the configured backend (MongoDB, Excel, etc.).

The `StoreBase` class in [`store/excel_store_base.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/excel_store_base.py) defines the interface, while platform-specific stores implement the concrete logic. For example, the Zhihu store uses the model's `.dict()` method to insert clean data:

```python
await self.db.insert_one(content.dict())

```

Source: [[`store/zhihu/_store_impl.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/zhihu/_store_impl.py)](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/zhihu/_store_impl.py)

At this stage, no further HTML sanitization is required because the extractors have already produced clean text.

## Complete Code Examples

### Parse a Zhihu Answer

```python
from media_platform.zhihu.help import ZhihuHelp
from tools.crawler_util import extract_text_from_html

# Assume raw_answer_json is the JSON payload from Zhihu API

zhihu = ZhihuHelp()
content = zhihu._extract_answer_content(raw_answer_json)

print("Title:", content.title)          # Clean text, no HTML tags

print("Body:", content.content_text)    # Plain text extracted by extract_text_from_html

```

*Key calls*: `_extract_answer_content` → `extract_text_from_html` at [`media_platform/zhihu/help.py#L111-L115`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/zhihu/help.py#L111-L115)

### Parse a Tieba Note List from Saved HTML

```python
from tests.test_tieba_extractor import TieBaExtractor
from pathlib import Path

html = Path("fixtures/search_keyword_notes.html").read_text()
notes = TieBaExtractor.extract_search_note_list(html)

for note in notes:
    print(note.title)   # Already stripped of HTML tags

```

*Key call*: `extract_search_note_list` internally uses `extract_text_from_html` to clean titles.

### Store Parsed Douyin Media

```python
from store.douyin._store_impl import DouyinStore, _extract_video_download_url, _extract_comment_image_list
from model.m_douyin import DouyinMedia

# aweme_detail is the raw JSON from Douyin API

media_dict = {
    "video_download_url": _extract_video_download_url(aweme_detail),
    "pictures": ",".join(_extract_comment_image_list(aweme_detail)),
    # ... other fields

}
media = DouyinMedia(**media_dict)

store = DouyinStore()
store.save(media)   # Persists clean data to the configured DB

```

*Key helpers*: `_extract_video_download_url`, `_extract_comment_image_list` defined in [`store/douyin/_store_impl.py#L181-L184`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/douyin/_store_impl.py#L181-L184)

## Summary

- **Low-level utilities** in [`tools/crawler_util.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/crawler_util.py) provide the `extract_text_from_html` function that strips HTML tags from raw content.
- **Platform extractors** like `ZhihuHelp` and `TieBaExtractor` apply these utilities to platform-specific JSON/HTML responses at [`media_platform/zhihu/help.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/zhihu/help.py) and [`tests/test_tieba_extractor.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tests/test_tieba_extractor.py).
- **Data models** in `model/*.py` validate the cleaned data into typed Python objects before persistence.
- **Store implementations** in `store/*/_store_impl.py` receive validated models and write them to MongoDB, Excel, or other backends without additional parsing.

## Frequently Asked Questions

### How does MediaCrawler handle HTML sanitization for extracted content?

MediaCrawler uses the `extract_text_from_html` function in [`tools/crawler_util.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/crawler_util.py) to remove script tags, style elements, and all remaining HTML markup using regular expressions. This utility is called by platform-specific extractors like `ZhihuHelp` before populating the data models, ensuring that stored content contains only plain text.

### Can I customize the data models for different storage requirements?

Yes. The data models in `model/*.py` (such as [`m_zhihu.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/m_zhihu.py) and [`m_douyin.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/m_douyin.py)) define the structure for each platform's content. You can modify these Pydantic-style classes to add fields, change types, or set default values. The store implementations in `store/*/_store_impl.py` will automatically use the updated model structure when persisting data.

### Where should I add parsing logic for a new social platform?

To parse data for a new platform, create a new extractor class in `media_platform/{platform}/help.py` that uses the utilities from [`tools/crawler_util.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/crawler_util.py) to clean HTML/JSON. Then define corresponding models in `model/m_{platform}.py` and implement the store logic in `store/{platform}/_store_impl.py`. Follow the existing patterns in the Zhihu and Tieba implementations for consistency.

### What is the difference between the extractor and store layers?

The **extractor layer** (e.g., [`media_platform/zhihu/help.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/zhihu/help.py)) transforms raw API responses into cleaned Python dictionaries. The **store layer** (e.g., [`store/zhihu/_store_impl.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/zhihu/_store_impl.py)) takes these dictionaries, validates them against data models, and persists them to the configured backend. The extractor handles parsing and cleaning, while the store handles validation and database operations.