# How to Crawl First-Level Comments with MediaCrawler: A Complete Guide

> Learn how to crawl first-level comments using MediaCrawler. This guide explains how to use the TieBaExtractor method for efficient data extraction of user details, content, and metadata.

- Repository: [程序员阿江-Relakkes/MediaCrawler](https://github.com/NanmiCoder/MediaCrawler)
- Tags: how-to-guide
- Published: 2026-06-27

---

**MediaCrawler extracts Baidu Tieba first-level comments by parsing HTML comment pages with the `TieBaExtractor.extract_tieba_note_parment_comments` method, which returns a list of `TiebaComment` Pydantic models containing user details, content, and metadata.**

The NanmiCoder/MediaCrawler repository provides a robust framework for scraping social media platforms, including dedicated extractors for Baidu Tieba discussions. When you need to crawl first-level comments with MediaCrawler, the library offers specialized XPath-based parsing logic that handles the complex DOM structure of Tieba comment pages. This guide explains the exact implementation found in the source code and provides ready-to-use examples for both synchronous and asynchronous workflows.

## Understanding the Tieba Extraction Architecture

### The Core Extractor Class

In [`media_platform/tieba/help.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/tieba/help.py), the `TieBaExtractor` class serves as the central parsing engine for Tieba content. The method `extract_tieba_note_parment_comments` specifically targets first-level comments by selecting DOM elements matching the XPath expression `//div[contains(@class,'l_post') and contains(@class,'j_l_post')]`. This selector identifies the comment containers that hold user data, content, and interaction metrics.

The extractor processes each comment through a seven-step pipeline:
1. **DOM Selection**: Identifies comment blocks using class-based XPath selectors
2. **Metadata Extraction**: Pulls JSON-encoded data from `data-field` attributes
3. **Text Cleaning**: Strips HTML tags from comment bodies using `utils.extract_text_from_html`
4. **Footer Parsing**: Extracts IP addresses and timestamps from `post-tail-wrap` elements
5. **User Resolution**: Resolves avatar URLs and user links from both DOM and JSON fields
6. **Forum Context**: Captures the Tieba name and link for each comment
7. **Model Instantiation**: Wraps everything in the `TiebaComment` Pydantic model

### Data Models and Schema

The extractor returns structured data using the `TiebaComment` Pydantic model defined in [`model/m_baidu_tieba.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/model/m_baidu_tieba.py). This model enforces type safety for fields including `comment_id`, `user_nickname`, `content`, `ip_location`, `publish_time`, and `sub_comment_count`. The strict schema ensures consistent data handling across different Tieba posts and forums.

## Implementing First-Level Comment Extraction

### Synchronous Crawling with Requests

For simple scripts, combine the `requests` library with the extractor to fetch and parse comment pages. The Tieba comment list view requires the `showComment=1` query parameter appended to the post URL.

```python
import requests
from media_platform.tieba.help import TieBaExtractor

def comment_page_url(post_id: str) -> str:
    return f"https://tieba.baidu.com/p/{post_id}?showComment=1"

post_id = "10559655942"
resp = requests.get(comment_page_url(post_id), timeout=10)
resp.raise_for_status()
html = resp.text

extractor = TieBaExtractor()
comments = extractor.extract_tieba_note_parment_comments(html, note_id=post_id)

for c in comments:
    print(f"[{c.comment_id}] {c.user_nickname}: {c.content[:80]}...")

```

### Asynchronous Extraction with HTTPX

The MediaCrawler codebase uses `httpx` for non-blocking I/O operations. This approach is essential when processing multiple posts concurrently or integrating with existing async pipelines.

```python
import asyncio
import httpx
from media_platform.tieba.help import TieBaExtractor

async def fetch_comments(post_id: str):
    url = f"https://tieba.baidu.com/p/{post_id}?showComment=1"
    async with httpx.AsyncClient() as client:
        r = await client.get(url, timeout=15.0)
        r.raise_for_status()
        html = r.text

    extractor = TieBaExtractor()
    comments = extractor.extract_tieba_note_parment_comments(html, note_id=post_id)

    for c in comments:
        print(f"{c.comment_id} → {c.user_nickname} – {c.content}")

# Run

asyncio.run(fetch_comments("10559655942"))

```

### Using the High-Level TiebaClient

For production use cases requiring pagination and automatic HTTP handling, [`media_platform/tieba/client.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/tieba/client.py) provides the `TiebaClient` class. This wrapper manages session state, retries, and comment pagination while internally calling the same extractor methods.

```python
from media_platform.tieba.client import TiebaClient

async def main():
    async with TiebaClient() as client:
        comments = await client.get_note_all_comments(
            note_id="10559655942",
            crawl_interval=0,
            max_count=1000
        )
        for c in comments:
            print(c.comment_id, c.user_nickname, c.content[:50])

# asyncio.run(main())

```

## Parsing Technical Details

The extraction process involves several precise operations to handle Tieba's complex markup:

**JSON Field Extraction**: The method extracts embedded metadata from `data-field` attributes using `extract_data_field_value`, which contains author information, content IDs, and portrait data. This structured JSON often contains more reliable user identifiers than the visible HTML.

**HTML Cleaning**: Raw comment content passes through `utils.extract_text_from_html` to strip HTML tags and normalize whitespace, ensuring clean text in the final `content` field.

**Metadata Parsing**: IP addresses and timestamps are parsed from the `post-tail-wrap` div elements using `extract_ip_and_pub_time`, which handles the footer text format variations across different Tieba clients.

**Avatar Resolution**: User avatars are resolved through a fallback chain: first checking the DOM img tag, then constructing a URL from the `portrait` field in the author JSON, using the pattern `https://gss0.bdstatic.com/6LZ1dD3d1sgCo2Kml5_Y_D3/sys/portrait/item/{portrait}`.

## Summary

- MediaCrawler provides dedicated Tieba extraction through `TieBaExtractor` in [`media_platform/tieba/help.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/tieba/help.py)
- The `extract_tieba_note_parment_comments` method parses first-level comments using XPath selectors on HTML from `?showComment=1` URLs
- Results are returned as typed `TiebaComment` Pydantic models with complete metadata including IP location and sub-comment counts
- Three implementation options exist: direct extractor usage with `requests`, async extraction with `httpx`, or the high-level `TiebaClient` with built-in pagination
- Utility functions in [`tools/utils.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/utils.py) handle HTML cleaning and metadata extraction automatically

## Frequently Asked Questions

### What is the difference between first-level and second-level comments in Tieba?

First-level comments (parent comments) appear directly under the original post, while second-level comments are replies to those parent comments. The `extract_tieba_note_parment_comments` method specifically targets the parent level by selecting `l_post j_l_post` divs, and the resulting `TiebaComment` objects include a `sub_comment_count` field indicating how many replies exist for each first-level comment.

### How does MediaCrawler handle anti-scraping measures on Tieba?

The framework uses standard HTTP headers and cookie-based sessions managed in [`client.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/client.py). For high-volume crawling, implement request delays using the `crawl_interval` parameter in `TiebaClient` methods, and rotate user agents or use proxy pools external to the core extraction logic. The extractor itself is stateless and focuses purely on parsing, leaving request strategy to the caller.

### Can I extract comments without using the Pydantic models?

While the public API returns `TiebaComment` objects, you could theoretically access the raw dictionary data from `extract_data_field_value` before model instantiation. However, this bypasses validation and is not recommended as the internal implementation may change; the Pydantic models provide the stable contract for the data schema.

### Where are the comment IP addresses and timestamps extracted from?

The IP location and publish time are extracted from the `post-tail-wrap` div elements within each comment container. The `extract_ip_and_pub_time` utility function parses these footer strings to populate the `ip_location` and `publish_time` fields of the `TiebaComment` model.