How to Crawl First-Level Comments with MediaCrawler: A Complete Guide

MediaCrawler extracts Baidu Tieba first-level comments by parsing HTML comment pages with the TieBaExtractor.extract_tieba_note_parment_comments method, which returns a list of TiebaComment Pydantic models containing user details, content, and metadata.

The NanmiCoder/MediaCrawler repository provides a robust framework for scraping social media platforms, including dedicated extractors for Baidu Tieba discussions. When you need to crawl first-level comments with MediaCrawler, the library offers specialized XPath-based parsing logic that handles the complex DOM structure of Tieba comment pages. This guide explains the exact implementation found in the source code and provides ready-to-use examples for both synchronous and asynchronous workflows.

Understanding the Tieba Extraction Architecture

The Core Extractor Class

In media_platform/tieba/help.py, the TieBaExtractor class serves as the central parsing engine for Tieba content. The method extract_tieba_note_parment_comments specifically targets first-level comments by selecting DOM elements matching the XPath expression //div[contains(@class,'l_post') and contains(@class,'j_l_post')]. This selector identifies the comment containers that hold user data, content, and interaction metrics.

The extractor processes each comment through a seven-step pipeline:

1. DOM Selection: Identifies comment blocks using class-based XPath selectors
2. Metadata Extraction: Pulls JSON-encoded data from data-field attributes
3. Text Cleaning: Strips HTML tags from comment bodies using utils.extract_text_from_html
4. Footer Parsing: Extracts IP addresses and timestamps from post-tail-wrap elements
5. User Resolution: Resolves avatar URLs and user links from both DOM and JSON fields
6. Forum Context: Captures the Tieba name and link for each comment
7. Model Instantiation: Wraps everything in the TiebaComment Pydantic model

Data Models and Schema

The extractor returns structured data using the TiebaComment Pydantic model defined in model/m_baidu_tieba.py. This model enforces type safety for fields including comment_id, user_nickname, content, ip_location, publish_time, and sub_comment_count. The strict schema ensures consistent data handling across different Tieba posts and forums.

Implementing First-Level Comment Extraction

Synchronous Crawling with Requests

For simple scripts, combine the requests library with the extractor to fetch and parse comment pages. The Tieba comment list view requires the showComment=1 query parameter appended to the post URL.

import requests
from media_platform.tieba.help import TieBaExtractor

def comment_page_url(post_id: str) -> str:
    return f"https://tieba.baidu.com/p/{post_id}?showComment=1"

post_id = "10559655942"
resp = requests.get(comment_page_url(post_id), timeout=10)
resp.raise_for_status()
html = resp.text

extractor = TieBaExtractor()
comments = extractor.extract_tieba_note_parment_comments(html, note_id=post_id)

for c in comments:
    print(f"[{c.comment_id}] {c.user_nickname}: {c.content[:80]}...")

Asynchronous Extraction with HTTPX

The MediaCrawler codebase uses httpx for non-blocking I/O operations. This approach is essential when processing multiple posts concurrently or integrating with existing async pipelines.

import asyncio
import httpx
from media_platform.tieba.help import TieBaExtractor

async def fetch_comments(post_id: str):
    url = f"https://tieba.baidu.com/p/{post_id}?showComment=1"
    async with httpx.AsyncClient() as client:
        r = await client.get(url, timeout=15.0)
        r.raise_for_status()
        html = r.text

    extractor = TieBaExtractor()
    comments = extractor.extract_tieba_note_parment_comments(html, note_id=post_id)

    for c in comments:
        print(f"{c.comment_id} → {c.user_nickname} – {c.content}")

# Run

asyncio.run(fetch_comments("10559655942"))

Using the High-Level TiebaClient

For production use cases requiring pagination and automatic HTTP handling, media_platform/tieba/client.py provides the TiebaClient class. This wrapper manages session state, retries, and comment pagination while internally calling the same extractor methods.

from media_platform.tieba.client import TiebaClient

async def main():
    async with TiebaClient() as client:
        comments = await client.get_note_all_comments(
            note_id="10559655942",
            crawl_interval=0,
            max_count=1000
        )
        for c in comments:
            print(c.comment_id, c.user_nickname, c.content[:50])

# asyncio.run(main())

Parsing Technical Details

The extraction process involves several precise operations to handle Tieba's complex markup:

JSON Field Extraction: The method extracts embedded metadata from data-field attributes using extract_data_field_value, which contains author information, content IDs, and portrait data. This structured JSON often contains more reliable user identifiers than the visible HTML.

HTML Cleaning: Raw comment content passes through utils.extract_text_from_html to strip HTML tags and normalize whitespace, ensuring clean text in the final content field.

Metadata Parsing: IP addresses and timestamps are parsed from the post-tail-wrap div elements using extract_ip_and_pub_time, which handles the footer text format variations across different Tieba clients.

Avatar Resolution: User avatars are resolved through a fallback chain: first checking the DOM img tag, then constructing a URL from the portrait field in the author JSON, using the pattern https://gss0.bdstatic.com/6LZ1dD3d1sgCo2Kml5_Y_D3/sys/portrait/item/{portrait}.

Summary

• MediaCrawler provides dedicated Tieba extraction through TieBaExtractor in media_platform/tieba/help.py
• The extract_tieba_note_parment_comments method parses first-level comments using XPath selectors on HTML from ?showComment=1 URLs
• Results are returned as typed TiebaComment Pydantic models with complete metadata including IP location and sub-comment counts
• Three implementation options exist: direct extractor usage with requests, async extraction with httpx, or the high-level TiebaClient with built-in pagination
• Utility functions in tools/utils.py handle HTML cleaning and metadata extraction automatically

Frequently Asked Questions

What is the difference between first-level and second-level comments in Tieba?

First-level comments (parent comments) appear directly under the original post, while second-level comments are replies to those parent comments. The extract_tieba_note_parment_comments method specifically targets the parent level by selecting l_post j_l_post divs, and the resulting TiebaComment objects include a sub_comment_count field indicating how many replies exist for each first-level comment.

How does MediaCrawler handle anti-scraping measures on Tieba?

The framework uses standard HTTP headers and cookie-based sessions managed in client.py. For high-volume crawling, implement request delays using the crawl_interval parameter in TiebaClient methods, and rotate user agents or use proxy pools external to the core extraction logic. The extractor itself is stateless and focuses purely on parsing, leaving request strategy to the caller.

Can I extract comments without using the Pydantic models?

While the public API returns TiebaComment objects, you could theoretically access the raw dictionary data from extract_data_field_value before model instantiation. However, this bypasses validation and is not recommended as the internal implementation may change; the Pydantic models provide the stable contract for the data schema.

Where are the comment IP addresses and timestamps extracted from?

The IP location and publish time are extracted from the post-tail-wrap div elements within each comment container. The extract_ip_and_pub_time utility function parses these footer strings to populate the ip_location and publish_time fields of the TiebaComment model.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}