How to Crawl Second-Level (Nested) Comments with MediaCrawler

To crawl second-level comments in MediaCrawler, first extract parent comments using TieBaExtractor.extract_tieba_note_parment_comments() which provides the sub_comment_count and comment_id needed to identify and fetch nested replies.

MediaCrawler provides a robust extraction framework for Baidu Tieba that handles nested comment structures. While the core extractor currently targets first-level (parent) comments, each returned object contains the metadata necessary to identify and retrieve second-level replies. This architecture ensures you can traverse the full comment hierarchy using the parent comment IDs and reply counts provided in the extraction results.

Understanding the Comment Hierarchy

Baidu Tieba organizes discussions in a two-level structure:

• First-level comments: Direct replies to the original post (parent comments)
• Second-level comments: Nested replies to specific parent comments (sub-comments)

In media_platform/tieba/help.py, the TieBaExtractor class treats these as distinct extraction targets. The method extract_tieba_note_parment_comments (note: "parment" appears to be shorthand for "parent") specifically handles the parent comment layer, returning a TiebaComment Pydantic model defined in model/m_baidu_tieba.py.

Each parent comment object includes a sub_comment_count field that indicates how many second-level replies exist for that specific comment. To access the actual content of nested replies, you use the parent comment's comment_id as the reference point for subsequent sub-comment extraction calls.

Extracting Parent Comments (First-Level)

Before you can access second-level comments, you must extract the parent layer. The extract_tieba_note_parment_comments method parses Tieba comment pages using XPath selectors to isolate individual comment blocks.

The extract_tieba_note_parment_comments Method

Located in media_platform/tieba/help.py (lines 776-836), this method processes HTML content containing the comment list and returns a list of TiebaComment objects:

def extract_tieba_note_parment_comments(self, page_content: str, note_id: str) -> List[TiebaComment]:
    """
    Extract Tieba post first-level comments from comment page
    """
    # 1️⃣ Locate each comment block

    xpath_selector = f"//div[{self._class_contains('l_post')} and {self._class_contains('j_l_post')}]"
    comment_list = Selector(text=page_content).xpath(xpath_selector)

    # 2️⃣ Gather forum metadata (name, link) for every comment

    content_selector = Selector(text=page_content)
    tieba_name, tieba_link = self._extract_forum_info(content_selector, page_content)

    result: List[TiebaComment] = []
    for comment_selector in comment_list:
        # 3️⃣ Pull the JSON‑encoded data‑field (author, content, etc.)

        comment_field_value: Dict = self.extract_data_field_value(comment_selector)
        comment_content_value = comment_field_value.get("content", {}) if comment_field_value else {}
        if not comment_content_value:
            continue

        # 4️⃣ Parse IP address & publish time from the comment footer

        other_info_content = comment_selector.xpath(
            f".//div[{self._class_contains('post-tail-wrap')}]"
        ).get(default="").strip()
        ip_location, publish_time = self.extract_ip_and_pub_time(other_info_content)

        # 5️⃣ Resolve user link & avatar (fallback to portrait URL if missing)

        user_selector = comment_selector.xpath(f".//a[{self._class_contains('p_author_name')}][1]")
        user_avatar = comment_selector.xpath(
            f".//a[{self._class_contains('p_author_face')}]//img/@src"
        ).get(default="").strip()
        if not user_avatar and comment_field_value.get("author", {}).get("portrait"):
            portrait = comment_field_value["author"]["portrait"]
            user_avatar = f"https://gss0.bdstatic.com/6LZ1dD3d1sgCo2Kml5_Y_D3/sys/portrait/item/{portrait}"

        # 6️⃣ Extract the comment’s text (HTML → plain text)

        content_html = comment_content_value.get("content") or comment_selector.xpath(
            f".//div[{self._class_contains('d_post_content')}]"
        ).get(default="")
        comment_text = utils.extract_text_from_html(content_html)

        # 7️⃣ Consolidate everything into a TiebaComment model

        tieba_comment = TiebaComment(
            comment_id=str(
                comment_content_value.get("post_id")
                or comment_selector.xpath("./@data-pid").get(default="")
            ),
            sub_comment_count=comment_content_value.get("comment_num") or 0,
            content=comment_text,
            note_url=const.TIEBA_URL + f"/p/{note_id}",
            user_link=self._absolute_url(user_selector.xpath("./@href").get(default="")),
            user_nickname=self._selector_text(comment_selector, f".//a[{self._class_contains('p_author_name')}][1]")
                or comment_field_value.get("author", {}).get("user_nickname")
                or comment_field_value.get("author", {}).get("user_name", ""),
            user_avatar=user_avatar,
            tieba_id=str(comment_content_value.get("forum_id", "")),
            tieba_name=tieba_name,
            tieba_link=tieba_link,
            ip_location=ip_location,
            publish_time=publish_time,
            note_id=note_id,
        )
        result.append(tieba_comment)
    return result

Key observation for second-level crawling: The sub_comment_count field (populated from comment_content_value.get("comment_num")) tells you exactly how many nested replies exist for each parent comment. When this value is greater than zero, you know second-level comments are available for that specific thread.

Accessing Second-Level Comment Data

To crawl second-level comments, you must leverage the parent comment extraction as a prerequisite:

1. Extract parent comments using the method shown above
2. Filter for comments with sub_comment_count > 0
3. Use the comment_id of each parent comment to fetch its sub-comment thread via Tieba's sub-comment API or additional extraction methods

The TiebaComment model in model/m_baidu_tieba.py provides the schema for these objects, ensuring type safety when handling the hierarchy.

Practical Implementation Examples

Fetching Parent Comments to Identify Second-Level Targets

import requests
from media_platform.tieba.help import TieBaExtractor

def fetch_comments_with_nested_replies(post_id: str):
    """Extract parent comments and identify which have second-level replies"""
    url = f"https://tieba.baidu.com/p/{post_id}?showComment=1"
    resp = requests.get(url, timeout=10)
    resp.raise_for_status()
    
    extractor = TieBaExtractor()
    comments = extractor.extract_tieba_note_parment_comments(resp.text, note_id=post_id)
    
    # Filter for comments with second-level replies

    comments_with_replies = [c for c in comments if c.sub_comment_count > 0]
    
    print(f"Found {len(comments_with_replies)} parent comments with nested replies")
    for c in comments_with_replies:
        print(f"Comment {c.comment_id} has {c.sub_comment_count} second-level replies")
        # Use c.comment_id to fetch sub-comments in subsequent calls

# Execute

fetch_comments_with_nested_replies("10559655942")

Async Pattern for Large-Scale Extraction

import asyncio
import httpx
from media_platform.tieba.help import TieBaExtractor

async def extract_nested_comment_metadata(post_id: str):
    """Async extraction of parent comments to prepare for second-level crawling"""
    url = f"https://tieba.baidu.com/p/{post_id}?showComment=1"
    async with httpx.AsyncClient() as client:
        r = await client.get(url, timeout=15.0)
        r.raise_for_status()
    
    extractor = TieBaExtractor()
    comments = extractor.extract_tieba_note_parment_comments(r.text, note_id=post_id)
    
    # Return only comments that have nested replies

    return [c for c in comments if c.sub_comment_count > 0]

# Use the comment IDs to fetch second-level content

async def main():
    parents_with_replies = await extract_nested_comment_metadata("10559655942")
    for parent in parents_with_replies:
        print(f"Parent {parent.comment_id}: {parent.sub_comment_count} replies")
        # Subsequent extraction would use parent.comment_id here

asyncio.run(main())

Key Files for Second-Level Comment Crawling

File	Purpose	Relevant Components
media_platform/tieba/help.py	Contains TieBaExtractor and extract_tieba_note_parment_comments for parent comment extraction	extract_tieba_note_parment_comments, extract_data_field_value
model/m_baidu_tieba.py	Pydantic models including TiebaComment with sub_comment_count field	TiebaComment.comment_id, TiebaComment.sub_comment_count
media_platform/tieba/client.py	High-level client that may include sub-comment fetching methods	TiebaClient class methods
tools/utils.py	Text extraction utilities for cleaning HTML content	extract_text_from_html

Summary

• Second-level comments in MediaCrawler are accessed by first extracting parent comments via extract_tieba_note_parment_comments in media_platform/tieba/help.py.
• The sub_comment_count field in each TiebaComment object indicates whether nested replies exist for a specific parent comment.
• The comment_id serves as the reference key for fetching sub-comment content in subsequent extraction steps.
• The extraction uses XPath selectors targeting <div class="l_post j_l_post"> elements and parses JSON data embedded in data-field attributes.
• The TiebaComment Pydantic model ensures consistent typing for comment metadata including reply counts and user information.

Frequently Asked Questions

How does MediaCrawler differentiate between first-level and second-level comments?

MediaCrawler treats first-level comments as the primary extraction target through the extract_tieba_note_parment_comments method. Each extracted parent comment includes a sub_comment_count integer field that specifies how many nested replies exist. To get the actual second-level comment content, you use the parent comment's comment_id to query Tieba's sub-comment endpoints or additional extraction methods, following the same architectural pattern but targeting different HTML selectors or API parameters.

What does the sub_comment_count field represent in the TiebaComment model?

The sub_comment_count field represents the number of second-level (nested) replies that exist for a specific parent comment. This value is extracted from the comment_num property in the JSON data-field embedded in Tieba's HTML. When this value is greater than zero, it indicates that the comment has nested replies that can be fetched separately using the parent comment's ID.

Can I use the TiebaClient to automatically fetch both levels of comments?

The TiebaClient class in client.py provides high-level orchestration for HTTP requests and pagination. While the provided analysis focuses on the low-level extraction method, the client likely implements methods to handle sub-comment fetching by leveraging the parent comment IDs returned from the initial extraction. You would typically call the client methods with parameters like crawl_sub_comments=True or similar, though the specific implementation depends on the version of the MediaCrawler repository.

Why is the method named extract_tieba_note_parment_comments?

The method name appears to use "parment" as shorthand for "parent" comments, distinguishing these first-level replies from sub-comments or second-level replies. This naming convention in the MediaCrawler codebase helps clarify that the extractor targets the parent comment layer—the top level of the reply hierarchy that contains metadata about its nested children via the sub_comment_count field.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}