# How to Crawl Second-Level (Nested) Comments with MediaCrawler

> Learn how to crawl second-level comments with MediaCrawler. Extract parent comments and their IDs to fetch nested replies efficiently. Improve your data. Get started now.

- Repository: [程序员阿江-Relakkes/MediaCrawler](https://github.com/NanmiCoder/MediaCrawler)
- Tags: how-to-guide
- Published: 2026-06-27

---

**To crawl second-level comments in MediaCrawler, first extract parent comments using `TieBaExtractor.extract_tieba_note_parment_comments()` which provides the `sub_comment_count` and `comment_id` needed to identify and fetch nested replies.**

MediaCrawler provides a robust extraction framework for Baidu Tieba that handles nested comment structures. While the core extractor currently targets first-level (parent) comments, each returned object contains the metadata necessary to identify and retrieve second-level replies. This architecture ensures you can traverse the full comment hierarchy using the parent comment IDs and reply counts provided in the extraction results.

## Understanding the Comment Hierarchy

Baidu Tieba organizes discussions in a two-level structure:

- **First-level comments**: Direct replies to the original post (parent comments)
- **Second-level comments**: Nested replies to specific parent comments (sub-comments)

In [`media_platform/tieba/help.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/tieba/help.py), the `TieBaExtractor` class treats these as distinct extraction targets. The method `extract_tieba_note_parment_comments` (note: "parment" appears to be shorthand for "parent") specifically handles the parent comment layer, returning a `TiebaComment` Pydantic model defined in [`model/m_baidu_tieba.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/model/m_baidu_tieba.py).

Each parent comment object includes a `sub_comment_count` field that indicates how many second-level replies exist for that specific comment. To access the actual content of nested replies, you use the parent comment's `comment_id` as the reference point for subsequent sub-comment extraction calls.

## Extracting Parent Comments (First-Level)

Before you can access second-level comments, you must extract the parent layer. The `extract_tieba_note_parment_comments` method parses Tieba comment pages using XPath selectors to isolate individual comment blocks.

### The `extract_tieba_note_parment_comments` Method

Located in [`media_platform/tieba/help.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/tieba/help.py) (lines 776-836), this method processes HTML content containing the comment list and returns a list of `TiebaComment` objects:

```python
def extract_tieba_note_parment_comments(self, page_content: str, note_id: str) -> List[TiebaComment]:
    """
    Extract Tieba post first-level comments from comment page
    """
    # 1️⃣ Locate each comment block

    xpath_selector = f"//div[{self._class_contains('l_post')} and {self._class_contains('j_l_post')}]"
    comment_list = Selector(text=page_content).xpath(xpath_selector)

    # 2️⃣ Gather forum metadata (name, link) for every comment

    content_selector = Selector(text=page_content)
    tieba_name, tieba_link = self._extract_forum_info(content_selector, page_content)

    result: List[TiebaComment] = []
    for comment_selector in comment_list:
        # 3️⃣ Pull the JSON‑encoded data‑field (author, content, etc.)

        comment_field_value: Dict = self.extract_data_field_value(comment_selector)
        comment_content_value = comment_field_value.get("content", {}) if comment_field_value else {}
        if not comment_content_value:
            continue

        # 4️⃣ Parse IP address & publish time from the comment footer

        other_info_content = comment_selector.xpath(
            f".//div[{self._class_contains('post-tail-wrap')}]"
        ).get(default="").strip()
        ip_location, publish_time = self.extract_ip_and_pub_time(other_info_content)

        # 5️⃣ Resolve user link & avatar (fallback to portrait URL if missing)

        user_selector = comment_selector.xpath(f".//a[{self._class_contains('p_author_name')}][1]")
        user_avatar = comment_selector.xpath(
            f".//a[{self._class_contains('p_author_face')}]//img/@src"
        ).get(default="").strip()
        if not user_avatar and comment_field_value.get("author", {}).get("portrait"):
            portrait = comment_field_value["author"]["portrait"]
            user_avatar = f"https://gss0.bdstatic.com/6LZ1dD3d1sgCo2Kml5_Y_D3/sys/portrait/item/{portrait}"

        # 6️⃣ Extract the comment’s text (HTML → plain text)

        content_html = comment_content_value.get("content") or comment_selector.xpath(
            f".//div[{self._class_contains('d_post_content')}]"
        ).get(default="")
        comment_text = utils.extract_text_from_html(content_html)

        # 7️⃣ Consolidate everything into a TiebaComment model

        tieba_comment = TiebaComment(
            comment_id=str(
                comment_content_value.get("post_id")
                or comment_selector.xpath("./@data-pid").get(default="")
            ),
            sub_comment_count=comment_content_value.get("comment_num") or 0,
            content=comment_text,
            note_url=const.TIEBA_URL + f"/p/{note_id}",
            user_link=self._absolute_url(user_selector.xpath("./@href").get(default="")),
            user_nickname=self._selector_text(comment_selector, f".//a[{self._class_contains('p_author_name')}][1]")
                or comment_field_value.get("author", {}).get("user_nickname")
                or comment_field_value.get("author", {}).get("user_name", ""),
            user_avatar=user_avatar,
            tieba_id=str(comment_content_value.get("forum_id", "")),
            tieba_name=tieba_name,
            tieba_link=tieba_link,
            ip_location=ip_location,
            publish_time=publish_time,
            note_id=note_id,
        )
        result.append(tieba_comment)
    return result

```

**Key observation for second-level crawling**: The `sub_comment_count` field (populated from `comment_content_value.get("comment_num")`) tells you exactly how many nested replies exist for each parent comment. When this value is greater than zero, you know second-level comments are available for that specific thread.

## Accessing Second-Level Comment Data

To crawl second-level comments, you must leverage the parent comment extraction as a prerequisite:

1. **Extract parent comments** using the method shown above
2. **Filter for comments with `sub_comment_count > 0`**
3. **Use the `comment_id`** of each parent comment to fetch its sub-comment thread via Tieba's sub-comment API or additional extraction methods

The `TiebaComment` model in [`model/m_baidu_tieba.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/model/m_baidu_tieba.py) provides the schema for these objects, ensuring type safety when handling the hierarchy.

## Practical Implementation Examples

### Fetching Parent Comments to Identify Second-Level Targets

```python
import requests
from media_platform.tieba.help import TieBaExtractor

def fetch_comments_with_nested_replies(post_id: str):
    """Extract parent comments and identify which have second-level replies"""
    url = f"https://tieba.baidu.com/p/{post_id}?showComment=1"
    resp = requests.get(url, timeout=10)
    resp.raise_for_status()
    
    extractor = TieBaExtractor()
    comments = extractor.extract_tieba_note_parment_comments(resp.text, note_id=post_id)
    
    # Filter for comments with second-level replies

    comments_with_replies = [c for c in comments if c.sub_comment_count > 0]
    
    print(f"Found {len(comments_with_replies)} parent comments with nested replies")
    for c in comments_with_replies:
        print(f"Comment {c.comment_id} has {c.sub_comment_count} second-level replies")
        # Use c.comment_id to fetch sub-comments in subsequent calls

# Execute

fetch_comments_with_nested_replies("10559655942")

```

### Async Pattern for Large-Scale Extraction

```python
import asyncio
import httpx
from media_platform.tieba.help import TieBaExtractor

async def extract_nested_comment_metadata(post_id: str):
    """Async extraction of parent comments to prepare for second-level crawling"""
    url = f"https://tieba.baidu.com/p/{post_id}?showComment=1"
    async with httpx.AsyncClient() as client:
        r = await client.get(url, timeout=15.0)
        r.raise_for_status()
    
    extractor = TieBaExtractor()
    comments = extractor.extract_tieba_note_parment_comments(r.text, note_id=post_id)
    
    # Return only comments that have nested replies

    return [c for c in comments if c.sub_comment_count > 0]

# Use the comment IDs to fetch second-level content

async def main():
    parents_with_replies = await extract_nested_comment_metadata("10559655942")
    for parent in parents_with_replies:
        print(f"Parent {parent.comment_id}: {parent.sub_comment_count} replies")
        # Subsequent extraction would use parent.comment_id here

asyncio.run(main())

```

## Key Files for Second-Level Comment Crawling

| File | Purpose | Relevant Components |
|------|---------|-------------------|
| [`media_platform/tieba/help.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/tieba/help.py) | Contains `TieBaExtractor` and `extract_tieba_note_parment_comments` for parent comment extraction | `extract_tieba_note_parment_comments`, `extract_data_field_value` |
| [`model/m_baidu_tieba.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/model/m_baidu_tieba.py) | Pydantic models including `TiebaComment` with `sub_comment_count` field | `TiebaComment.comment_id`, `TiebaComment.sub_comment_count` |
| [`media_platform/tieba/client.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/tieba/client.py) | High-level client that may include sub-comment fetching methods | `TiebaClient` class methods |
| [`tools/utils.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/utils.py) | Text extraction utilities for cleaning HTML content | `extract_text_from_html` |

## Summary

- **Second-level comments** in MediaCrawler are accessed by first extracting parent comments via `extract_tieba_note_parment_comments` in [`media_platform/tieba/help.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/tieba/help.py).
- The **`sub_comment_count`** field in each `TiebaComment` object indicates whether nested replies exist for a specific parent comment.
- The **`comment_id`** serves as the reference key for fetching sub-comment content in subsequent extraction steps.
- The extraction uses **XPath selectors** targeting `<div class="l_post j_l_post">` elements and parses JSON data embedded in `data-field` attributes.
- The `TiebaComment` Pydantic model ensures consistent typing for comment metadata including reply counts and user information.

## Frequently Asked Questions

### How does MediaCrawler differentiate between first-level and second-level comments?

MediaCrawler treats first-level comments as the primary extraction target through the `extract_tieba_note_parment_comments` method. Each extracted parent comment includes a `sub_comment_count` integer field that specifies how many nested replies exist. To get the actual second-level comment content, you use the parent comment's `comment_id` to query Tieba's sub-comment endpoints or additional extraction methods, following the same architectural pattern but targeting different HTML selectors or API parameters.

### What does the `sub_comment_count` field represent in the TiebaComment model?

The `sub_comment_count` field represents the number of second-level (nested) replies that exist for a specific parent comment. This value is extracted from the `comment_num` property in the JSON data-field embedded in Tieba's HTML. When this value is greater than zero, it indicates that the comment has nested replies that can be fetched separately using the parent comment's ID.

### Can I use the TiebaClient to automatically fetch both levels of comments?

The `TiebaClient` class in [`client.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/client.py) provides high-level orchestration for HTTP requests and pagination. While the provided analysis focuses on the low-level extraction method, the client likely implements methods to handle sub-comment fetching by leveraging the parent comment IDs returned from the initial extraction. You would typically call the client methods with parameters like `crawl_sub_comments=True` or similar, though the specific implementation depends on the version of the MediaCrawler repository.

### Why is the method named `extract_tieba_note_parment_comments`?

The method name appears to use "parment" as shorthand for "parent" comments, distinguishing these first-level replies from sub-comments or second-level replies. This naming convention in the MediaCrawler codebase helps clarify that the extractor targets the parent comment layer—the top level of the reply hierarchy that contains metadata about its nested children via the `sub_comment_count` field.