How to Generate Word Clouds from MediaCrawler Comment Data

Set ENABLE_GET_COMMENTS and ENABLE_GET_WORDCLOUD to True in your configuration, then call await AsyncFileWriter(platform, crawler_type).generate_wordcloud_from_comments() to produce PNG visualizations and JSON frequency files from stored comment data.

MediaCrawler is a high-performance asynchronous crawler for extracting social media content. When properly configured, it transforms raw comment JSON data into visual word clouds using Chinese text segmentation and frequency analysis. This guide explains the complete pipeline from configuration to image generation based on the actual implementation in the NanmiCoder/MediaCrawler repository.

Configuration and Prerequisites

Before generating word clouds, you must enable the feature flags in your configuration file. Open config/base_config.py and verify the following settings:

ENABLE_GET_COMMENTS = True      # Required to store comment data

ENABLE_GET_WORDCLOUD = True     # Required to enable word cloud generation

These boolean flags control whether the AsyncFileWriter class initializes the word cloud subsystem during instantiation. When both are enabled, the writer prepares the necessary paths and loads stop-word configurations defined by STOP_WORDS_FILE and FONT_PATH in your config directory.

The Word Cloud Generation Pipeline

Step 1: Data Collection and Storage

During a crawl operation, MediaCrawler writes each comment to a file following the pattern:

<save_root>/<platform>/<file_type>/<crawler_type>_comments_<date>.<ext>

The AsyncFileWriter._get_file_path method determines the exact location based on the platform name and crawler type. Comments are stored as either single JSON objects or JSONL (one JSON object per line) format depending on your configuration.

Step 2: Triggering the Generation Process

After data collection completes (or at any time with existing files), invoke the word cloud generator:

import asyncio
from tools.async_file_writer import AsyncFileWriter

async def generate_cloud():
    writer = AsyncFileWriter(platform="tieba", crawler_type="my_crawl")
    await writer.generate_wordcloud_from_comments()

asyncio.run(generate_cloud())

The generate_wordcloud_from_comments method reads the stored comment file, extracts text from the content field (falling back to comment_text or text if necessary), and passes the cleaned data to the word cloud engine.

Step 3: Text Processing and Frequency Analysis

Inside tools/words.py, the AsyncWordCloudGenerator class handles the heavy lifting. The generate_word_frequency_and_cloud method performs four critical operations:

1. Tokenization: Uses jieba to segment Chinese text into individual words
2. Filtering: Removes stop words defined in config.STOP_WORDS_FILE and applies custom word lists from config.CUSTOM_WORDS
3. Frequency counting: Builds a Counter object mapping words to occurrence counts
4. Serialization: Saves the frequency dictionary to a JSON file with the suffix _word_freq.json

The method then calls generate_word_cloud to render the top 20 most frequent terms as a PNG image using matplotlib and the font specified by config.FONT_PATH.

Step 4: Concurrent Safety

Both the file writer and plotting routines use asyncio.Lock mechanisms to prevent race conditions. The AsyncFileWriter maintains self.lock for file operations, while AsyncWordCloudGenerator uses a module-level plot_lock to ensure matplotlib canvas operations do not conflict during simultaneous crawls.

Code Examples

Basic Usage with AsyncFileWriter

import asyncio
from tools.async_file_writer import AsyncFileWriter

async def run_wordcloud():
    # Initialize writer for specific platform and job

    writer = AsyncFileWriter(platform="tieba", crawler_type="batch_001")
    
    # Generates:

    # - <prefix>_word_freq.json (frequency data)

    # - <prefix>_word_cloud.png (visualization)

    await writer.generate_wordcloud_from_comments()

asyncio.run(run_wordcloud())

Direct Generator Access

For testing or processing existing comment data without the file writer:

import asyncio
from tools.words import AsyncWordCloudGenerator

async def process_existing_comments():
    generator = AsyncWordCloudGenerator()
    
    # comments: list of dicts with "content" key

    comments = [
        {"content": "This product is amazing quality"},
        {"content": "Fast shipping and good service"}
    ]
    
    await generator.generate_word_frequency_and_cloud(
        comments,
        save_words_prefix="output/analysis"
    )

asyncio.run(process_existing_comments())

Key Files and Architecture

Understanding these source files helps with customization and debugging:

• tools/async_file_writer.py: Coordinates persistence and launches word cloud generation. Contains AsyncFileWriter.__init__ for configuration validation and generate_wordcloud_from_comments for orchestration.

• tools/words.py: Implements AsyncWordCloudGenerator with generate_word_frequency_and_cloud for tokenization/counting and generate_word_cloud for matplotlib rendering.

• config/base_config.py: Central configuration defining ENABLE_GET_COMMENTS, ENABLE_GET_WORDCLOUD, STOP_WORDS_FILE, and FONT_PATH.

Summary

• Enable both flags: ENABLE_GET_COMMENTS and ENABLE_GET_WORDCLOUD must be set to True in config/base_config.py
• Use AsyncFileWriter: Call await AsyncFileWriter(platform, crawler_type).generate_wordcloud_from_comments() to process stored data
• Dependencies required: Install jieba, wordcloud, matplotlib, and aiofiles for the pipeline to function
• Output files: The system produces a JSON frequency file (*_word_freq.json) and a PNG visualization (*_word_cloud.png)
• Thread-safe: Built-in asyncio.Lock protection prevents file corruption during concurrent operations

Frequently Asked Questions

What file formats does MediaCrawler use for comment storage?

MediaCrawler stores comments as either JSON (single object) or JSONL (newline-delimited JSON objects) depending on your configuration. The AsyncFileWriter automatically detects the format when reading files for word cloud generation, extracting the content field by default with fallbacks to comment_text or text.

How do I customize stop words or add domain-specific terms?

Define your stop words in the file specified by STOP_WORDS_FILE in your configuration (typically a text file with one word per line). Add domain-specific terms to CUSTOM_WORDS in config/base_config.py to ensure specific phrases are preserved during jieba tokenization. Changes take effect on the next crawl or manual generation call.

Can I generate word clouds from existing comment files without re-crawling?

Yes. Instantiate AsyncFileWriter with the original platform and crawler_type parameters used during the initial crawl, then call await writer.generate_wordcloud_from_comments(). The method reads the existing file path determined by _get_file_path and processes the historical data without requiring new network requests.

Why is my word cloud generation returning empty output?

Empty output typically occurs when: (1) ENABLE_GET_WORDCLOUD is disabled in configuration, (2) the comment file does not exist at the expected path, (3) all words are filtered by the stop words list, or (4) the content field is missing from comment objects. Verify that jieba is installed and that FONT_PATH points to a valid system font for Chinese character rendering.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}