How to Configure Custom Stop Words for Word Clouds in MediaCrawler

You can configure custom stop words for word clouds in MediaCrawler by setting the STOP_WORDS_FILE constant in config/base_config.py to point to a UTF-8 text file containing one stop word per line, which the WordCloudGenerator class in tools/words.py automatically loads and applies during jieba tokenization.

MediaCrawler is an open-source crawler framework that generates word cloud visualizations from collected text data using Chinese word segmentation. The system filters out common noise terms by referencing an external stop words file, allowing you to fully customize which tokens get excluded from the final visualization without modifying the core tokenization logic.

How Stop Word Filtering Works in MediaCrawler

The word cloud generation pipeline in MediaCrawler uses jieba for text tokenization, then filters tokens against a configurable stop word list. Understanding this three-stage workflow helps you customize the behavior effectively.

Configuration Layer (config/base_config.py)

The file path for stop words is defined as the constant STOP_WORDS_FILE. By default, this points to ./docs/hit_stopwords.txt, but you can redirect this to any custom file path.

Processing Layer (tools/words.py)

The WordCloudGenerator class handles the actual filtering. When instantiated, it calls self.load_stop_words() to read the file specified in config/base_config.py into a Python set. During tokenization, it checks word not in self.stop_words to determine whether to include each token in the word cloud.

Step-by-Step Configuration

Follow these steps to implement your own stop word list.

Create a Custom Stop Words File

Create a UTF-8 encoded text file with one stop word per line. Place this file anywhere in your repository.

# my_stopwords.txt

the
and
of
to
认为
是的

Update the Configuration Path

Edit config/base_config.py to point to your new file:

# config/base_config.py

STOP_WORDS_FILE = "./my_stopwords.txt"

If you prefer to keep the default hit_stopwords.txt location, simply edit that file directly to add or remove terms.

Environment Variable Override (Optional)

For deployments where you cannot modify the config file, use an environment variable. Update config/base_config.py to read from the environment with a fallback:

import os

STOP_WORDS_FILE = os.getenv(
    "MEDIA_CRAWLER_STOP_WORDS",
    "./docs/hit_stopwords.txt"
)

Then export the variable before running:

export MEDIA_CRAWLER_STOP_WORDS=./my_stopwords.txt
python -m main

Implementation Details

The WordCloudGenerator class in tools/words.py automatically applies your custom list when generating visualizations. After configuration changes, the next crawler run loads the new stop words without requiring code modifications to the generator itself.

from tools.words import WordCloudGenerator

wc = WordCloudGenerator()

# The generate_cloud method automatically filters against your custom stop words

cloud_svg = wc.generate_cloud(text_corpus)

Summary

• Stop word configuration in MediaCrawler is controlled by the STOP_WORDS_FILE constant in config/base_config.py.
• The default path points to ./docs/hit_stopwords.txt, but you can specify any UTF-8 text file.
• Format requirements are simple: one word per line, encoded in UTF-8.
• The WordCloudGenerator class in tools/words.py loads and applies these filters automatically during the jieba tokenization process.
• Environment variables provide a flexible alternative for production deployments.

Frequently Asked Questions

What file format does MediaCrawler expect for stop words?

MediaCrawler expects a plain text file encoded in UTF-8 with one stop word per line. The WordCloudGenerator class reads this file in tools/words.py and stores the terms in a Python set for O(1) lookup during tokenization.

Can I use multiple stop word files simultaneously?

The current implementation in config/base_config.py accepts a single file path. To use multiple files, concatenate them into one file or modify tools/words.py to loop through a list of paths in the load_stop_words() method.

Does MediaCrawler support English stop words?

Yes. While the default hit_stopwords.txt contains Chinese stop words, the filtering mechanism is language-agnostic. Simply create a file with English words like "the", "and", and "of", set STOP_WORDS_FILE to its path, and the jieba tokenizer will filter them alongside any Chinese terms.

Do I need to restart the crawler after changing stop words?

Yes. The WordCloudGenerator loads stop words during instantiation, so you must restart the crawler for changes to take effect. The file is not hot-reloaded during runtime.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}