# How to Configure Custom Stop Words for Word Clouds in MediaCrawler

> Learn how to configure custom stop words for word clouds in MediaCrawler. Easily set the STOP_WORDS_FILE constant to customize your word cloud generation. Improve your text analysis.

- Repository: [程序员阿江-Relakkes/MediaCrawler](https://github.com/NanmiCoder/MediaCrawler)
- Tags: how-to-guide
- Published: 2026-06-27

---

**You can configure custom stop words for word clouds in MediaCrawler by setting the `STOP_WORDS_FILE` constant in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py) to point to a UTF-8 text file containing one stop word per line, which the `WordCloudGenerator` class in [`tools/words.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/words.py) automatically loads and applies during jieba tokenization.**

MediaCrawler is an open-source crawler framework that generates word cloud visualizations from collected text data using Chinese word segmentation. The system filters out common noise terms by referencing an external stop words file, allowing you to fully customize which tokens get excluded from the final visualization without modifying the core tokenization logic.

## How Stop Word Filtering Works in MediaCrawler

The word cloud generation pipeline in MediaCrawler uses **jieba** for text tokenization, then filters tokens against a configurable stop word list. Understanding this three-stage workflow helps you customize the behavior effectively.

### Configuration Layer ([`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py))

The file path for stop words is defined as the constant `STOP_WORDS_FILE`. By default, this points to [`./docs/hit_stopwords.txt`](https://github.com/NanmiCoder/MediaCrawler/blob/main/./docs/hit_stopwords.txt), but you can redirect this to any custom file path.

### Processing Layer ([`tools/words.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/words.py))

The **`WordCloudGenerator`** class handles the actual filtering. When instantiated, it calls `self.load_stop_words()` to read the file specified in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py) into a Python set. During tokenization, it checks `word not in self.stop_words` to determine whether to include each token in the word cloud.

## Step-by-Step Configuration

Follow these steps to implement your own stop word list.

### Create a Custom Stop Words File

Create a UTF-8 encoded text file with one stop word per line. Place this file anywhere in your repository.

```text

# my_stopwords.txt

the
and
of
to
认为
是的

```

### Update the Configuration Path

Edit [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py) to point to your new file:

```python

# config/base_config.py

STOP_WORDS_FILE = "./my_stopwords.txt"

```

If you prefer to keep the default [`hit_stopwords.txt`](https://github.com/NanmiCoder/MediaCrawler/blob/main/hit_stopwords.txt) location, simply edit that file directly to add or remove terms.

### Environment Variable Override (Optional)

For deployments where you cannot modify the config file, use an environment variable. Update [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py) to read from the environment with a fallback:

```python
import os

STOP_WORDS_FILE = os.getenv(
    "MEDIA_CRAWLER_STOP_WORDS",
    "./docs/hit_stopwords.txt"
)

```

Then export the variable before running:

```bash
export MEDIA_CRAWLER_STOP_WORDS=./my_stopwords.txt
python -m main

```

## Implementation Details

The `WordCloudGenerator` class in [`tools/words.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/words.py) automatically applies your custom list when generating visualizations. After configuration changes, the next crawler run loads the new stop words without requiring code modifications to the generator itself.

```python
from tools.words import WordCloudGenerator

wc = WordCloudGenerator()

# The generate_cloud method automatically filters against your custom stop words

cloud_svg = wc.generate_cloud(text_corpus)

```

## Summary

- **Stop word configuration** in MediaCrawler is controlled by the `STOP_WORDS_FILE` constant in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py).
- **The default path** points to [`./docs/hit_stopwords.txt`](https://github.com/NanmiCoder/MediaCrawler/blob/main/./docs/hit_stopwords.txt), but you can specify any UTF-8 text file.
- **Format requirements** are simple: one word per line, encoded in UTF-8.
- **The `WordCloudGenerator` class** in [`tools/words.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/words.py) loads and applies these filters automatically during the jieba tokenization process.
- **Environment variables** provide a flexible alternative for production deployments.

## Frequently Asked Questions

### What file format does MediaCrawler expect for stop words?

MediaCrawler expects a plain text file encoded in UTF-8 with one stop word per line. The `WordCloudGenerator` class reads this file in [`tools/words.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/words.py) and stores the terms in a Python set for O(1) lookup during tokenization.

### Can I use multiple stop word files simultaneously?

The current implementation in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py) accepts a single file path. To use multiple files, concatenate them into one file or modify [`tools/words.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/words.py) to loop through a list of paths in the `load_stop_words()` method.

### Does MediaCrawler support English stop words?

Yes. While the default [`hit_stopwords.txt`](https://github.com/NanmiCoder/MediaCrawler/blob/main/hit_stopwords.txt) contains Chinese stop words, the filtering mechanism is language-agnostic. Simply create a file with English words like "the", "and", and "of", set `STOP_WORDS_FILE` to its path, and the jieba tokenizer will filter them alongside any Chinese terms.

### Do I need to restart the crawler after changing stop words?

Yes. The `WordCloudGenerator` loads stop words during instantiation, so you must restart the crawler for changes to take effect. The file is not hot-reloaded during runtime.