# How to Customize the Stopwords List for Corpus Preprocessing in rime-lmdg

> Customize stopwords in rime-lmdg to improve segmentation accuracy. Set STOPWORDS_ENABLED True and populate the 停用词表 directory with your custom UTF-8 encoded stopword lists.

- Repository: [amzxyz/rime-lmdg](https://github.com/amzxyz/rime-lmdg)
- Tags: how-to-guide
- Published: 2026-02-24

---

**Set `STOPWORDS_ENABLED = True` in `语法模型构建.py` and populate the `停用词表` directory with UTF-8 encoded text files containing one stopword per line to filter noise during Jieba tokenization.**

The rime-lmdg repository provides a specialized pipeline for building grammar models for Rime input method engines. Customizing the **stopwords list** used during corpus preprocessing directly improves **segmentation accuracy** by eliminating high-frequency noise tokens before n-gram statistics are calculated. According to the source code, the preprocessing system loads stopwords from a configurable directory and filters them during the `segment_corpus()` phase.

## Understanding the Stopwords Architecture

The stopwords mechanism in rime-lmdg consists of four interconnected components defined in `语法模型构建.py`:

- **`STOPWORDS_DIR`**: A constant defaulting to `'停用词表'` that specifies the folder path containing stopword files (lines 19-20)
- **`STOPWORDS_ENABLED`**: A boolean flag defaulting to `False` that controls whether filtering is active (line 27)
- **`load_stopwords_from_directory()`**: A utility function that recursively reads all text files in the directory and builds a Python `set` for O(1) lookup performance (lines 31-45)
- **`segment_corpus()`**: The processing function that applies Jieba tokenization and removes any token found in the stopwords set before writing to `分词后.txt` (lines 100-109)

## Step-by-Step Customization Guide

### 1. Prepare the Stopwords Directory

Create or edit the `停用词表` folder at the repository root. Add one or more UTF-8 encoded `.txt` files with one stopword per line:

```text
的
了
以及
和
是
在

```

The `load_stopwords_from_directory()` function recursively scans this directory, so you can organize stopwords into multiple files (e.g., [`common.txt`](https://github.com/amzxyz/rime-lmdg/blob/main/common.txt), [`domain.txt`](https://github.com/amzxyz/rime-lmdg/blob/main/domain.txt), [`noise.txt`](https://github.com/amzxyz/rime-lmdg/blob/main/noise.txt)) for better maintainability.

### 2. Enable Stopword Filtering

Open `语法模型构建.py` and modify the configuration constants near line 27:

```python

# 语法模型构建.py – line 27

STOPWORDS_ENABLED = True          # ✅ Activate stopword filtering

STOPWORDS_DIR = '停用词表'         # Modify if using a custom path

```

When `STOPWORDS_ENABLED` is `True`, the `main()` function passes the loaded stopwords set to `segment_corpus()`, which filters tokens after Jieba segmentation.

### 3. (Optional) Extend the Loader Logic

The default loader strips whitespace but treats all text literally. For case-insensitive matching or comment support, modify `load_stopwords_from_directory()` around lines 31-45:

```python
def load_stopwords_from_directory(directory):
    """Load stopwords from every file in *directory* with comment support."""
    stopwords = set()
    if not os.path.exists(directory):
        print(f"警告：停用词目录 {directory} 不存在，未加载任何停用词。")
        return stopwords
    
    for root, _, files in os.walk(directory):
        for file in files:
            file_path = os.path.join(root, file)
            with open(file_path, 'r', encoding='utf-8') as f:
                for line in f:
                    word = line.strip()
                    # Ignore empty lines and lines starting with #

                    if word and not word.startswith('#'):
                        stopwords.add(word.lower())  # Normalize to lowercase

    print(f"已加载 {len(stopwords)} 个停用词。")
    return stopwords

```

This modification ignores lines starting with `#` and converts all entries to lowercase, useful when processing mixed-case corpora.

## Verification and Testing

After configuration, execute the pipeline and inspect the segmented output:

```bash

# Run the preprocessing pipeline

python 语法模型构建.py

# Verify stopwords were removed from the output

head -n 20 分词后.txt

```

You should observe that tokens listed in your `停用词表` files (e.g., `的`, `了`) no longer appear in `分词后.txt`. This reduction in noise tokens improves the statistical quality of downstream n-gram generation and the final language model accuracy.

## Summary

- **Location**: Stopwords reside in the `停用词表` directory as plain UTF-8 text files
- **Activation**: Toggle `STOPWORDS_ENABLED = True` in `语法模型构建.py` (line 27) to enable filtering
- **Customization**: Add domain-specific terms to reduce noise in specialized corpora
- **Performance**: The implementation uses a Python `set` for constant-time lookup during segmentation
- **Verification**: Check `分词后.txt` to confirm target tokens have been removed before n-gram processing

## Frequently Asked Questions

### Where does rime-lmdg store the default stopwords list?

The repository expects stopwords in the `停用词表` directory at the project root, as defined by the `STOPWORDS_DIR` constant in `语法模型构建.py` (line 20). This folder is not created automatically; you must manually add `.txt` files containing one stopword per line for the `load_stopwords_from_directory()` function to process.

### What file format should I use for custom stopwords?

Use UTF-8 encoded plain text files with the `.txt` extension. Each line should contain exactly one stopword without punctuation or delimiters. The loader in lines 31-45 strips whitespace automatically but preserves internal characters, so entries like `的` or `然而` work correctly.

### How do I completely disable stopword filtering?

Set `STOPWORDS_ENABLED = False` in `语法模型构建.py` (line 27). When disabled, the `segment_corpus()` function skips the set lookup entirely, passing all Jieba tokens through to `分词后.txt` regardless of the `停用词表` directory contents.

### Can I use multiple stopword files for different domains?

Yes. The `load_stopwords_from_directory()` function recursively walks the `STOPWORDS_DIR` directory using `os.walk()`, loading every `.txt` file it finds. Organize domain-specific stopwords into separate files (e.g., [`medical.txt`](https://github.com/amzxyz/rime-lmdg/blob/main/medical.txt), [`legal.txt`](https://github.com/amzxyz/rime-lmdg/blob/main/legal.txt)) within subdirectories, and the loader will aggregate them into a single unified set for filtering.