How to Customize the Stopwords List for Corpus Preprocessing in rime-lmdg

Set STOPWORDS_ENABLED = True in 语法模型构建.py and populate the 停用词表 directory with UTF-8 encoded text files containing one stopword per line to filter noise during Jieba tokenization.

The rime-lmdg repository provides a specialized pipeline for building grammar models for Rime input method engines. Customizing the stopwords list used during corpus preprocessing directly improves segmentation accuracy by eliminating high-frequency noise tokens before n-gram statistics are calculated. According to the source code, the preprocessing system loads stopwords from a configurable directory and filters them during the segment_corpus() phase.

Understanding the Stopwords Architecture

The stopwords mechanism in rime-lmdg consists of four interconnected components defined in 语法模型构建.py:

• STOPWORDS_DIR: A constant defaulting to '停用词表' that specifies the folder path containing stopword files (lines 19-20)
• STOPWORDS_ENABLED: A boolean flag defaulting to False that controls whether filtering is active (line 27)
• load_stopwords_from_directory(): A utility function that recursively reads all text files in the directory and builds a Python set for O(1) lookup performance (lines 31-45)
• segment_corpus(): The processing function that applies Jieba tokenization and removes any token found in the stopwords set before writing to 分词后.txt (lines 100-109)

Step-by-Step Customization Guide

1. Prepare the Stopwords Directory

Create or edit the 停用词表 folder at the repository root. Add one or more UTF-8 encoded .txt files with one stopword per line:

的
了
以及
和
是
在

The load_stopwords_from_directory() function recursively scans this directory, so you can organize stopwords into multiple files (e.g., common.txt, domain.txt, noise.txt) for better maintainability.

2. Enable Stopword Filtering

Open 语法模型构建.py and modify the configuration constants near line 27:

# 语法模型构建.py – line 27

STOPWORDS_ENABLED = True          # ✅ Activate stopword filtering

STOPWORDS_DIR = '停用词表'         # Modify if using a custom path

When STOPWORDS_ENABLED is True, the main() function passes the loaded stopwords set to segment_corpus(), which filters tokens after Jieba segmentation.

3. (Optional) Extend the Loader Logic

The default loader strips whitespace but treats all text literally. For case-insensitive matching or comment support, modify load_stopwords_from_directory() around lines 31-45:

def load_stopwords_from_directory(directory):
    """Load stopwords from every file in *directory* with comment support."""
    stopwords = set()
    if not os.path.exists(directory):
        print(f"警告：停用词目录 {directory} 不存在，未加载任何停用词。")
        return stopwords
    
    for root, _, files in os.walk(directory):
        for file in files:
            file_path = os.path.join(root, file)
            with open(file_path, 'r', encoding='utf-8') as f:
                for line in f:
                    word = line.strip()
                    # Ignore empty lines and lines starting with #

                    if word and not word.startswith('#'):
                        stopwords.add(word.lower())  # Normalize to lowercase

    print(f"已加载 {len(stopwords)} 个停用词。")
    return stopwords

This modification ignores lines starting with # and converts all entries to lowercase, useful when processing mixed-case corpora.

Verification and Testing

After configuration, execute the pipeline and inspect the segmented output:

# Run the preprocessing pipeline

python 语法模型构建.py

# Verify stopwords were removed from the output

head -n 20 分词后.txt

You should observe that tokens listed in your 停用词表 files (e.g., 的, 了) no longer appear in 分词后.txt. This reduction in noise tokens improves the statistical quality of downstream n-gram generation and the final language model accuracy.

Summary

• Location: Stopwords reside in the 停用词表 directory as plain UTF-8 text files
• Activation: Toggle STOPWORDS_ENABLED = True in 语法模型构建.py (line 27) to enable filtering
• Customization: Add domain-specific terms to reduce noise in specialized corpora
• Performance: The implementation uses a Python set for constant-time lookup during segmentation
• Verification: Check 分词后.txt to confirm target tokens have been removed before n-gram processing

Frequently Asked Questions

Where does rime-lmdg store the default stopwords list?

The repository expects stopwords in the 停用词表 directory at the project root, as defined by the STOPWORDS_DIR constant in 语法模型构建.py (line 20). This folder is not created automatically; you must manually add .txt files containing one stopword per line for the load_stopwords_from_directory() function to process.

What file format should I use for custom stopwords?

Use UTF-8 encoded plain text files with the .txt extension. Each line should contain exactly one stopword without punctuation or delimiters. The loader in lines 31-45 strips whitespace automatically but preserves internal characters, so entries like 的 or 然而 work correctly.

How do I completely disable stopword filtering?

Set STOPWORDS_ENABLED = False in 语法模型构建.py (line 27). When disabled, the segment_corpus() function skips the set lookup entirely, passing all Jieba tokens through to 分词后.txt regardless of the 停用词表 directory contents.

Can I use multiple stopword files for different domains?

Yes. The load_stopwords_from_directory() function recursively walks the STOPWORDS_DIR directory using os.walk(), loading every .txt file it finds. Organize domain-specific stopwords into separate files (e.g., medical.txt, legal.txt) within subdirectories, and the loader will aggregate them into a single unified set for filtering.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}