Corpus File Encoding in rime-lmdg: Default UTF-8 Settings and Custom Configuration

The rime-lmdg repository defaults to UTF-8 for all corpus files, allowing custom encodings via the FILE_ENCODING constant or direct open() argument overrides.

The wanxiang branch of the amzxyz/rime-lmdg repository provides data preprocessing scripts for Chinese language models. Every script that reads or writes corpus files assumes UTF-8 encoding, ensuring full Unicode compatibility for Chinese characters without byte-order mark complications.

Default UTF-8 Encoding Implementation

The default encoding is explicitly set in three primary entry points for corpus preprocessing:

• wanxiang/json语料解析.py – Opens source and destination files with encoding='utf-8' at lines 4‑5 when parsing JSON-line corpora.
• wanxiang/TXT清洗.py – Reads and writes plain-text files using encoding='utf-8' at lines 34‑35 during line-by-line cleaning operations.
• wanxiang/多线程分词.py – Defines a module-level constant FILE_ENCODING = "utf-8" at line 11, which is passed to open() calls at line 37 during multi-threaded tokenization.

Because UTF-8 supports the entire Unicode range including CJK characters, it serves as the safest default for multilingual corpora.

How to Specify Custom Encoding for Corpus Files

When processing legacy corpora stored in GB18030 or other regional encodings, you can override the default in two ways:

Modify the FILE_ENCODING Constant

For the multi-threaded tokenizer, edit the constant declaration in wanxiang/多线程分词.py:

# Line 11 in 多线程分词.py

FILE_ENCODING = "gb18030"  # Replace "utf-8" with your target codec

The script propagates this value to all open() calls, including the input file reader at line 37.

Override open() Calls Directly

For ad-hoc processing in other scripts, modify the encoding= argument directly. In wanxiang/json语料解析.py, change lines 4‑5:

with open(input_file, 'r', encoding='gb18030') as infile:
    with open(output_file, 'w', encoding='gb18030') as outfile:

This method requires editing the source file but takes effect immediately without external dependencies.

Practical Code Examples

Run the JSON cleaner with GB18030 encoding after editing the file:

python wanxiang/json语料解析.py \
    --input legacy_corpus.json \
    --output cleaned_utf8.txt

Configure the multi-threaded tokenizer for legacy encodings:

# In wanxiang/多线程分词.py, line 11

FILE_ENCODING = "gb18030"

# Then execute normally

python wanxiang/多线程分词.py input.txt output.txt

Summary

• UTF-8 is the hardcoded default in json语料解析.py, TXT清洗.py, and 多线程分词.py.
• Change FILE_ENCODING in 多线程分词.py (line 11) to switch the tokenizer to a custom codec.
• Edit open() arguments directly in other scripts for one-off encoding changes.
• No environment variables are required; all encoding configuration is pure Python.

Frequently Asked Questions

What is the default encoding for corpus files in rime-lmdg?

All scripts in the wanxiang branch use UTF-8 as the default encoding. The codebase explicitly passes encoding='utf-8' to Python's built-in open() function in every file I/O operation.

How do I process GB18030 encoded corpus files?

Edit wanxiang/多线程分词.py and change the FILE_ENCODING constant from "utf-8" to "gb18030" at line 11. Alternatively, modify the encoding= parameter in the open() calls of json语料解析.py or TXT清洗.py to match your source file's encoding.

Can I use environment variables to set the encoding?

No. The rime-lmdg scripts do not read encoding settings from environment variables. You must modify the Python source code directly, either by updating the FILE_ENCODING constant or the individual open() calls.

Which script should I modify for batch processing with custom encoding?

Use wanxiang/多线程分词.py for batch processing. Changing the single FILE_ENCODING constant at line 11 applies the custom encoding to all file operations in the multi-threaded pipeline, making it the most efficient choice for large-scale corpus conversion.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}