# Corpus File Encoding in rime-lmdg: Default UTF-8 Settings and Custom Configuration

> Discover rime-lmdg's default UTF-8 corpus file encoding. Learn how to specify custom encodings using FILE_ENCODING or open arguments for flexible file handling.

- Repository: [amzxyz/rime-lmdg](https://github.com/amzxyz/rime-lmdg)
- Tags: internals
- Published: 2026-02-24

---

**The rime-lmdg repository defaults to UTF-8 for all corpus files, allowing custom encodings via the `FILE_ENCODING` constant or direct `open()` argument overrides.**

The `wanxiang` branch of the [amzxyz/rime-lmdg](https://github.com/amzxyz/rime-lmdg) repository provides data preprocessing scripts for Chinese language models. Every script that reads or writes corpus files assumes **UTF-8** encoding, ensuring full Unicode compatibility for Chinese characters without byte-order mark complications.

## Default UTF-8 Encoding Implementation

The default encoding is explicitly set in three primary entry points for corpus preprocessing:

- **`wanxiang/json语料解析.py`** – Opens source and destination files with `encoding='utf-8'` at lines 4‑5 when parsing JSON-line corpora.
- **`wanxiang/TXT清洗.py`** – Reads and writes plain-text files using `encoding='utf-8'` at lines 34‑35 during line-by-line cleaning operations.
- **`wanxiang/多线程分词.py`** – Defines a module-level constant `FILE_ENCODING = "utf-8"` at line 11, which is passed to `open()` calls at line 37 during multi-threaded tokenization.

Because UTF-8 supports the entire Unicode range including CJK characters, it serves as the safest default for multilingual corpora.

## How to Specify Custom Encoding for Corpus Files

When processing legacy corpora stored in **GB18030** or other regional encodings, you can override the default in two ways:

### Modify the FILE_ENCODING Constant

For the multi-threaded tokenizer, edit the constant declaration in `wanxiang/多线程分词.py`:

```python

# Line 11 in 多线程分词.py

FILE_ENCODING = "gb18030"  # Replace "utf-8" with your target codec

```

The script propagates this value to all `open()` calls, including the input file reader at line 37.

### Override open() Calls Directly

For ad-hoc processing in other scripts, modify the `encoding=` argument directly. In `wanxiang/json语料解析.py`, change lines 4‑5:

```python
with open(input_file, 'r', encoding='gb18030') as infile:
    with open(output_file, 'w', encoding='gb18030') as outfile:

```

This method requires editing the source file but takes effect immediately without external dependencies.

## Practical Code Examples

Run the JSON cleaner with GB18030 encoding after editing the file:

```bash
python wanxiang/json语料解析.py \
    --input legacy_corpus.json \
    --output cleaned_utf8.txt

```

Configure the multi-threaded tokenizer for legacy encodings:

```python

# In wanxiang/多线程分词.py, line 11

FILE_ENCODING = "gb18030"

# Then execute normally

python wanxiang/多线程分词.py input.txt output.txt

```

## Summary

- **UTF-8 is the hardcoded default** in `json语料解析.py`, `TXT清洗.py`, and `多线程分词.py`.
- **Change `FILE_ENCODING`** in `多线程分词.py` (line 11) to switch the tokenizer to a custom codec.
- **Edit `open()` arguments** directly in other scripts for one-off encoding changes.
- **No environment variables** are required; all encoding configuration is pure Python.

## Frequently Asked Questions

### What is the default encoding for corpus files in rime-lmdg?

All scripts in the `wanxiang` branch use **UTF-8** as the default encoding. The codebase explicitly passes `encoding='utf-8'` to Python's built-in `open()` function in every file I/O operation.

### How do I process GB18030 encoded corpus files?

Edit `wanxiang/多线程分词.py` and change the `FILE_ENCODING` constant from `"utf-8"` to `"gb18030"` at line 11. Alternatively, modify the `encoding=` parameter in the `open()` calls of `json语料解析.py` or `TXT清洗.py` to match your source file's encoding.

### Can I use environment variables to set the encoding?

No. The rime-lmdg scripts do not read encoding settings from environment variables. You must modify the Python source code directly, either by updating the `FILE_ENCODING` constant or the individual `open()` calls.

### Which script should I modify for batch processing with custom encoding?

Use `wanxiang/多线程分词.py` for batch processing. Changing the single `FILE_ENCODING` constant at line 11 applies the custom encoding to all file operations in the multi-threaded pipeline, making it the most efficient choice for large-scale corpus conversion.