# How to Develop a Migration Strategy for Legacy RIME Dictionaries to the Wanxiang Format Without Data Loss

> **The `amzxyz/rime-lmdg` repository provides a lossless, two-phase Python pipeline that converts legacy RIME dictionaries to the Wanxiang format by first normalizing tone marks and then injecting auxiliary codes, while preservi...

- Repository: [amzxyz/rime-lmdg](https://github.com/amzxyz/rime-lmdg)
- Tags: migration-guide
- Published: 2026-02-24

---

**The `amzxyz/rime-lmdg` repository provides a lossless, two-phase Python pipeline that converts legacy RIME dictionaries to the Wanxiang format by first normalizing tone marks and then injecting auxiliary codes, while preserving YAML headers, comments, and user-db markers through atomic read-modify-write operations.**

Migrating from legacy RIME [`.dict.yaml`](https://github.com/amzxyz/rime-lmdg/blob/main/.dict.yaml) files to the Wanxiang schema requires handling subtle differences in pinyin annotation and auxiliary-code segmentation. The Wanxiang project supplies dedicated scripts that automate this conversion without stripping metadata or corrupting existing user data. This guide details the exact file paths, function behaviors, and command sequences needed to execute a zero-downtime migration.

## Understanding the Two-Phase Migration Architecture

The Wanxiang migration strategy operates as two orthogonal, reversible transformations:

1. **Tone Normalization** — Ensures every character carries a valid **pinyin-with-tone** annotation using the bundled `pypinyin` library.
2. **Auxiliary-Code Injection** — Appends user-defined auxiliary segments (`;aux`) to each syllable while preserving any existing suffixes.

Both phases are implemented as pure-Python scripts in the `wanxiang/` directory that perform atomic read-modify-write cycles. The original files remain untouched until you explicitly overwrite them, eliminating the risk of destructive changes.

### Risk Mitigation Mechanisms

The scripts implement specific guards against common data-loss scenarios:

- **Header Preservation**: Lines starting with `---`, `#`, or YAML front-matter delimiters are copied verbatim without parsing.
- **User-Db Detection**: The `is_userdb_head()` function recognizes Rime-specific markers (`#@/db_type\tuserdb` and `# Rime user dictionary`) to switch column layouts automatically.

- **Selective Skipping**: A `skip_set` (containing entries like [`duoyin.dict.yaml`](https://github.com/amzxyz/rime-lmdg/blob/main/duoyin.dict.yaml) and [`zi.dict.yaml`](https://github.com/amzxyz/rime-lmdg/blob/main/zi.dict.yaml)) prevents unintended rewrites of dictionaries that follow custom schemas.
- **Suffix Isolation**: The regular expression `AUX_SEP_REGEX = r'[;\[]'` splits existing pinyin from auxiliary suffixes (`;xx` or `[xx]`), re-attaching them after transformation.
- **Atomic Output**: All writes target a separate destination directory (`dst`), leaving source files intact until manual deployment.

## Phase 1 — Tone Normalization

Begin the migration by running `wanxiang/rime固定或用户词典刷新为带声调编码.py`. This script recursively processes both fixed dictionaries ([`.dict.yaml`](https://github.com/amzxyz/rime-lmdg/blob/main/.dict.yaml)) and Rime user databases, injecting tone marks via `pypinyin.Style.TONE`.

### Core Processing Logic

The entry point `process_files(path_in, path_out)` traverses the input directory and delegates to `process_single_file(src, dst)` for each `.txt` or `.yaml` file. Inside `process_single_file()`, the script:

1. Detects YAML headers and comments (copied unchanged).
2. Checks for user-db markers via `is_userdb_head()` to set a `userdb` boolean flag.
3. Dispatches each data line to either `normal_line(cols)` (fixed dictionaries) or `userdb_line(cols)` (user databases).
4. Applies `tone_mark(seg)` to generate tone-accurate pinyin while preserving existing auxiliary suffixes.

### Running the Tone Script

Execute the script with your legacy dictionary path and a temporary output location:

```bash
python3 wanxiang/rime固定或用户词典刷新为带声调编码.py \
    /path/to/legacy_dicts \
    /path/to/toned_output \
    pinyin_data   # optional: folder with custom phrase dictionaries

```

**Input Example (fixed dictionary, missing pinyin):**

```

独孤	12345

```

**Output After Tone Normalization:**

```

独孤	 dú;gu 12345

```

The space preceding the pinyin segment is intentional; the script preserves any existing auxiliary suffixes found after the `AUX_SEP_REGEX` delimiters.

## Phase 2 — Auxiliary-Code Injection

After tone normalization, run `wanxiang/rime固定或用户词典刷新为带辅助码编码.py` to append auxiliary codes. This script requires a metadata file mapping characters to their auxiliary codes.

### Preparing the Auxiliary Metadata

Create a plain-text file (e.g., [`aux.txt`](https://github.com/amzxyz/rime-lmdg/blob/main/aux.txt)) where each line follows `汉字<TAB>aux`. The parser tolerates three formats: `char\taux`, `char\t;aux`, or `char\taux`. Example:

```

独	du
孤	gu

```

The `load_aux_metadata(path)` function parses this into a dictionary consumed by `build_seg_by_aux(word, aux_map)`, which returns auxiliary strings for each character in a word.

### Merging Codes with Pinyin

The `refresh_aux(cols, word, aux_map, userdb)` function reconstructs the pinyin field by merging tone-marked syllables with auxiliary codes in the format `pinyin;aux`. For fixed dictionaries, it operates on column 1; for user-db files, it processes column 0 and ensures the output ends with a trailing space as required by Rime.

Execute the script (adjust the constants inside the file or use environment variables):

```bash
python3 wanxiang/rime固定或用户词典刷新为带辅助码编码.py

```

**Input (from Phase 1):**

```

独孤	 dú;gu 12345

```

**Final Output:**

```

独孤	 dú;du gu;gu 12345

```

Each syllable now carries both tone information and the injected auxiliary code separated by a semicolon.

## Complete End-to-End Migration Workflow

Follow this sequence to migrate your production dictionaries without service interruption:

1. **Backup**: Create a complete copy of your existing dictionary tree.
   ```bash
   cp -r rime/dicts rime/dicts.backup
   ```

2. **Tone Normalization**: Run the tone script against the backup, outputting to an intermediate directory.
   ```bash
   python3 wanxiang/rime固定或用户词典刷新为带声调编码.py \
       rime/dicts.backup \
       rime/dicts_toned \
       pinyin_data
   ```

3. **Prepare Auxiliary File**: Generate [`aux.txt`](https://github.com/amzxyz/rime-lmdg/blob/main/aux.txt) with character-to-code mappings.

4. **Auxiliary Injection**: Process the toned dictionaries.
   ```bash
   python3 wanxiang/rime固定或用户词典刷新为带辅助码编码.py
   # Ensure INPUT_PATH points to rime/dicts_toned and OUTPUT_PATH to rime/dicts_wanxiang

   ```

5. **Verification**: Spot-check entries to confirm the transformation.
   - Fixed dictionaries should show `word\tpinyin;aux pinyin;aux\tweight`.
   - User-db files should show `pinyin;aux pinyin;aux \tword\t...`.

6. **Deploy**: Replace your Rime schema's `dicts/` folder with the final output directory and reload the input method.

### One-Liner Automation

For batch processing in CI/CD pipelines:

```bash

# Step 1: Tone

python3 wanxiang/rime固定或用户词典刷新为带声调编码.py legacy temp pinyin_data && \

# Step 2: Aux

python3 wanxiang/rime固定或用户词典刷新为带辅助码编码.py

# Result available in the OUTPUT_PATH defined in the auxiliary script

```

## Summary

- **Tone Normalization** is performed by `wanxiang/rime固定或用户词典刷新为带声调编码.py`, which uses `pypinyin` to inject tone marks while preserving auxiliary suffixes via `AUX_SEP_REGEX`.
- **Auxiliary-Code Injection** is handled by `wanxiang/rime固定或用户词典刷新为带辅助码编码.py`, merging `pinyin;aux` segments through `refresh_aux()` based on a user-supplied metadata file.
- **Data Integrity** is guaranteed by atomic writes to separate output directories, verbatim copying of YAML headers/comments, and automatic detection of user-db formats via header markers.
- **Rollback Capability** exists at every stage because source files are never modified in-place; only the final deployment step alters the live Rime configuration.

## Frequently Asked Questions

### What is the Wanxiang format and why migrate to it?

The Wanxiang format is a modernized RIME dictionary schema that enforces strict tonal pinyin annotations and structured auxiliary-code segmentation to improve input accuracy and candidate ranking. According to the `amzxyz/rime-lmdg` source code, it separates tonal information from auxiliary hints using semicolon delimiters (e.g., `ni;hao`), enabling more precise linguistic modeling than legacy flat-pinyin dictionaries.

### How does the migration script prevent data loss when processing user databases?

The script detects user-db files by scanning for the markers `#@/db_type\tuserdb` or `# Rime user dictionary` in `is_userdb_head()`. Once flagged, it switches to the user-db column layout where the first column is the pinyin segment, ensuring frequency and timestamp columns remain aligned. All original comments and YAML front-matter are copied verbatim before any transformation occurs.

### Can I migrate dictionaries that already contain auxiliary codes?

Yes. The tone-normalization script uses `AUX_SEP_REGEX = r'[;\[]'` to isolate existing auxiliary suffixes (whether semicolon or bracket format) before applying tone marks, then re-attaches them afterward. The auxiliary-code injection script subsequently merges these preserved suffixes with the new auxiliary metadata, preventing duplication or truncation.

### What happens if the tone script encounters a dictionary in the skip list?

The `skip_set` (defined in `wanxiang/rime固定或用户词典刷新为带声调编码.py`) contains filenames like [`duoyin.dict.yaml`](https://github.com/amzxyz/rime-lmdg/blob/main/duoyin.dict.yaml) and [`zi.dict.yaml`](https://github.com/amzxyz/rime-lmdg/blob/main/zi.dict.yaml) that follow non-standard schemas. When the script encounters these files, it copies them unchanged to the output directory, ensuring that specialized dictionaries requiring manual handling are not corrupted by automated tone injection.