How to Develop a Migration Strategy for Legacy RIME Dictionaries to the Wanxiang Format Without Data Loss

The amzxyz/rime-lmdg repository provides a lossless, two-phase Python pipeline that converts legacy RIME dictionaries to the Wanxiang format by first normalizing tone marks and then injecting auxiliary codes, while preserving YAML headers, comments, and user-db markers through atomic read-modify-write operations.

Migrating from legacy RIME .dict.yaml files to the Wanxiang schema requires handling subtle differences in pinyin annotation and auxiliary-code segmentation. The Wanxiang project supplies dedicated scripts that automate this conversion without stripping metadata or corrupting existing user data. This guide details the exact file paths, function behaviors, and command sequences needed to execute a zero-downtime migration.

Understanding the Two-Phase Migration Architecture

The Wanxiang migration strategy operates as two orthogonal, reversible transformations:

1. Tone Normalization — Ensures every character carries a valid pinyin-with-tone annotation using the bundled pypinyin library.
2. Auxiliary-Code Injection — Appends user-defined auxiliary segments (;aux) to each syllable while preserving any existing suffixes.

Both phases are implemented as pure-Python scripts in the wanxiang/ directory that perform atomic read-modify-write cycles. The original files remain untouched until you explicitly overwrite them, eliminating the risk of destructive changes.

Risk Mitigation Mechanisms

The scripts implement specific guards against common data-loss scenarios:

• Header Preservation: Lines starting with ---, #, or YAML front-matter delimiters are copied verbatim without parsing.

• User-Db Detection: The is_userdb_head() function recognizes Rime-specific markers (#@/db_type\tuserdb and # Rime user dictionary) to switch column layouts automatically.

• Selective Skipping: A skip_set (containing entries like duoyin.dict.yaml and zi.dict.yaml) prevents unintended rewrites of dictionaries that follow custom schemas.

• Suffix Isolation: The regular expression AUX_SEP_REGEX = r'[;\[]' splits existing pinyin from auxiliary suffixes (;xx or [xx]), re-attaching them after transformation.

• Atomic Output: All writes target a separate destination directory (dst), leaving source files intact until manual deployment.

Phase 1 — Tone Normalization

Begin the migration by running wanxiang/rime固定或用户词典刷新为带声调编码.py. This script recursively processes both fixed dictionaries (.dict.yaml) and Rime user databases, injecting tone marks via pypinyin.Style.TONE.

Core Processing Logic

The entry point process_files(path_in, path_out) traverses the input directory and delegates to process_single_file(src, dst) for each .txt or .yaml file. Inside process_single_file(), the script:

1. Detects YAML headers and comments (copied unchanged).
2. Checks for user-db markers via is_userdb_head() to set a userdb boolean flag.
3. Dispatches each data line to either normal_line(cols) (fixed dictionaries) or userdb_line(cols) (user databases).
4. Applies tone_mark(seg) to generate tone-accurate pinyin while preserving existing auxiliary suffixes.

Running the Tone Script

Execute the script with your legacy dictionary path and a temporary output location:

python3 wanxiang/rime固定或用户词典刷新为带声调编码.py \
    /path/to/legacy_dicts \
    /path/to/toned_output \
    pinyin_data   # optional: folder with custom phrase dictionaries

Input Example (fixed dictionary, missing pinyin):

独孤	12345

Output After Tone Normalization:

独孤	 dú;gu 12345

The space preceding the pinyin segment is intentional; the script preserves any existing auxiliary suffixes found after the AUX_SEP_REGEX delimiters.

Phase 2 — Auxiliary-Code Injection

After tone normalization, run wanxiang/rime固定或用户词典刷新为带辅助码编码.py to append auxiliary codes. This script requires a metadata file mapping characters to their auxiliary codes.

Preparing the Auxiliary Metadata

Create a plain-text file (e.g., aux.txt) where each line follows 汉字<TAB>aux. The parser tolerates three formats: char\taux, char\t;aux, or char\taux. Example:

独	du
孤	gu

The load_aux_metadata(path) function parses this into a dictionary consumed by build_seg_by_aux(word, aux_map), which returns auxiliary strings for each character in a word.

Merging Codes with Pinyin

The refresh_aux(cols, word, aux_map, userdb) function reconstructs the pinyin field by merging tone-marked syllables with auxiliary codes in the format pinyin;aux. For fixed dictionaries, it operates on column 1; for user-db files, it processes column 0 and ensures the output ends with a trailing space as required by Rime.

Execute the script (adjust the constants inside the file or use environment variables):

python3 wanxiang/rime固定或用户词典刷新为带辅助码编码.py

Input (from Phase 1):

独孤	 dú;gu 12345

Final Output:

独孤	 dú;du gu;gu 12345

Each syllable now carries both tone information and the injected auxiliary code separated by a semicolon.

Complete End-to-End Migration Workflow

Follow this sequence to migrate your production dictionaries without service interruption:

1. Backup: Create a complete copy of your existing dictionary tree.

   cp -r rime/dicts rime/dicts.backup

2. Tone Normalization: Run the tone script against the backup, outputting to an intermediate directory.

   python3 wanxiang/rime固定或用户词典刷新为带声调编码.py \
       rime/dicts.backup \
       rime/dicts_toned \
       pinyin_data

3. Prepare Auxiliary File: Generate aux.txt with character-to-code mappings.

4. Auxiliary Injection: Process the toned dictionaries.

   python3 wanxiang/rime固定或用户词典刷新为带辅助码编码.py
   # Ensure INPUT_PATH points to rime/dicts_toned and OUTPUT_PATH to rime/dicts_wanxiang
   

5. Verification: Spot-check entries to confirm the transformation.

   • Fixed dictionaries should show word\tpinyin;aux pinyin;aux\tweight.
   • User-db files should show pinyin;aux pinyin;aux \tword\t....

6. Deploy: Replace your Rime schema's dicts/ folder with the final output directory and reload the input method.

One-Liner Automation

For batch processing in CI/CD pipelines:

# Step 1: Tone

python3 wanxiang/rime固定或用户词典刷新为带声调编码.py legacy temp pinyin_data && \

# Step 2: Aux

python3 wanxiang/rime固定或用户词典刷新为带辅助码编码.py

# Result available in the OUTPUT_PATH defined in the auxiliary script

Summary

• Tone Normalization is performed by wanxiang/rime固定或用户词典刷新为带声调编码.py, which uses pypinyin to inject tone marks while preserving auxiliary suffixes via AUX_SEP_REGEX.
• Auxiliary-Code Injection is handled by wanxiang/rime固定或用户词典刷新为带辅助码编码.py, merging pinyin;aux segments through refresh_aux() based on a user-supplied metadata file.
• Data Integrity is guaranteed by atomic writes to separate output directories, verbatim copying of YAML headers/comments, and automatic detection of user-db formats via header markers.
• Rollback Capability exists at every stage because source files are never modified in-place; only the final deployment step alters the live Rime configuration.

Frequently Asked Questions

What is the Wanxiang format and why migrate to it?

The Wanxiang format is a modernized RIME dictionary schema that enforces strict tonal pinyin annotations and structured auxiliary-code segmentation to improve input accuracy and candidate ranking. According to the amzxyz/rime-lmdg source code, it separates tonal information from auxiliary hints using semicolon delimiters (e.g., ni;hao), enabling more precise linguistic modeling than legacy flat-pinyin dictionaries.

How does the migration script prevent data loss when processing user databases?

The script detects user-db files by scanning for the markers #@/db_type\tuserdb or # Rime user dictionary in is_userdb_head(). Once flagged, it switches to the user-db column layout where the first column is the pinyin segment, ensuring frequency and timestamp columns remain aligned. All original comments and YAML front-matter are copied verbatim before any transformation occurs.

Can I migrate dictionaries that already contain auxiliary codes?

Yes. The tone-normalization script uses AUX_SEP_REGEX = r'[;\[]' to isolate existing auxiliary suffixes (whether semicolon or bracket format) before applying tone marks, then re-attaches them afterward. The auxiliary-code injection script subsequently merges these preserved suffixes with the new auxiliary metadata, preventing duplication or truncation.

What happens if the tone script encounters a dictionary in the skip list?

The skip_set (defined in wanxiang/rime固定或用户词典刷新为带声调编码.py) contains filenames like duoyin.dict.yaml and zi.dict.yaml that follow non-standard schemas. When the script encounters these files, it copies them unchanged to the output directory, ensuring that specialized dictionaries requiring manual handling are not corrupted by automated tone injection.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}