How rime固定或用户词典刷新为带声调编码.py Converts Auxiliary Codes in Rime Dictionaries

The script preserves auxiliary code suffixes like ;sc or [xx] by isolating the phonetic root using the AUX_SEP_REGEX pattern, converting only the root to tone-marked pinyin with pypinyin, then reattaching the original auxiliary suffix unchanged.

The rime固定或用户词典刷新为带声调编码.py script in the amzxyz/rime-lmdg repository automates the conversion of flat pinyin into tone-marked encoding for Rime input method dictionaries. This utility processes both standard dictionary files and user database exports, ensuring that auxiliary codes—supplemental suffixes used for double pinyin or shape-based filtering—remain intact during the tone conversion process. Understanding how the script handles auxiliary code (辅助码) separation is essential for maintaining complex Rime schemas without breaking existing encoding schemes.

How Auxiliary Code Detection Works

The script identifies auxiliary code boundaries using a dedicated regular expression defined at line 20 of wanxiang/rime固定或用户词典刷新为带声调编码.py:

AUX_SEP_REGEX = r'[;\[]'

This pattern matches two specific separator characters that indicate the start of an auxiliary suffix:

• Semicolon (;) – Used for style codes like ;sc (simplified Chinese preference) or ;um (user-defined markers)
• Opening bracket ([) – Used for structural annotations like [xx] or [shape]

When processing dictionary lines, the script applies this regex to split each pinyin segment precisely at the boundary between the phonetic root and its auxiliary suffix.

The Three-Step Conversion Process

The conversion logic resides primarily in the tone_mark() function (lines 60-66) and follows a strict segmentation workflow to ensure auxiliary codes pass through unmodified.

Step 1: Isolating the Phonetic Root

For each pinyin segment, the script splits the string using AUX_SEP_REGEX to separate the tone-convertible portion from the auxiliary metadata:

root = re.split(AUX_SEP_REGEX, seg)[0]  # Text before ; or [

suffix = seg[len(root):]                # Everything from ; or [ onwards

This extraction ensures that a segment like bin;sc decomposes into root bin and suffix ;sc, while zhang[xx] becomes zhang and [xx].

Step 2: Applying Tone Marks with pypinyin

With the root isolated, the script calls the pypinyin library to inject tone diacritics:

from pypinyin import pinyin, Style

# Inside tone_mark():

toned_root = pinyin(root, style=Style.TONE, heteronym=False, strict=False)[0][0]

The Style.TONE parameter instructs pypinyin to return the phonetic string with appropriate tone marks (e.g., transforming bin into bīn or zhang into zhāng), while auxiliary suffixes remain excluded from this linguistic processing.

Step 3: Reconstructing the Segment

The final assembly concatenates the tone-marked root with the preserved suffix:

new_seg = toned_root + suffix

This approach guarantees that bin;sc becomes bīn;sc—the tone mark appears only on the phonetic component, while the auxiliary code ;sc maintains its original form and position.

Dictionary Format Handlers

The script implements two distinct processing pipelines to accommodate different Rime dictionary structures, both preserving auxiliary codes using the same core logic.

Processing Fixed Dictionary Lines with normal_line()

Located at lines 71-95 in wanxiang/rime固定或用户词典刷新为带声调编码.py, the normal_line() function handles standard .dict.yaml files with three-column formatting (entry, pinyin, frequency). It iterates through space-separated pinyin segments in the second column:

def normal_line(cols):
    # cols[1] contains space-separated pinyin segments

    segs = cols[1].split(' ')
    new_segs = [tone_mark(seg) for seg in segs]
    cols[1] = ' '.join(new_segs)
    return '\t'.join(cols)

Each segment passes through tone_mark(), ensuring that auxiliary codes attached to any syllable in multi-character words receive proper treatment.

Processing User Database Lines with userdb_line()

The userdb_line() function (lines 98-116) processes userdb.txt exports where the first column contains the pinyin string and the second column contains the Chinese character entry:

def userdb_line(cols):
    # cols[0] contains space-separated pinyin

    segs = cols[0].split(' ')
    new_segs = [tone_mark(seg) for seg in segs]
    cols[0] = ' '.join(new_segs)
    return '\t'.join(cols)

This handler applies identical auxiliary code preservation logic to user-specific vocabulary entries, maintaining consistency between fixed dictionaries and personal user data.

Complete Code Examples

Example 1: Fixed Dictionary with Semicolon Auxiliary Codes

Input line (tab-separated columns):

编码	bin;sc	100

Processing breakdown:

Original Segment	Phonetic Root	Auxiliary Suffix	Tone-Added Root	Final Segment
bin;sc	bin	;sc	bīn	bīn;sc

Output line:

编码	bīn;sc	100

Example 2: User Database with Bracket Auxiliary Codes

Input line (userdb format):

bin;sc ma[shape]	编码

Processing flow:

1. Split first column into segments: ['bin;sc', 'ma[shape]']
2. Process bin;sc → bīn;sc
3. Process ma[shape] → mǎ[shape]
4. Rejoin with tab separator

Output line:

bīn;sc mǎ[shape]	编码

Both examples demonstrate that only the phonetic root receives tonal annotation, while auxiliary markers (;sc, [shape]) remain unmodified and properly positioned.

Summary

• Auxiliary code preservation relies on AUX_SEP_REGEX (r'[;\[]') to detect suffix boundaries at semicolons or opening brackets in wanxiang/rime固定或用户词典刷新为带声调编码.py.
• Root isolation occurs through regex splitting, separating tone-convertible phonetics from static metadata before processing.
• Tone conversion uses pypinyin with Style.TONE exclusively on the isolated root, ensuring linguistic accuracy without corrupting auxiliary schemes.
• Format flexibility is provided by normal_line() (lines 71-95) for standard dictionaries and userdb_line() (lines 98-116) for user database files, both implementing identical auxiliary code handling.
• Suffix reconstruction maintains the exact original auxiliary code characters, preventing data loss in complex Rime input schemas.

Frequently Asked Questions

What are auxiliary codes (辅助码) in Rime dictionaries?

Auxiliary codes are supplemental suffixes appended to pinyin segments to enable additional filtering or encoding schemes, such as double pinyin shortcuts or character shape components. Common formats include semicolon-separated codes (;sc for simplified Chinese preference) or bracket-enclosed annotations ([xx] for structural markers). These suffixes follow the phonetic spelling but precede any frequency or weight values in dictionary entries.

Does the script modify the auxiliary code suffixes during conversion?

No, the script explicitly preserves auxiliary code suffixes unchanged. The tone_mark() function isolates the auxiliary portion using AUX_SEP_REGEX, processes only the phonetic root through pypinyin, then reattaches the original suffix via string concatenation. This ensures that ;sc, [xx], or any custom auxiliary markers remain exactly as they appeared in the source dictionary.

How does the script handle multiple auxiliary codes in a single pinyin segment?

The script handles multiple auxiliary codes by treating everything from the first separator match as the preserved suffix. Since AUX_SEP_REGEX matches either ; or [, a segment like pin;sc[alt] splits at the first character (;), resulting in root pin and suffix ;sc[alt]. The entire suffix string passes through unchanged, maintaining complex compound auxiliary codes intact while only the leading phonetic root receives tone marks.

Which dictionary formats does this script support?

The script supports two primary Rime dictionary formats implemented in wanxiang/rime固定或用户词典刷新为带声调编码.py: standard fixed dictionaries (three-column tab-separated format processed by normal_line() at lines 71-95) and user database exports (two-column format with pinyin first, processed by userdb_line() at lines 98-116). Both handlers iterate through space-separated pinyin segments and apply identical auxiliary code preservation logic, ensuring consistent tone conversion across dictionary types.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}