How to Tune RIME Grammar Configuration: `collocation_max_length` and `collocation_penalty` Deep Dive

The collocation_max_length (default 7) and collocation_penalty (default -10) parameters control how RIME handles multi-character phrases, determining maximum phrase length and the scoring penalty applied to low-frequency collocations.

The RIME (Rime Input Method Engine) relies on statistical language modeling to rank input candidates. In the amzxyz/rime-lmdg repository, grammar configuration parameters defined in wanxiang/README.md (lines 43-47) govern how the engine segments text into collocations and applies scoring penalties against dictionary data. Tuning these values directly impacts the balance between input precision and candidate recall.

Understanding collocation_max_length and collocation_min_length

These parameters define the boundaries for what constitutes a valid collocation in the language model, interacting with entries in wanxiang/pinyin_data/单字.dict.yaml (single-character) and wanxiang/pinyin_data/词组.dict.yaml (multi-character phrase) files.

collocation_max_length: Maximum Phrase Boundary

According to wanxiang/README.md, collocation_max_length defaults to 7 characters. This parameter sets the upper limit on how long a multi-character phrase can remain intact before the engine splits it into smaller units for scoring.

Raising this value allows longer fixed expressions—such as "中华人民共和国"—to remain whole during candidate ranking, improving recognition of specialized terminology. Lowering the value forces more aggressive segmentation, which can increase processing speed and reduce false positives when handling very long input strings.

collocation_min_length: Minimum Sequence Threshold

The companion parameter collocation_min_length defaults to 2 characters. It establishes the shortest sequence that RIME will treat as a collocation rather than independent characters.

Decreasing this value to 1 would theoretically allow single characters, while keeping it at 2 ensures common bi-grams stay together. Raising it above 2 forces the engine to treat short pairs as separate entities, which helps when the dictionary contains ambiguous two-character entries that frequently combine incorrectly.

Tuning collocation_penalty and Related Scoring Parameters

Three penalty parameters in the RIME grammar configuration adjust scoring based on collocation quality and context.

collocation_penalty: Filtering Unlikely Phrases

collocation_penalty carries a default value of -10. This score modifier applies to candidates containing collocations that do not match high-frequency entries in the language model.

Making this penalty more negative (e.g., -15) aggressively discourages rare or unlikely word combinations, sharpening precision for standard vocabulary. Conversely, a milder penalty (e.g., -5) allows less common collocations to surface, which benefits specialized domains like medicine or law where technical phrases appear infrequently in general corpora.

non_collocation_penalty and weak_collocation_penalty: Context Adjustments

When the input context suggests a collocation should exist but the candidate lacks one, non_collocation_penalty (default -20) applies. A strongly negative value here forces the engine to prefer candidates containing collocations, improving output fluency for continuous text. Reducing the magnitude (e.g., to -10) relaxes this bias, proving useful when typing acronyms or isolated characters where phrase continuity matters less.

For collocations with probabilities below a specific threshold, weak_collocation_penalty (default -35) adds an extra penalty layer. Increasing the absolute value filters out rare collocations aggressively, reducing noise in general typing scenarios. Lowering it (e.g., to -20) preserves domain-specific jargon that might otherwise be eliminated as statistically weak.

How Collocation Parameters Interact During Scoring

The RIME engine applies these parameters in a specific sequence when evaluating input candidates against dictionary files:

1. Base Language Model Scoring: RIME first computes a probability score for each candidate based on the underlying dictionary data.

2. Collocation Detection: The engine checks whether candidates contain sequences falling between collocation_min_length and collocation_max_length.

3. Penalty Application:

   • If a collocation exists but shows low probability, collocation_penalty modifies the score.
   • If context indicates a collocation should be present but is missing, non_collocation_penalty applies.
   • If the collocation probability falls below the weak threshold, weak_collocation_penalty adds to the standard penalty.

This layered approach allows fine-grained control over the recall-precision trade-off, determining whether RIME prioritizes common phrases or preserves rare technical vocabulary.

Implementing Custom Grammar Configuration in RIME

To apply these parameters, create a custom schema file that overrides the defaults documented in wanxiang/README.md. The following Python snippet generates a custom.schema.yaml with tuned collocation settings:

import yaml

# Base schema skeleton referencing amzxyz/rime-lmdg structure

schema = {
    "schema": {
        "schema_id": "custom",
        "name": "Custom Schema with Collocation Tuning",
        "version": "2024-02-24",
    },
    "engine": {
        "segmentor": {
            "type": "simple",
        },
        # Collocation tuning section

        "collocation": {
            "max_length": 8,          # collocation_max_length

            "min_length": 2,          # collocation_min_length

            "penalty": -12,           # collocation_penalty

            "non_penalty": -18,       # non_collocation_penalty

            "weak_penalty": -30,      # weak_collocation_penalty

        },
    },
}

# Write to file

with open("custom.schema.yaml", "w", encoding="utf-8") as f:
    yaml.safe_dump(schema, f, allow_unicode=True)

print("✅ custom.schema.yaml generated with collocation tuning.")

Place the generated file in your RIME user configuration directory (e.g., ~/.config/ibus/rime/), then rebuild the schema using the deployment tool:

# Deploy the custom schema as referenced in wanxiang/wanxiang-tools.py workflow

rime_deployer --build custom.schema.yaml

Summary

• collocation_max_length (default 7) controls the upper bound of multi-character phrases; increase it to preserve long technical terms, decrease it for faster segmentation.
• collocation_min_length (default 2) sets the minimum collocation size; adjust to control whether bi-grams stay united or split.
• collocation_penalty (default -10) penalizes low-frequency collocations; make more negative to improve precision, less negative to retain rare terms.
• non_collocation_penalty (default -20) applies when expected collocations are missing; tune to balance fluency versus isolated character input.
• weak_collocation_penalty (default -35) filters statistically weak phrases; increase magnitude to reduce noise, decrease to keep domain jargon.
• Configuration changes take effect in custom.schema.yaml files deployed via rime_deployer, with defaults documented in wanxiang/README.md (lines 43-47).

Frequently Asked Questions

What happens if I set collocation_max_length too high in RIME?

Setting collocation_max_length above 10-12 characters allows extremely long strings to remain intact as single candidates. While this helps with fixed expressions like long organization names or idioms, it can degrade input responsiveness and increase memory usage as the engine evaluates longer sequences against wanxiang/pinyin_data/词组.dict.yaml entries. The processing overhead grows with candidate length, potentially slowing generation during active typing.

How does collocation_penalty differ from weak_collocation_penalty?

collocation_penalty applies to any collocation not matching high-frequency entries, while weak_collocation_penalty specifically targets collocations whose probabilities fall below an internal threshold. According to the amzxyz/rime-lmdg implementation, weak penalties stack atop standard penalties, creating a two-tier filtering system. Use collocation_penalty for general frequency filtering and weak_collocation_penalty to aggressively eliminate borderline rare phrases.

Where are the default RIME grammar configuration values defined?

The default values for all five parameters reside in wanxiang/README.md at lines 43-47 of the amzxyz/rime-lmdg repository. These defaults—collocation_max_length: 7, collocation_min_length: 2, collocation_penalty: -10, non_collocation_penalty: -20, and weak_collocation_penalty: -35—serve as the baseline for the Wanxiang language model distributed with the project.

Can I tune these parameters for specific domains like medical terminology?

Yes. For specialized domains, decrease weak_collocation_penalty toward -20 (from the default -35) to prevent the engine from filtering rare technical phrases. Simultaneously, increase collocation_max_length to 8 or 9 to accommodate longer compound medical terms, and relax collocation_penalty to -5 or -8 to allow lower-frequency collocations to surface in candidates. Test changes using representative domain text and iterate based on candidate ranking performance.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}