How 对数归一化 (Logarithmic Normalization) Shapes Word Frequencies in Rime-LMDG

对数归一化 compresses the dynamic range of n-gram probabilities by converting log-probabilities from the ARPA model into scaled integer frequencies, reducing storage size while preserving statistical relationships for the Rime input method.

The amzxyz/rime-lmdg repository implements a sophisticated grammar model construction pipeline that transforms raw text into structured n-gram frequencies. At the heart of this pipeline, implemented in 语法模型构建.py, lies the 对数归一化 process that bridges the probabilistic output of KenLM with the integer-based frequency tables required by the grammar builder.

The 对数归一化 Pipeline in 语法模型构建.py

Extracting Log-Probabilities from ARPA

The process begins when the script reads the ARPA-format language model generated by KenLM. Unlike raw count-based models, KenLM stores log-probabilities (base-10 or natural log) for every n-gram.

In the extract_ngrams function (lines 92-96), the code parses each n-gram line and exponentiates the log-probability to recover the raw probability:

logprob, ngram = ngram_line_match.groups()
prob = math.exp(float(logprob))      # ← logarithmic normalization step

Scaling to Integer Frequencies

After recovering the probability, the write_frequencies_to_file function (lines 15-22) applies the final normalization by scaling against the total count of n-grams for that specific order:

total_count = ngrams_counts.get(order, 1)
freq = round(prob * total_count)    # ← integer frequency after normalization

file.write(f"{ngram}\t{freq}\n")

This multiplication by total_count ensures that the resulting frequencies preserve the relative statistical weight of each n-gram within its order class.

How 对数归一化 Affects Word Frequencies

The logarithmic normalization process fundamentally reshapes the frequency distribution in four key ways:

• Dynamic Range Compression: Exponentiating log-probabilities smooths extreme values. While high-frequency n-grams retain significant integer values, the gap between common and rare n-grams shrinks dramatically compared to raw count models.

• Storage Efficiency: After scaling and rounding, low-probability n-grams often resolve to 0 or small integers, significantly reducing the file size of the final grammar tables. As noted in the repository documentation, this produces "词频经过对数归一化处理，缩短词频易于维护且文件储存更少的字节" (word frequencies undergo logarithmic normalization, shortening frequencies for easier maintenance and reduced file storage).

• Statistical Consistency: By using order-specific total counts (ngram 1=…, ngram 2=… from the ARPA header), the normalization preserves the relative probability distribution within each n-gram order, ensuring the downstream build_grammar binary receives statistically meaningful input.

• Deterministic Output: The mathematical operations (exponentiation, multiplication, rounding) produce identical integer frequencies for identical ARPA inputs, ensuring reproducible grammar generation across builds.

Implementation Details and File Structure

The normalization logic resides entirely within wanxiang/语法模型构建.py in the wanxiang branch. The script processes per-order files (ngram_1_.txt, ngram_2_.txt, etc.), converting lines like:

的 我们	24567

Here, 24567 represents the normalized frequency derived from the original log-probability, scaled by the total unigram count and rounded to an integer.

Summary

• 对数归一化 in Rime-LMDG converts KenLM log-probabilities into integer frequencies through exponentiation and order-specific scaling.
• The implementation in 语法模型构建.py uses math.exp() to reverse the logarithmic storage format, then multiplies by total n-gram counts to maintain statistical proportions.
• This approach compresses the dynamic range of frequencies, reduces storage requirements, and ensures deterministic output for reproducible grammar builds.
• The final integer frequencies power the build_grammar binary to create the .gram files used by the Rime input method.

Frequently Asked Questions

What is the purpose of 对数归一化 in the Rime-LMDG grammar model?

对数归一化 bridges the gap between KenLM's probabilistic output and Rime's integer-based frequency requirements. It converts log-probabilities into compact integer frequencies that maintain statistical relationships while minimizing file size, making the grammar tables efficient for input method deployment.

How does logarithmic normalization differ from simple frequency counting?

Simple frequency counting uses raw occurrence counts, which can vary wildly across several orders of magnitude. Logarithmic normalization instead uses the probabilistic estimates from the ARPA model, exponentiates them to recover probabilities, and scales them uniformly. This compresses the range between high and low frequency terms, preventing rare n-grams from having negligibly small weights while keeping file sizes manageable.

Where in the codebase is the 对数归一化 logic implemented?

The normalization logic is implemented in the extract_ngrams and write_frequencies_to_file functions within wanxiang/语法模型构建.py (lines 15-22 and 92-96). The code reads the ARPA file generated by KenLM, processes each n-gram's log-probability, and writes the normalized integer frequencies to per-order text files.

Why use math.exp() instead of directly using the log-probabilities?

The ARPA format stores log-probabilities (negative values representing log10 or natural log of probabilities) for numerical stability during language model training. The build_grammar binary and Rime's engine expect positive integer frequencies. Using math.exp() converts these log-probabilities back to linear probability space before scaling and rounding, ensuring the final frequencies reflect the actual likelihood ratios intended by the language model.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}