Exact KenLM ARPA File Format and RIME Grammar Parsing: A Technical Deep Dive

The KenLM ARPA format uses tab-separated log-probabilities, n-grams, and optional back-off weights organized in \data\ and \<n>-grams: sections, which the amzxyz/rime-lmdg repository parses by extracting header counts, converting log-probabilities to integer frequencies via probability math, and compiling them into phrase-frequency tables for RIME's build_grammar binary.

The amzxyz/rime-lmdg repository bridges KenLM language modeling with the RIME input method engine by converting standard ARPA files into specialized grammar models. Understanding the exact KenLM ARPA file format and RIME's parsing pipeline is essential for developers building custom Chinese input method dictionaries with statistical grammar support.

KenLM ARPA File Format Structure

When the repository's generate_arpa function invokes KenLM's lmplz tool【语法模型构建.py†L16-L45】, it produces a standard ARPA-format language model file with a specific structure.

Header and Metadata Section

Every ARPA file begins with a \data\ block that declares the count of n-grams for each order:

\data\
ngram 1=123456
ngram 2=234567
ngram 3=345678

These counts represent the total number of unigrams, bigrams, and trigrams respectively, stored as key-value pairs where the format is ngram <order>=<count>.

N-gram Data Blocks

Following the header, the file contains separate sections for each n-gram order prefixed with backslashes:

\1-grams:
-9.2103 <s> -0.1234
-8.9234 你好   -0.5678
-7.2345 世界

\2-grams:
-3.4567 <s> 你好  -0.0321
-2.9876 你好 世界  -0.0145

\3-grams:
-1.2345 <s> 你好 世界  -0.0012

\end\

Each line within these blocks follows a strict tab-separated format with three fields:

1. Log-probability (floating-point): The natural log probability of the n-gram
2. The n-gram (space-separated tokens): The actual word sequence
3. Back-off weight (floating-point, optional): Present for all orders except the highest (typically trigrams), used for smoothing

The fields are tab-separated in actual KenLM output, though displayed with spaces for readability in documentation.

How RIME Parses ARPA Files in 语法模型构建.py

The repository implements a three-stage parsing pipeline in 语法模型构建.py to transform KenLM's probabilistic output into RIME's frequency-based grammar format.

Extracting N-gram Counts from the Header

The extract_ngram_counts function scans the ARPA header until reaching the first \1-grams: block【语法模型构建.py†L60-L76】:

if line.startswith("ngram"):
    order = int(parts[0].split()[1])
    count = int(parts[1])
    ngrams_counts[order] = count
elif line.startswith("\\1-grams:"):
    break

This populates a dictionary like {1: 123456, 2: 234567, 3: 345678} that serves as the denominator for converting relative probabilities into absolute frequencies later in the pipeline.

Parsing Individual N-grams with Regex

The extract_ngrams generator function walks through the ARPA body using a precise regular expression to extract structured data【语法模型构建.py†L81-L99】:

ngram_line_pattern = re.compile(r"^(-?\d+\.\d+)\t(.+?)(?:\t-?\d+\.\d+)?$")

if line.startswith("\\") and "-grams:" in line:
    current_order = int(line.split('-')[0][1:])

ngram_line_match = ngram_line_pattern.match(line)
if ngram_line_match:
    logprob, ngram = ngram_line_match.groups()
    prob = math.exp(float(logprob))
    yield current_order, ngram.strip(), prob

The regex ^(-?\d+\.\d+)\t(.+?)(?:\t-?\d+\.\d+)?$ captures:

• Group 1: The log-probability (mandatory)
• Group 2: The n-gram phrase (mandatory)
• Optional non-capturing group: The back-off weight (ignored for RIME's purposes)

The code tracks current_order by detecting section headers like \1-grams:, \2-grams:, etc., and converts log-probabilities to linear probabilities using math.exp for downstream frequency calculations.

Converting Log-Probabilities to Integer Frequencies

The write_frequencies_to_file function receives the counts from the header parsing stage and the probability generator to produce RIME-compatible frequency files【语法模型构建.py†L100-L124】:

total_count = ngrams_counts.get(order, 1)
freq = round(prob * total_count)
file.write(f"{ngram}\t{freq}\n")

This conversion assumes that prob * total_count approximates the original corpus frequency count. The output follows the RIME "phrase ↔ frequency" convention: each line contains the n-gram phrase, a tab character, and the calculated integer frequency.

Compiling the Final .gram File for RIME

After generating per-order frequency files (ngram_1_.txt, ngram_2_.txt, ngram_3_.txt), the generate_gram_file function merges them and invokes the external build_grammar binary【语法模型构建.py†L94-L103】:

./build_grammar zh-hans < merge1_2_3.txt

The build_grammar tool expects exactly the "phrasefrequency" format produced by the previous stage and outputs a binary .gram file that RIME loads as a grammar model for input prediction and ranking.

Complete ARPA Parsing Implementation

Below is a minimal, runnable implementation mirroring the repository's parsing logic:

import re
import math
from pathlib import Path

def parse_arpa(arpa_path):
    """Parse KenLM ARPA file yielding (order, phrase, probability, total_count)."""
    ngram_counts = {}
    
    # Stage 1: Extract header counts

    with open(arpa_path, encoding='utf-8') as f:
        for line in f:
            line = line.strip()
            if line.startswith('ngram'):
                parts = line.split('=')
                order = int(parts[0].split()[1])
                count = int(parts[1])
                ngram_counts[order] = count
            elif line.startswith('\\1-grams:'):
                break
    
    # Stage 2: Parse body with regex

    ngram_pat = re.compile(r'^(-?\d+\.\d+)\t(.+?)(?:\t-?\d+\.\d+)?$')
    current_order = 0
    
    with open(arpa_path, encoding='utf-8') as f:
        for line in f:
            line = line.strip()
            if line.startswith('\\') and '-grams:' in line:
                current_order = int(line.split('-')[0][1:])
                continue
            
            m = ngram_pat.match(line)
            if m:
                logp, phrase = m.groups()
                prob = math.exp(float(logp))
                total = ngram_counts.get(current_order, 1)
                yield current_order, phrase, prob, total

def write_rime_files(arpa_path: str, template: str = "ngram_{}_.txt"):
    """Convert ARPA to RIME phrase-frequency files."""
    for order, phrase, prob, total in parse_arpa(arpa_path):
        freq = round(prob * total)
        out_path = Path(template.format(order))
        out_path.parent.mkdir(parents=True, exist_ok=True)
        
        with out_path.open('a', encoding='utf-8') as f:
            f.write(f'{phrase}\t{freq}\n')

Running write_rime_files('model.arpa') creates the necessary input files for the build_grammar compilation step.

Summary

• KenLM Output: The lmplz command generates standard ARPA files with \data\ headers and \<n>-grams: sections containing tab-separated log-probabilities, phrases, and optional back-off weights.
• Header Parsing: The extract_ngram_counts function in 语法模型构建.py captures total n-gram counts to establish frequency baselines.
• Body Extraction: extract_ngrams uses regex pattern r"^(-?\d+\.\d+)\t(.+?)(?:\t-?\d+\.\d+)?$" to parse log-probabilities and phrases while ignoring back-off weights.
• Frequency Conversion: The pipeline converts log-probabilities to linear probabilities via math.exp, then multiplies by total counts to generate integer frequencies.
• RIME Compilation: The build_grammar binary consumes phrase-frequency tables to produce binary .gram files that RIME loads as grammar models for Chinese input prediction.

Frequently Asked Questions

What is the exact field separator in KenLM ARPA files?

KenLM uses tab characters (\t) to separate fields within each n-gram line. While the log-probability, n-gram tokens (which may contain spaces), and back-off weight appear as distinct columns, the tokens themselves are space-separated words. The repository's regex pattern ^(-?\d+\.\d+)\t(.+?)(?:\t-?\d+\.\d+)?$ explicitly looks for tabs between the probability and phrase fields.

Why does RIME need integer frequencies instead of log-probabilities?

RIME's build_grammar binary was designed to consume corpus frequency counts rather than normalized probabilities. By multiplying the linear probability (derived via math.exp from KenLM's log-probability) by the total n-gram count from the ARPA header, the repository reconstructs approximate integer frequencies that match the statistical expectations of the RIME grammar engine.

Can I use ARPA files with higher than 3-gram orders in this pipeline?

Yes, the parsing logic in extract_ngrams【语法模型构建.py†L81-L99】 dynamically detects any \<n>-grams: section header and processes it accordingly. However, you must ensure that the build_grammar binary and your target RIME configuration support the specific n-gram order, as higher orders significantly increase memory requirements and binary file size.

What happens to the back-off weights in the ARPA file?

The repository's regex pattern (?:\t-?\d+\.\d+)?$ captures but discards the optional back-off weight field. For RIME grammar compilation, these weights are unnecessary because the build_grammar tool constructs its own data structures from the absolute frequency counts. The parsing logic focuses exclusively on the log-probability and phrase content.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}