# Exact KenLM ARPA File Format and RIME Grammar Parsing: A Technical Deep Dive

> Learn the exact KenLM ARPA file format and how RIME parses it for grammar rules. Understand log-probabilities, back-off weights, and frequency table compilation for RIME's build_grammar binary.

- Repository: [amzxyz/rime-lmdg](https://github.com/amzxyz/rime-lmdg)
- Tags: deep-dive
- Published: 2026-02-24

---

**The KenLM ARPA format uses tab-separated log-probabilities, n-grams, and optional back-off weights organized in `\data\` and `\<n>-grams:` sections, which the amzxyz/rime-lmdg repository parses by extracting header counts, converting log-probabilities to integer frequencies via probability math, and compiling them into phrase-frequency tables for RIME's `build_grammar` binary.**

The amzxyz/rime-lmdg repository bridges KenLM language modeling with the RIME input method engine by converting standard ARPA files into specialized grammar models. Understanding the exact KenLM ARPA file format and RIME's parsing pipeline is essential for developers building custom Chinese input method dictionaries with statistical grammar support.

## KenLM ARPA File Format Structure

When the repository's `generate_arpa` function invokes KenLM's `lmplz` tool【语法模型构建.py†L16-L45】, it produces a standard ARPA-format language model file with a specific structure.

### Header and Metadata Section

Every ARPA file begins with a `\data\` block that declares the count of n-grams for each order:

```text
\data\
ngram 1=123456
ngram 2=234567
ngram 3=345678

```

These counts represent the total number of unigrams, bigrams, and trigrams respectively, stored as key-value pairs where the format is `ngram <order>=<count>`.

### N-gram Data Blocks

Following the header, the file contains separate sections for each n-gram order prefixed with backslashes:

```text
\1-grams:
-9.2103 <s> -0.1234
-8.9234 你好   -0.5678
-7.2345 世界

\2-grams:
-3.4567 <s> 你好  -0.0321
-2.9876 你好 世界  -0.0145

\3-grams:
-1.2345 <s> 你好 世界  -0.0012

\end\

```

Each line within these blocks follows a strict tab-separated format with three fields:

1. **Log-probability** (floating-point): The natural log probability of the n-gram
2. **The n-gram** (space-separated tokens): The actual word sequence
3. **Back-off weight** (floating-point, optional): Present for all orders except the highest (typically trigrams), used for smoothing

The fields are **tab-separated** in actual KenLM output, though displayed with spaces for readability in documentation.

## How RIME Parses ARPA Files in `语法模型构建.py`

The repository implements a three-stage parsing pipeline in `语法模型构建.py` to transform KenLM's probabilistic output into RIME's frequency-based grammar format.

### Extracting N-gram Counts from the Header

The `extract_ngram_counts` function scans the ARPA header until reaching the first `\1-grams:` block【语法模型构建.py†L60-L76】:

```python
if line.startswith("ngram"):
    order = int(parts[0].split()[1])
    count = int(parts[1])
    ngrams_counts[order] = count
elif line.startswith("\\1-grams:"):
    break

```

This populates a dictionary like `{1: 123456, 2: 234567, 3: 345678}` that serves as the denominator for converting relative probabilities into absolute frequencies later in the pipeline.

### Parsing Individual N-grams with Regex

The `extract_ngrams` generator function walks through the ARPA body using a precise regular expression to extract structured data【语法模型构建.py†L81-L99】:

```python
ngram_line_pattern = re.compile(r"^(-?\d+\.\d+)\t(.+?)(?:\t-?\d+\.\d+)?$")

if line.startswith("\\") and "-grams:" in line:
    current_order = int(line.split('-')[0][1:])

ngram_line_match = ngram_line_pattern.match(line)
if ngram_line_match:
    logprob, ngram = ngram_line_match.groups()
    prob = math.exp(float(logprob))
    yield current_order, ngram.strip(), prob

```

The regex `^(-?\d+\.\d+)\t(.+?)(?:\t-?\d+\.\d+)?$` captures:
- Group 1: The log-probability (mandatory)
- Group 2: The n-gram phrase (mandatory)
- Optional non-capturing group: The back-off weight (ignored for RIME's purposes)

The code tracks `current_order` by detecting section headers like `\1-grams:`, `\2-grams:`, etc., and converts log-probabilities to linear probabilities using `math.exp` for downstream frequency calculations.

### Converting Log-Probabilities to Integer Frequencies

The `write_frequencies_to_file` function receives the counts from the header parsing stage and the probability generator to produce RIME-compatible frequency files【语法模型构建.py†L100-L124】:

```python
total_count = ngrams_counts.get(order, 1)
freq = round(prob * total_count)
file.write(f"{ngram}\t{freq}\n")

```

This conversion assumes that `prob * total_count` approximates the original corpus frequency count. The output follows the **RIME "phrase ↔ frequency"** convention: each line contains the n-gram phrase, a tab character, and the calculated integer frequency.

### Compiling the Final `.gram` File for RIME

After generating per-order frequency files ([`ngram_1_.txt`](https://github.com/amzxyz/rime-lmdg/blob/main/ngram_1_.txt), [`ngram_2_.txt`](https://github.com/amzxyz/rime-lmdg/blob/main/ngram_2_.txt), [`ngram_3_.txt`](https://github.com/amzxyz/rime-lmdg/blob/main/ngram_3_.txt)), the `generate_gram_file` function merges them and invokes the external `build_grammar` binary【语法模型构建.py†L94-L103】:

```bash
./build_grammar zh-hans < merge1_2_3.txt

```

The `build_grammar` tool expects exactly the **"phrase<TAB>frequency"** format produced by the previous stage and outputs a binary `.gram` file that RIME loads as a **grammar model** for input prediction and ranking.

## Complete ARPA Parsing Implementation

Below is a minimal, runnable implementation mirroring the repository's parsing logic:

```python
import re
import math
from pathlib import Path

def parse_arpa(arpa_path):
    """Parse KenLM ARPA file yielding (order, phrase, probability, total_count)."""
    ngram_counts = {}
    
    # Stage 1: Extract header counts

    with open(arpa_path, encoding='utf-8') as f:
        for line in f:
            line = line.strip()
            if line.startswith('ngram'):
                parts = line.split('=')
                order = int(parts[0].split()[1])
                count = int(parts[1])
                ngram_counts[order] = count
            elif line.startswith('\\1-grams:'):
                break
    
    # Stage 2: Parse body with regex

    ngram_pat = re.compile(r'^(-?\d+\.\d+)\t(.+?)(?:\t-?\d+\.\d+)?$')
    current_order = 0
    
    with open(arpa_path, encoding='utf-8') as f:
        for line in f:
            line = line.strip()
            if line.startswith('\\') and '-grams:' in line:
                current_order = int(line.split('-')[0][1:])
                continue
            
            m = ngram_pat.match(line)
            if m:
                logp, phrase = m.groups()
                prob = math.exp(float(logp))
                total = ngram_counts.get(current_order, 1)
                yield current_order, phrase, prob, total

def write_rime_files(arpa_path: str, template: str = "ngram_{}_.txt"):
    """Convert ARPA to RIME phrase-frequency files."""
    for order, phrase, prob, total in parse_arpa(arpa_path):
        freq = round(prob * total)
        out_path = Path(template.format(order))
        out_path.parent.mkdir(parents=True, exist_ok=True)
        
        with out_path.open('a', encoding='utf-8') as f:
            f.write(f'{phrase}\t{freq}\n')

```

Running `write_rime_files('model.arpa')` creates the necessary input files for the `build_grammar` compilation step.

## Summary

- **KenLM Output**: The `lmplz` command generates standard ARPA files with `\data\` headers and `\<n>-grams:` sections containing tab-separated log-probabilities, phrases, and optional back-off weights.
- **Header Parsing**: The `extract_ngram_counts` function in `语法模型构建.py` captures total n-gram counts to establish frequency baselines.
- **Body Extraction**: `extract_ngrams` uses regex pattern `r"^(-?\d+\.\d+)\t(.+?)(?:\t-?\d+\.\d+)?$"` to parse log-probabilities and phrases while ignoring back-off weights.
- **Frequency Conversion**: The pipeline converts log-probabilities to linear probabilities via `math.exp`, then multiplies by total counts to generate integer frequencies.
- **RIME Compilation**: The `build_grammar` binary consumes phrase-frequency tables to produce binary `.gram` files that RIME loads as grammar models for Chinese input prediction.

## Frequently Asked Questions

### What is the exact field separator in KenLM ARPA files?

KenLM uses **tab characters (`\t`)** to separate fields within each n-gram line. While the log-probability, n-gram tokens (which may contain spaces), and back-off weight appear as distinct columns, the tokens themselves are space-separated words. The repository's regex pattern `^(-?\d+\.\d+)\t(.+?)(?:\t-?\d+\.\d+)?$` explicitly looks for tabs between the probability and phrase fields.

### Why does RIME need integer frequencies instead of log-probabilities?

RIME's `build_grammar` binary was designed to consume corpus frequency counts rather than normalized probabilities. By multiplying the linear probability (derived via `math.exp` from KenLM's log-probability) by the total n-gram count from the ARPA header, the repository reconstructs approximate integer frequencies that match the statistical expectations of the RIME grammar engine.

### Can I use ARPA files with higher than 3-gram orders in this pipeline?

Yes, the parsing logic in `extract_ngrams`【语法模型构建.py†L81-L99】 dynamically detects any `\<n>-grams:` section header and processes it accordingly. However, you must ensure that the `build_grammar` binary and your target RIME configuration support the specific n-gram order, as higher orders significantly increase memory requirements and binary file size.

### What happens to the back-off weights in the ARPA file?

The repository's regex pattern `(?:\t-?\d+\.\d+)?$` captures but discards the optional back-off weight field. For RIME grammar compilation, these weights are unnecessary because the `build_grammar` tool constructs its own data structures from the absolute frequency counts. The parsing logic focuses exclusively on the log-probability and phrase content.