# How to Troubleshoot Common Errors During KenLM ARPA File Generation

> Troubleshoot KenLM ARPA file generation errors in rime-lmdg. Learn to diagnose missing binaries, memory, and permission issues by inspecting exit codes and console output.

- Repository: [amzxyz/rime-lmdg](https://github.com/amzxyz/rime-lmdg)
- Tags: how-to-guide
- Published: 2026-02-24

---

**Most KenLM ARPA generation failures in rime-lmdg stem from missing binaries, insufficient memory, or permission issues in the temporary cache directory, all of which can be diagnosed by inspecting the exit codes and console output from the `generate_arpa` function.**

The rime-lmdg repository automates language model construction for RIME input methods using KenLM's `lmplz` tool. When you troubleshoot common errors encountered during KenLM's ARPA file generation, you are typically debugging the external binary invocation managed by the **`generate_arpa`** function in `wanxiang/语法模型构建.py`.

## Understanding the ARPA Generation Pipeline

The workflow in `wanxiang/语法模型构建.py` orchestrates the following steps:

1. **Create a temporary cache directory** (`~/ARPAtmp`) – lines [20‑24](https://github.com/amzxyz/rime-lmdg/blob/wanxiang/wanxiang/语法模型构建.py#L20-L24).
2. **Build the `lmplz` command** with the desired n‑gram order, input text (`SEGMENTED_FILE`), output ARPA file (`ARPA_FILE`), cache dir (`‑T`) and memory limits – lines [28‑35](https://github.com/amzxyz/rime-lmdg/blob/wanxiang/wanxiang/语法模型构建.py#L28-L35).
3. **Execute the command** via `os.system` – line [38](https://github.com/amzxyz/rime-lmdg/blob/wanxiang/wanxiang/语法模型构建.py#L38).
4. **Raise an exception** if the exit code ≠ 0 – lines [41‑42](https://github.com/amzxyz/rime-lmdg/blob/wanxiang/wanxiang/语法模型构建.py#L41-L42).
5. **Clean up the temporary directory** in a `finally` block – lines [45‑48](https://github.com/amzxyz/rime-lmdg/blob/wanxiang/wanxiang/语法模型构建.py#L45-L48).

Because the process hinges on an external binary, most errors stem from environment issues rather than Python logic.

## Common Error Symptoms and Solutions

### Exit Code 127: lmplz Binary Not Found

A **`RuntimeError: 生成 ARPA 文件失败，退出代码: 127`** indicates that the `lmplz` binary is not found or not executable.

- **Diagnose**: Run `which lmplz` in a terminal. If the command returns nothing, the binary is missing from your `$PATH`.
- **Fix**: Install KenLM (`sudo apt install kenlm` or compile from source) and ensure the binary directory is added to your system `PATH`.

### Exit Codes 1‑3: Resource Constraints or Malformed Input

Exit codes between **1 and 3** typically signal insufficient RAM, disk space exhaustion, or malformed input text.

- **Diagnose**: Check the console output printed by `print(f"执行命令：{cmd}")` on line 37. Copy the printed command and run it manually in a shell to view detailed KenLM error messages.
- **Fix**:
  - Reduce the memory limit by adjusting the `‑S` flag (e.g., `‑S 2G` or lower) in the command construction on line 34.
  - Split the segmented corpus into smaller chunks or filter extremely long lines in `preprocess_corpus`.
  - Ensure the temporary directory (`~/ARPAtmp`) has adequate free space.

### Permission Denied on Temporary Directory

Failures when creating or cleaning `~/ARPAtmp` indicate the user lacks write permissions on the home directory or the cache folder exists with restrictive permissions.

- **Diagnose**: Verify permissions with `ls -ld ~/ARPAtmp`.
- **Fix**:
  - Delete the folder manually and rerun the script.
  - Change ownership: `chmod u+rwx ~/ARPAtmp`.
  - Override the path by modifying the `tmp_dir` parameter to a writable location (e.g., `/tmp/ARPAtmp`).

### Empty or Missing ARPA Output

If the ARPA file is empty or never appears, `lmplz` likely terminated early because the input file (`SEGMENTED_FILE`, typically `分词后.txt`) is empty or contains non‑UTF‑8 bytes.

- **Diagnose**: Open `SEGMENTED_FILE` and inspect the first few lines for content and encoding.
- **Fix**:
  - Re‑run the preprocessing and segmentation steps (see the `segment_corpus` function).
  - Ensure `SEGMENTED_FILE` is encoded in UTF‑8 (the script opens files with `encoding='utf-8'`).

### Cleanup Failures After Generation

When **`clean_temp_directory`** fails (exception printed), files in the temporary directory may still be opened by another process, or the OS is blocking recursive deletion.

- **Diagnose**: Inspect the printed error message from the cleanup function.
- **Fix**:
  - Ensure no other process is using files under `~/ARPAtmp`.
  - Manually delete the folder after the script finishes if automatic cleanup fails.

## Debugging Techniques

- **Print the exact command** – The script already outputs the command on line 37 via `print(f"执行命令：{cmd}")`.
- **Run the command manually** – Copy‑paste the printed string into a shell; KenLM will output granular diagnostics about vocabulary size and memory usage.
- **Capture log files** – Modify the command string to redirect output: `f"{cmd} > lmplz.log 2>&1"` and inspect `lmplz.log` after failure.
- **Validate input size** – Before calling `generate_arpa`, add `print(f"Segmented file size: {os.path.getsize(segmented_file)} bytes")` to confirm the input is non‑empty.

## Practical Code Solutions

### Verify lmplz Availability Before Execution

Add this check at the start of your pipeline to fail fast with a clear message:

```python
import shutil, sys

if not shutil.which("lmplz"):
    sys.exit("Error: KenLM's 'lmplz' binary is not in PATH. Install KenLM first.")
else:
    print("KenLM is available.")

```

### Run generate_arpa with a Custom Temporary Directory

Use this modified function to override the default cache location and reduce memory footprint:

```python
import os

def generate_arpa(segmented_file, arpa_file, ngram_order=3, tmp_dir="/tmp/kenlm_tmp"):
    if not os.path.exists(tmp_dir):
        os.makedirs(tmp_dir)
    cmd = (
        f"lmplz -o {ngram_order} "
        f"--text {segmented_file} "
        f"--arpa {arpa_file} "
        f"-T {tmp_dir} "
        f"-S 2G "                     # smaller memory footprint

        f"--prune 0 75 300"
    )
    print(f"Running: {cmd}")
    exit_code = os.system(cmd)
    if exit_code != 0:
        raise RuntimeError(f"ARPA generation failed, exit code: {exit_code}")
    # clean up as before ...

```

### Capture KenLM Output for Deep Inspection

Replace `os.system` with `subprocess` to capture stdout and stderr programmatically:

```python
import subprocess
import os

def generate_arpa_with_logging(segmented_file, arpa_file, ngram_order=3):
    tmp_dir = os.path.expanduser("~/ARPAtmp")
    os.makedirs(tmp_dir, exist_ok=True)
    cmd = [
        "lmplz",
        f"-o{ngram_order}",
        "--text", segmented_file,
        "--arpa", arpa_file,
        "-T", tmp_dir,
        "-S", "2G",
        "--prune", "0", "75", "300"
    ]

    result = subprocess.run(cmd, capture_output=True, text=True)
    print("KenLM stdout:", result.stdout)
    print("KenLM stderr:", result.stderr)

    if result.returncode != 0:
        raise RuntimeError(f"ARPA generation failed, code {result.returncode}")
    # clean-up ...

```

## Summary

- **Exit code 127** means `lmplz` is not installed or not in your system `PATH`.
- **Exit codes 1‑3** indicate resource exhaustion (RAM/disk) or malformed UTF‑8 input in the segmented corpus.
- **Permission errors** on `~/ARPAtmp` require fixing directory ownership or switching to a writable temporary path like `/tmp/kenlm_tmp`.
- Always validate that `分词后.txt` (the `SEGMENTED_FILE`) is non‑empty and properly encoded before invoking `generate_arpa`.
- Use `subprocess.run` with `capture_output=True` to capture detailed KenLM diagnostics instead of relying solely on exit codes.

## Frequently Asked Questions

### Why does generate_arpa fail with exit code 127?

Exit code 127 indicates the `lmplz` binary cannot be found in your system `PATH`. Verify installation by running `which lmplz` in your terminal. If the command is not found, install KenLM via your package manager or compile it from source, then ensure the binary location is exported in your shell configuration.

### How do I reduce memory usage when processing large corpora?

Modify the `‑S` flag in the command construction on line 34 of `wanxiang/语法模型构建.py`. Change `‑S 80%` (or higher values) to `‑S 2G` or `‑S 1G` to limit KenLM to a specific RAM allocation. For extremely large datasets, split the input corpus into smaller chunks using the `preprocess_corpus` logic before calling `generate_arpa`.

### What file encoding does KenLM expect for the input text?

KenLM and the rime-lmdg pipeline expect **UTF‑8** encoded text. The script explicitly opens files with `encoding='utf-8'`. If your segmented file (`分词后.txt`) contains non‑UTF-8 bytes, `lmplz` may crash or produce empty ARPA files. Re‑run the `segment_corpus` step to ensure clean, UTF‑8 output.

### Where is the temporary cache directory created?

By default, the script creates `~/ARPAtmp` in the user's home directory (lines 20‑24). You can override this by passing a different path to the `tmp_dir` parameter in `generate_arpa`, such as `/tmp/ARPAtmp`, which is useful if your home directory has strict quota or permission restrictions.