# How to Benchmark Jieba Multi-Threading Configurations in Rime-LMDG

> Benchmark jieba multi-threading configurations effectively measure performance across worker counts. Optimize your Rime-LMDG setup for speed and efficiency.

- Repository: [amzxyz/rime-lmdg](https://github.com/amzxyz/rime-lmdg)
- Tags: performance
- Published: 2026-02-24

---

**Benchmark jieba multi-threading configurations by measuring wall-clock time, CPU utilization, and memory consumption across different worker counts, comparing jieba's built-in parallel mode against custom ThreadPoolExecutor implementations using a fixed corpus with multiple repetitions.**

The **rime-lmdg** repository implements two distinct strategies for accelerating Chinese word segmentation with the jieba library. Understanding how to benchmark jieba multi-threading configurations allows you to optimize throughput for both single large documents and distributed file processing workflows.

## Understanding Jieba Parallelism in Rime-LMDG

The codebase provides two architectural approaches to parallelism. **Internal jieba parallelism** (`jieba.enable_parallel`) splits input text into chunks and distributes them across Python threads sharing a global dictionary. This mode is implemented in `语法模型构建.py` at line 55, where `jieba.enable_parallel()` activates parallel segmentation for the `'all'` mode.

**Custom ThreadPoolExecutor parallelism** processes multiple files simultaneously while segmenting each file sequentially. Implemented in `多线程分词.py` (lines 6-8 define `MAX_WORKERS = 20`, lines 66-71 implement the pool), this approach wraps file-level operations including OpenCC conversion in `process_file_stream()` (lines 33-44).

## Benchmarking Methodology

Follow these steps to ensure statistically valid results:

1. **Prepare a fixed corpus**: Use a representative file like `分词后.txt` or the `autodl-tmp/语料清洗后` directory consistently across all tests.
2. **Isolate configurations**: Test one `SEGMENT_MODE` (`'accurate'`, `'all'`, or `'search'`) or ThreadPoolExecutor worker count per run.
3. **Warm-up execution**: Run one segmentation pass before timing to load dictionaries and avoid initialization overhead.
4. **Measure performance**: Wrap operations with `time.perf_counter()` for high-resolution timing.
5. **Monitor resources**: Optionally use `psutil` to capture CPU percentage and RSS memory during execution.
6. **Repeat trials**: Perform at least three runs per configuration and average results to mitigate OS scheduling noise.

## Benchmark 1: Jieba Internal Parallel Mode

Use this script to test `jieba.enable_parallel()` with varying worker counts:

```python

# benchmark_jieba_internal.py

import os, time, jieba
from tqdm import tqdm

# Load custom dictionaries as implemented in the pipeline

custom_dir = "autodl-tmp/自定义分词词典"
if os.path.isdir(custom_dir):
    for fn in os.listdir(custom_dir):
        jieba.load_userdict(os.path.join(custom_dir, fn))

# Configure internal parallelism

MAX_WORKERS = 8  # Experiment with 1, 2, 4, 8...

jieba.enable_parallel(MAX_WORKERS)

# Warm-up to load dictionaries into memory

_ = jieba.lcut("热身文本，用于加载字典", HMM=True)

input_path = "autodl-tmp/语料清洗后/sample.txt"
output_path = "tmp_seg.txt"

def segment_file():
    with open(input_path, "r", encoding="utf-8") as fin, \
         open(output_path, "w", encoding="utf-8") as fout:
        for line in tqdm(fin, desc="Segmenting"):
            words = jieba.lcut(line.strip(), HMM=True)
            fout.write(" ".join(words) + "\n")

# Benchmark with 3 repetitions

times = []
for run in range(3):
    start = time.perf_counter()
    segment_file()
    elapsed = time.perf_counter() - start
    times.append(elapsed)
    print(f"Run {run+1}: {elapsed:.2f}s")

print(f"Average: {sum(times)/len(times):.2f}s (workers={MAX_WORKERS})")

```

Vary `MAX_WORKERS` from 1 to your CPU core count. Performance gains plateau after exceeding physical cores because this mode uses shared memory without process overhead.

## Benchmark 2: Custom ThreadPoolExecutor

Test file-level parallelism using this implementation based on `多线程分词.py`:

```python

# benchmark_jieba_threadpool.py

import os, time, jieba
from concurrent.futures import ThreadPoolExecutor, as_completed
from tqdm import tqdm

MAX_WORKERS = 12  # Test 4, 8, 12, 20...

INPUT_FOLDER = "autodl-tmp/语料清洗后"
OUTPUT_FOLDER = "tmp_parallel"
os.makedirs(OUTPUT_FOLDER, exist_ok=True)

# Initialize dictionaries once per process

CUSTOM_DICT_DIR = "autodl-tmp/自定义分词词典"
if os.path.isdir(CUSTOM_DICT_DIR):
    for fn in os.listdir(CUSTOM_DICT_DIR):
        jieba.load_userdict(os.path.join(CUSTOM_DICT_DIR, fn))

def segment_file(in_path, out_path):
    with open(in_path, "r", encoding="utf-8") as fin, \
         open(out_path, "w", encoding="utf-8") as fout:
        for line in fin:
            words = jieba.lcut(line.strip(), HMM=True)
            fout.write(" ".join(words) + "\n")

def benchmark():
    files = [
        (os.path.join(INPUT_FOLDER, f),
         os.path.join(OUTPUT_FOLDER, f + "_seg.txt"))
        for f in os.listdir(INPUT_FOLDER)
        if os.path.isfile(os.path.join(INPUT_FOLDER, f))
    ]
    
    start = time.perf_counter()
    with ThreadPoolExecutor(max_workers=MAX_WORKERS) as executor:
        futures = {executor.submit(segment_file, src, dst): (src, dst) 
                  for src, dst in files}
        for future in tqdm(as_completed(futures), total=len(futures)):
            future.result()
    return time.perf_counter() - start

times = [benchmark() for _ in range(3)]
for i, t in enumerate(times, 1):
    print(f"Run {i}: {t:.2f}s")
print(f"Average: {sum(times)/len(times):.2f}s (workers={MAX_WORKERS})")

```

Note that `多线程分词.py` explicitly disables internal parallel mode (lines 13-14) to avoid conflicts when using the ThreadPoolExecutor approach.

## Architectural Trade-offs

Choose the appropriate strategy based on your workload characteristics:

- **Internal parallel mode**: Optimal for single large documents requiring minimal code changes. The implementation in `语法模型构建.py` shares dictionaries across threads, reducing memory overhead but limiting scalability to CPU core count.
- **ThreadPoolExecutor**: Better for many independent files with pre/post-processing requirements. As shown in `多线程分词.py`, this approach allows overlapping I/O and CPU operations, though each worker incurs dictionary loading overhead through `load_custom_dict()`.

## Summary

- **rime-lmdg** implements two distinct jieba parallelization strategies: internal chunk-based parallelism and custom file-level ThreadPoolExecutor.
- Benchmark both approaches using `time.perf_counter()` with at least three repetitions per configuration.
- Internal parallelism (`jieba.enable_parallel`) works best for large single files, configured in `语法模型构建.py`.
- ThreadPoolExecutor excels at processing multiple files simultaneously, implemented in `多线程分词.py` with configurable `MAX_WORKERS`.
- Always warm up the jieba dictionary before timing to eliminate initialization bias.
- Vary worker counts from 1 to CPU core count plus a small margin to identify optimal throughput.

## Frequently Asked Questions

### What is the difference between jieba's internal parallel mode and the ThreadPoolExecutor approach?

Jieba's internal parallel mode splits individual texts into chunks processed by shared threads, ideal for single large documents. The ThreadPoolExecutor approach processes entire files in parallel workers, better suited for batch processing multiple independent files with additional preprocessing like OpenCC conversion.

### How many worker threads should I configure for optimal jieba performance?

Start with `os.cpu_count()` for internal parallel mode, as gains diminish beyond physical cores. For ThreadPoolExecutor, test values between CPU count and CPU count plus four, since file I/O operations benefit from slight oversubscription without excessive context switching.

### Why does the rime-lmdg repository disable internal parallelism in the multi-threading script?

The `多线程分词.py` file disables `jieba.enable_parallel()` (lines 13-14) to prevent thread contention. Running nested parallelism—both internal jieba workers and external ThreadPoolExecutor workers—creates excessive thread overhead and degrades performance rather than improving it.

### Should I load custom dictionaries before benchmarking jieba multi-threading?

Yes. Always load custom dictionaries from `autodl-tmp/自定义分词词典` and execute a warm-up segmentation before timing. This ensures dictionary loading costs—significant in the ThreadPoolExecutor approach where each worker initializes independently—do not skew your benchmark results.