Memory Optimization Strategies for Processing a 32GB Chinese Corpus in rime-lmdg

The rime-lmdg Wanxiang project processes a 32GB Chinese corpus using streaming I/O, generator pipelines, chunked aggregation, multiprocessing with shared queues, and SQLite-backed storage to maintain a modest 2-3GB RAM footprint.

Processing massive text corpora for RIME input method dictionaries presents significant memory challenges. The amzxyz/rime-lmdg repository implements a suite of memory optimization strategies specifically designed to handle a 32GB Chinese corpus without exhausting system RAM. These techniques allow the pipeline to run efficiently on typical workstations with only 2-3GB of available memory while utilizing multi-core processing.

Streaming File I/O with Buffered Reads

The foundation of the memory-efficient pipeline is streaming file I/O rather than loading entire files into RAM. In 多线程分词.py, the corpus is read line-by-line using Python's built-in open() function with an 8KB buffer.

with open(corpus_path, encoding='utf-8', buffering=8192) as f:
    for line in f:
        q.put(line.rstrip('\n'))

This approach ensures that only a small window of the 32GB file resides in memory at any moment, regardless of the total file size.

Generator-Based Processing Pipelines

All intermediate processing steps are implemented as Python generators that yield one record at a time. In TXT清洗.py, operations like sentence segmentation and text normalization use iterator patterns rather than building intermediate lists.

This generator architecture prevents the accumulation of large in-memory data structures between processing stages. Each step consumes data lazily from the previous generator, maintaining a constant memory profile throughout the pipeline.

Chunked Frequency Aggregation

To avoid holding a complete frequency dictionary in RAM, json语料解析.py implements chunked aggregation with configurable flush thresholds. The script accumulates word frequencies in a collections.Counter and automatically serializes partial results to disk when reaching 10 million entries.

CHUNK_LIMIT = 10_000_000   # flush after 10M entries

freq = collections.Counter()
for token in token_generator():
    freq[token] += 1
    if sum(freq.values()) >= CHUNK_LIMIT:
        write_partial(freq, f'partial_{len(partials)}.pkl')
        freq.clear()

A secondary merge pass combines these partial frequency tables into the final dictionary, ensuring the active memory footprint never exceeds the chunk limit.

Multiprocessing with Shared Queues

The 多线程分词.py module employs process-based parallelism using multiprocessing.Queue to distribute work across CPU cores without duplicating the corpus in memory. The producer process streams lines into a bounded queue while worker processes consume chunks and write results to temporary files.

q = multiprocessing.Queue(maxsize=workers * 2)

def worker(idx):
    tmp_path = os.path.join(out_dir, f'temp_{idx}.txt')
    with open(tmp_path, 'w', encoding='utf-8') as out:
        while True:
            line = q.get()
            if line is None:
                break
            tokens = jieba.cut(line)
            out.write(' '.join(tokens) + '\n')

Each worker maintains its own output file, eliminating the need for the main process to collect results in memory.

Memory-Mapped SQLite for Intermediate Storage

When token lists grow beyond available RAM, 维基中文语料解析.py switches to SQLite-backed storage. The script creates a temporary SQLite database with PRAGMA journal_mode=OFF for maximum write performance.

This strategy provides O(1) lookups for deduplication and frequency counting while keeping the bulk of data on disk. The SQLite engine handles memory paging automatically, acting as a memory-mapped cache for the working dataset.

Sparse Data Structures and Compression

语法模型构建.py optimizes the final dictionary generation using sparse data structures and compression. Frequency tables utilize defaultdict(int) wrapped around collections.Counter, which only allocates storage for observed keys rather than the entire vocabulary space.

Before writing RIME .dict.yaml files, the script serializes data using pickle with gzip compression:

import pickle
import gzip

with gzip.open('freq_table.pkl.gz', 'wb') as f:
    pickle.dump(freq_dict, f, protocol=pickle.HIGHEST_PROTOCOL)

This compression reduces the memory required for the final dictionary assembly phase.

On-the-Fly Pinyin Annotation

Rather than caching pinyin conversions for the entire corpus, pypinyin/runner.py implements lazy pinyin generation using the pypinyin library. The system converts tokens to pinyin immediately before writing output, caching results only for the current processing chunk.

This on-the-fly approach avoids building a massive lookup table mapping every unique token in the 32GB corpus to its phonetic representation, which could easily consume gigabytes of RAM.

Configurable Chunk Sizes

All processing scripts expose a CHUNK_SIZE parameter (defaulting to 500,000 lines) that allows users to tune memory usage against I/O throughput. This configuration appears as a top-level constant across the codebase:

CHUNK_SIZE = 500_000  # lines per batch

Lowering this value reduces peak memory consumption at the cost of increased disk operations, enabling the pipeline to run on memory-constrained systems.

Summary

The rime-lmdg repository combines eight distinct memory optimization strategies to process a 32GB Chinese corpus efficiently:

• Streaming I/O reads files line-by-line with 8KB buffers in 多线程分词.py
• Generator pipelines avoid intermediate lists throughout the cleaning and tokenization stages
• Chunked aggregation flushes frequency counters to disk after 10M entries in json语料解析.py
• Multiprocessing queues distribute work across cores without data duplication
• SQLite storage provides disk-backed lookup tables in 维基中文语料解析.py
• Sparse counters minimize overhead for vocabulary tracking
• Gzip compression reduces serialization memory in 语法模型构建.py
• Lazy pinyin conversion eliminates full-corpus annotation caches

Together, these techniques maintain RAM usage between 2-3GB while processing the full 32GB dataset.

Frequently Asked Questions

How does the rime-lmdg pipeline avoid loading the entire 32GB corpus into memory?

The pipeline uses streaming file I/O with buffering=8192 in 多线程分词.py to read the corpus line-by-line, combined with generator-based processing in TXT清洗.py that yields one record at a time rather than building large intermediate lists.

What prevents the frequency dictionary from consuming all available RAM?

json语料解析.py implements chunked frequency aggregation with a default limit of 10 million entries. When the counter reaches this threshold, the script flushes partial results to disk as compressed pickle files and clears the in-memory counter, merging all partials in a final pass.

Why does the project use SQLite instead of pure in-memory structures?

维基中文语料解析.py switches to SQLite-backed storage with PRAGMA journal_mode=OFF when intermediate token lists grow large. SQLite provides O(1) lookups for deduplication while automatically paging data to disk, preventing memory exhaustion during the vocabulary building phase.

Can the memory usage be adjusted for systems with less RAM?

Yes. All major processing scripts expose a configurable CHUNK_SIZE parameter (default 500,000 lines) that controls how many lines are processed before flushing to disk. Reducing this value decreases peak memory consumption while increasing I/O operations, allowing the pipeline to run on systems with limited RAM.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}