How the RAM‑First Pipeline Handles Memory Management and LZ4 Compression in Codebase‑Memory‑MCP

The RAM‑first pipeline achieves ultra‑fast indexing by loading all source files into memory, compressing them with LZ4 high‑compression mode, and storing the data in an in‑memory SQLite database that gets flushed to disk in a single operation.

The RAM‑first pipeline is the core indexing strategy in the codebase‑memory‑mcp repository. It eliminates disk I/O bottlenecks during ingestion by keeping the entire workload in RAM while using LZ4 compression to maintain manageable memory usage. This approach produces a compact, ready‑to‑query SQLite database through a single, atomic write operation.

How LZ4 High‑Compression Mode Reduces Memory Footprint

The pipeline compresses every source file using the LZ4 HC (high‑compression) mode before storing it. This reduces the memory footprint of raw source text while preserving the decompression speed that LZ4 is known for.

The compression logic resides in internal/cbm/lz4_store.c, which wraps the vendored LZ4 library. Instead of calling LZ4 APIs directly, the pipeline uses a simplified interface:

/* From internal/cbm/lz4_store.c */
int lz4_compress(const char *src, char *dst, int srcLen, int dstCap) {
    return LZ4_compress_HC(src, dst, srcLen, dstCap, LZ4_HC_LEVEL);
}

The lz4_compress function uses LZ4_compress_HC with the default high‑compression level (LZ4_HC_LEVEL). This provides better compression ratios than the fast mode while maintaining throughput suitable for in‑memory operations. The compressed output is bound by LZ4_compressBound(), ensuring sufficient destination buffer allocation.

In‑Memory SQLite Storage for Sub‑Millisecond Queries

After compression, the data flows into an in‑memory SQLite database that never touches the filesystem during the ingestion phase. This configuration allows SQLite’s page cache and the built‑in FTS5 tokenizer to operate entirely in RAM, enabling BM25 full‑text search queries with sub‑millisecond latency.

The pipeline leverages SQLite’s ability to run as a :memory: database, avoiding the overhead of fsync operations and disk page writes during the indexing process. The FTS5 extension tokenizes and indexes the compressed content without intermediate disk spills, creating a searchable inverted index that remains resident in RAM until the final dump.

Single‑Dump Architecture and Memory Lifecycle

The RAM‑first pipeline follows a single‑dump strategy that defers all disk writes until indexing completes. This architecture minimizes I/O overhead and ensures predictable memory usage throughout the ingestion run.

From RAM to Disk

Once all files are processed, the pipeline uses SQLite’s online backup API to flush the in‑memory database to a single on‑disk file. The implementation in src/pipeline/pipeline.c orchestrates this transfer:

/* After all files are ingested, dump the in‑memory DB to disk */
void dump_database(const char *out_path) {
    sqlite3_backup *bk = sqlite3_backup_init(disk_db, "main", mem_db, "main");
    sqlite3_backup_step(bk, -1);
    sqlite3_backup_finish(bk);
    sqlite3_close(mem_db);   // releases all RAM‑held structures
}

The sqlite3_backup_step call with -1 copies all pages in a single transaction, producing a compact, portable SQLite file ready for production queries.

Guaranteed Memory Release

After the dump completes, the pipeline explicitly releases all temporary allocations. The sqlite3_close(mem_db) call frees the in‑memory SQLite instance, while the LZ4‑compressed buffers are deallocated immediately after insertion. This guarantees that the large temporary memory usage exists only for the duration of the indexing run, returning RAM to the operating system before the process exits.

Code Walkthrough: Pipeline Implementation

The entry point for the RAM‑first pipeline is src/pipeline/pipeline.c, which coordinates the ingestion, compression, and storage steps. Each file follows this path:

/* In the pipeline, each file is read, compressed, and stored */
void ingest_file(const char *path) {
    char *raw = read_file(path);
    char *compressed = malloc(LZ4_compressBound(strlen(raw)));
    int csize = lz4_compress(raw, compressed, strlen(raw), 
                             LZ4_compressBound(strlen(raw)));
    sqlite3_exec(db, "INSERT INTO files (path, data) VALUES (?, ?)",
                 sqlite3_bind_text, path, compressed, csize);
    free(raw);
    free(compressed);
}

The ingest_file function demonstrates the complete lifecycle: raw text is read, compressed using the lz4_compress wrapper, bound to SQLite parameters, and then both the raw and compressed buffers are freed. This pattern repeats for every source file in the repository, maintaining constant memory pressure regardless of the total corpus size.

Summary

• The RAM‑first pipeline achieves speed by keeping the entire indexing workload in RAM, eliminating disk I/O during ingestion.
• LZ4 HC compression in internal/cbm/lz4_store.c reduces memory footprint while allowing fast decompression via the LZ4_compress_HC API.
• An in‑memory SQLite database with FTS5 support provides sub‑millisecond BM25 full‑text search without filesystem interaction.
• The single‑dump approach writes the final database to disk in one operation using sqlite3_backup_step, avoiding fragmented writes.
• Explicit memory cleanup after the dump ensures temporary RAM usage is released immediately, preventing memory bloat in long‑running processes.

Frequently Asked Questions

What is the difference between the RAM‑first pipeline and the incremental pipeline?

The RAM‑first pipeline loads all files into memory and compresses them before writing a single database file to disk, while the incremental pipeline (implemented in src/pipeline/pipeline_incremental.c) writes to disk progressively as files are processed. The RAM‑first approach trades temporary memory usage for indexing speed, whereas the incremental mode uses less RAM but incurs higher I/O overhead.

Why does the pipeline use LZ4 HC instead of standard LZ4 or ZSTD?

The pipeline uses LZ4 HC (high‑compression) mode because it provides a better compression ratio than standard LZ4 while maintaining decompression speeds that match the fast mode. According to the codebase‑memory‑mcp source code, the fixed LZ4_HC_LEVEL setting offers sufficient compression for source code text without the complexity of integrating ZSTD or the memory overhead of dictionary training.

How does the pipeline ensure data integrity during the final dump?

Data integrity is maintained through SQLite’s built‑in backup mechanism. The sqlite3_backup_init and sqlite3_backup_step functions in src/pipeline/pipeline.c perform a transactional copy of the in‑memory database to disk, ensuring that the output file is either fully written or not modified at all if the process is interrupted.

Can the RAM‑first pipeline handle repositories larger than available RAM?

No, the RAM‑first pipeline is designed for repositories that fit within available system memory, as it holds the entire compressed corpus plus SQLite overhead in RAM during indexing. For repositories exceeding RAM capacity, the incremental pipeline in src/pipeline/pipeline_incremental.c should be used instead, as it streams data to disk without the large temporary memory allocation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}