RAM-first Pipeline Memory Management Strategy in codebase-memory-mcp

The RAM-first pipeline keeps every indexing phase completely in memory, compresses source data with LZ4 HC, buffers the graph in an in-memory SQLite database, and performs a single atomic dump at the end, explicitly releasing all allocated memory back to the operating system.

The codebase-memory-mcp repository implements an ultra-fast indexing system designed to handle large codebases without intermediate disk writes. Its core innovation is the RAM-first pipeline, a memory management strategy that minimizes I/O overhead by maintaining all transient data structures in RAM until the final persistence step. This approach ensures that the process never exceeds the peak RAM required for the in-memory structures, making it efficient for repositories of any size.

Core Memory Architecture

The RAM-first pipeline is built on two foundational techniques: aggressive compression during ingestion and an in-memory graph buffer that replaces traditional on-disk temporary files.

LZ4 HC Compression During Discovery

Before any data enters the graph structure, source files are compressed using LZ4 HC (High Compression). This happens immediately upon reading during the bulk load phase.

According to the pipeline orchestrator in src/pipeline/pipeline.c, the process is explicitly commented as: Bulk load sources (read + LZ4 HC compress)【L7‑9】. By compressing data at the boundary between discovery and graph construction, the system minimizes the memory footprint of raw source text while keeping it available for analysis.

In-Memory Graph Buffer

Instead of writing intermediate nodes and edges to temporary files, the pipeline stores them in a dedicated graph buffer (cbm_gbuf_t). This buffer functions as an in-memory SQLite database using the :memory: connection string.

The function cbm_gbuf_dump_to_sqlite(p->gbuf, db_path) operates on this in-memory connection, allowing the graph to be queried and manipulated using standard SQL without ever touching the disk. Only when the entire indexing process completes does this buffer flush to a persistent SQLite file.

The Six-Phase Execution Flow

The RAM-first strategy follows a strict sequence designed to maximize throughput while controlling memory usage:

1. Read and Compress: Each source file is read and LZ4-HC compressed before entering the graph.
2. Populate Graph Buffer: Nodes and edges are stored in the cbm_gbuf_t structure backed by in-memory SQLite.
3. Run Analysis Passes: Structure parsing, definition extraction, and LSP cross-resolution all execute against the in-memory buffer.
4. Collect Memory: After the heavy parallel extraction phase, the pipeline explicitly calls cbm_mem_collect() to reclaim large allocator pages before proceeding.
5. Atomic Dump: The entire graph is flushed to a persistent SQLite file in a single operation via cbm_gbuf_dump_to_sqlite().
6. Release Resources: The graph buffer, registry, and temporary caches are freed, returning all RAM to the OS.

Memory Pressure Mitigation and Cleanup

Explicit Garbage Collection

To prevent out-of-memory errors on very large repositories, the pipeline implements manual memory reclamation. After the parallel extraction phase completes, the orchestrator calls cbm_mem_collect()【L46‑48】 to force the release of extraction allocator pages back to the heap.

This explicit collection occurs before the registry build phase, ensuring that peak memory usage from extraction does not compound with the requirements of building the final symbol registry.

Final Dump and Release

The persistence phase handles the transition from volatile to durable storage atomically. The cbm_gbuf_dump_to_sqlite() function writes the in-memory graph to the final database path (typically ~/.cache/codebase-memory-mcp/<project>.db), after which cbm_gbuf_free() destroys the buffer.

The README confirms this behavior: "Memory is released back to the OS after indexing completes"【L33‑34】.

Memory Profiling

Throughout execution, the pipeline logs current and peak RSS at each phase boundary using functions like log_phase_mem("registry_build")【L30‑35】. This telemetry allows operators to verify that memory is actually being released between phases and that the RAM-first strategy is functioning as expected.

Implementation Examples

To initialize a RAM-first pipeline in C, create a new pipeline instance and run it:

/* Create a new pipeline – operates in RAM-first mode by default */
cbm_pipeline_t *p = cbm_pipeline_new("/path/to/repo", NULL, CBM_MODE_FULL);

/* Optionally enable persistence (writes the final DB) */
cbm_pipeline_set_persistence(p, true);

/* Run the pipeline – all steps happen in RAM */
int rc = cbm_pipeline_run(p);
if (rc != 0) {
    fprintf(stderr, "Indexing failed: %d\n", rc);
}

/* After run() returns, the in-memory graph is freed and the DB lives at
   ~/.cache/codebase-memory-mcp/<project>.db */
cbm_pipeline_free(p);

For command-line usage, the RAM-first pipeline is the default for full indexing:

# Index a repository (RAM-first mode is the default for full indexing)

codebase-memory-mcp cli index_repository '{"repo_path":"/my/project"}'

Summary

• The RAM-first pipeline maintains all indexing data in memory using LZ4 HC compression and an in-memory SQLite graph buffer (cbm_gbuf_t).
• Memory pressure is mitigated via explicit cbm_mem_collect() calls after the extraction phase to prevent OOM errors.
• A single atomic dump occurs at the end via cbm_gbuf_dump_to_sqlite(), after which cbm_gbuf_free() and cleanup routines return all RAM to the OS.
• Phase boundary logging in src/pipeline/pipeline.c provides visibility into current and peak memory usage.
• This strategy eliminates intermediate disk I/O, making it optimal for high-performance codebase indexing.

Frequently Asked Questions

What is the RAM-first pipeline in codebase-memory-mcp?

The RAM-first pipeline is the default indexing strategy in codebase-memory-mcp that performs all discovery, parsing, and graph construction in RAM without intermediate disk writes. It compresses source files using LZ4 HC and stores the graph in an in-memory SQLite database (:memory:), only persisting to disk once at the end of the process.

How does the pipeline handle memory pressure on large repositories?

The pipeline explicitly calls cbm_mem_collect() in src/pipeline/pipeline.c【L46‑48】 after the parallel extraction phase to reclaim large allocator pages before building the symbol registry. This prevents the cumulative memory usage from extraction and registry construction from exceeding available RAM.

When is data written to disk during the RAM-first indexing process?

Data is written to disk exactly once, at the very end of the pipeline execution. The function cbm_gbuf_dump_to_sqlite() flushes the entire in-memory graph buffer to a persistent SQLite file (located at ~/.cache/codebase-memory-mcp/<project>.db). Prior to this final dump, all operations occur against the in-memory SQLite connection.

How can I verify that memory is actually being released after indexing?

The pipeline logs resident set size (RSS) metrics at each phase boundary using log_phase_mem() calls【L30‑35】. By monitoring these logs, you can observe the peak memory usage during extraction and confirm that RSS drops after cbm_pipeline_run() completes and the cleanup routines free the gbuf, registry, and temporary caches.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}