How the RAM-First Indexing Pipeline Handles LZ4 Compression in Codebase-Memory-MCP

The RAM-first indexing pipeline uses LZ4 high-compression (HC) mode during the Bulk Load phase to compress source files before storing them in an in-memory graph buffer, enabling low-memory-footprint indexing with O(1) decompression speed for downstream analysis passes.

The Codebase-Memory-MCP project implements a memory-first architecture that keeps the entire project graph in RAM throughout the multi-pass indexing workflow. By compressing source text using vendored LZ4 wrappers rather than storing raw bytes, the pipeline minimizes memory pressure on large codebases while maintaining instant random access to file contents for symbol extraction and semantic analysis.

Seven-Phase RAM-First Pipeline Architecture

The pipeline orchestrates its work across seven distinct phases defined in src/pipeline/pipeline.c, operating entirely on an in-memory graph buffer (cbm_gbuf_t) before serializing results to SQLite.

Phase	Description	Source File
Discover	Recursively walks the repository, applying exclude filters	discover/discover.c
Structure	Creates Project → Folder → Package → File hierarchy nodes	pipeline/pass_pkgmap.c
Bulk Load	Loads source files, compresses with LZ4 HC, stores compressed blobs	pipeline/pipeline.c
Extract Definitions	Parses compressed sources and extracts symbol nodes	pipeline/pass_definitions.c
Resolve Semantic Edges	Resolves imports, calls, and usage relationships	pipeline/pass_semantic.c
Post-passes	Optional analysis: test detection, Git history, HTTP links	pipeline/pass_*.c
Dump	Serializes the graph to graph.db.zst via SQLite	pipeline/pipeline.c, store/store.c

LZ4 Compression in the Bulk Load Phase

During phase 3 (Bulk Load), the pipeline passes every source file through a thin LZ4 wrapper defined in internal/cbm/vendored/lz4/lz4hc.h. The wrapper exposes three functions that handle buffer sizing, compression, and decompression:

• cbm_lz4_bound() — Computes the maximum compressed size required for buffer allocation
• cbm_lz4_compress_hc() — Applies LZ4 high-compression mode for optimal size reduction
• cbm_lz4_decompress() — Restores original text with constant-time complexity

The compression sequence implemented in src/pipeline/pipeline.c follows this pattern:

int bound = cbm_lz4_bound(src_len);
int comp_len = cbm_lz4_compress_hc(src, src_len, dst_buf, bound);

Downstream passes access the original source by invoking:

int out_len = cbm_lz4_decompress(comp_buf, comp_len, out_buf, src_len);

Memory and Performance Characteristics

The LZ4 HC mode provides a favorable trade-off for the RAM-first architecture. Because LZ4 decompression is O(1) and operates at memory-copy speeds, the pipeline can store source text in a compact form within the graph buffer while maintaining near-instant random access for the Extract Definitions and Semantic Edge resolution phases.

Vendored LZ4 Implementation

The project embeds a vendored copy of the LZ4 library to ensure deterministic behavior across platforms:

• Header: internal/cbm/vendored/lz4/lz4hc.h — Declares the cbm_lz4_* wrapper API
• Source: internal/cbm/vendored/lz4/lz4hc.c — Contains the high-compression algorithm derived from upstream LZ4

The test suite in tests/test_lz4.c validates round-trip correctness, ensuring that compressed blobs decompress to byte-identical source text and that error conditions (such as insufficient buffer sizes) are handled gracefully.

Practical Code Examples

Compressing and Decompressing Source Text

This pattern from tests/test_lz4.c demonstrates the basic compression workflow used throughout the pipeline:

#include "pipeline/pipeline.h"

void process_source(const char *src, int src_len) {
    int bound = cbm_lz4_bound(src_len);
    char *compressed = malloc(bound);
    int comp_len = cbm_lz4_compress_hc(src, src_len, compressed, bound);
    if (comp_len < 0) { /* handle compression error */ }

    char *decompressed = malloc(src_len);
    int dec_len = cbm_lz4_decompress(compressed, comp_len, decompressed, src_len);
    
    assert(dec_len == src_len);
    assert(memcmp(src, decompressed, src_len) == 0);
    
    free(compressed);
    free(decompressed);
}

Loading Compressed Blobs into the Graph Buffer

The bulk load implementation in src/pipeline/pipeline.c illustrates how compressed sources enter the in-memory graph:

static int load_source_into_gbuf(cbm_pipeline_t *p, const char *path) {
    char *src = NULL;
    size_t src_len = 0;
    if (!cbm_read_file(path, &src, &src_len))
        return -1;

    int bound = cbm_lz4_bound((int)src_len);
    char *cbuf = malloc(bound);
    int comp_len = cbm_lz4_compress_hc(src, (int)src_len, cbuf, bound);
    free(src);
    
    if (comp_len < 0) { free(cbuf); return -1; }

    cbm_gbuf_add_blob(p->gbuf, path, cbuf, comp_len, (int)src_len);
    return 0;
}

Executing the Full Pipeline

To run the RAM-first indexing process from a client application:

int main(int argc, char **argv) {
    const char *repo = argv[1];
    cbm_pipeline_t *pl = cbm_pipeline_new(repo, NULL, CBM_MODE_FULL);
    if (!pl) return 1;

    cbm_pipeline_set_persistence(pl, true);
    int rc = cbm_pipeline_run(pl);
    
    printf("Indexing finished with status %d, project %s\n",
           rc, cbm_pipeline_project_name(pl));
    
    cbm_pipeline_free(pl);
    return rc;
}

Summary

• The RAM-first indexing pipeline processes codebases entirely in memory across seven phases before persisting the final graph to SQLite
• LZ4 HC compression is applied during the Bulk Load phase to store source text compactly within the cbm_gbuf_t graph buffer
• The wrapper functions cbm_lz4_bound(), cbm_lz4_compress_hc(), and cbm_lz4_decompress() in internal/cbm/vendored/lz4/lz4hc.h provide the compression interface
• Compressed blobs enable O(1) random access decompression, allowing downstream passes to parse source code without disk I/O
• The vendored implementation in internal/cbm/vendored/lz4/lz4hc.c is validated by tests/test_lz4.c to ensure correctness

Frequently Asked Questions

Why does the pipeline use LZ4 HC instead of standard LZ4?

The high-compression (HC) mode trades slightly higher CPU usage during the initial Bulk Load phase for improved compression ratios. Since the RAM-first architecture keeps all source text in memory simultaneously, maximizing compression reduces memory pressure for large repositories, while decompression speed remains identical to standard LZ4.

Where are the compressed source files stored during indexing?

Compressed source blobs reside in the in-memory graph buffer (cbm_gbuf_t) allocated during pipeline initialization. They remain in RAM throughout all seven phases and are only written to disk during the final Dump phase when the complete graph serializes to graph.db.zst.

How does the pipeline handle LZ4 compression errors?

The wrapper functions return negative values on failure. For example, cbm_lz4_compress_hc() returns a negative integer if compression fails, which the bulk load logic in src/pipeline/pipeline.c checks to abort processing for that specific file while allowing the pipeline to continue indexing remaining sources.

Can the compression level be tuned in the LZ4 wrapper?

The current vendored implementation in internal/cbm/vendored/lz4/lz4hc.c uses default high-compression settings optimized for source code text. The wrapper API (cbm_lz4_compress_hc) does not expose compression level parameters, providing instead a fixed balance of compression ratio and speed suitable for the memory-first indexing workload.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}