# How the RAM-First Indexing Pipeline Handles LZ4 Compression in Codebase-Memory-MCP

> Learn how the RAM-first indexing pipeline uses LZ4 HC compression to minimize memory footprint and achieve O(1) decompression speed for efficient graph analysis.

- Repository: [Martin Vogel/codebase-memory-mcp](https://github.com/DeusData/codebase-memory-mcp)
- Tags: internals
- Published: 2026-07-28

---

**The RAM-first indexing pipeline uses LZ4 high-compression (HC) mode during the Bulk Load phase to compress source files before storing them in an in-memory graph buffer, enabling low-memory-footprint indexing with O(1) decompression speed for downstream analysis passes.**

The Codebase-Memory-MCP project implements a memory-first architecture that keeps the entire project graph in RAM throughout the multi-pass indexing workflow. By compressing source text using vendored LZ4 wrappers rather than storing raw bytes, the pipeline minimizes memory pressure on large codebases while maintaining instant random access to file contents for symbol extraction and semantic analysis.

## Seven-Phase RAM-First Pipeline Architecture

The pipeline orchestrates its work across seven distinct phases defined in [`src/pipeline/pipeline.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pipeline.c), operating entirely on an in-memory graph buffer (`cbm_gbuf_t`) before serializing results to SQLite.

| Phase | Description | Source File |
|-------|-------------|-------------|
| **Discover** | Recursively walks the repository, applying exclude filters | [`discover/discover.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/discover/discover.c) |
| **Structure** | Creates Project → Folder → Package → File hierarchy nodes | [`pipeline/pass_pkgmap.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/pipeline/pass_pkgmap.c) |
| **Bulk Load** | Loads source files, compresses with LZ4 HC, stores compressed blobs | [`pipeline/pipeline.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/pipeline/pipeline.c) |
| **Extract Definitions** | Parses compressed sources and extracts symbol nodes | [`pipeline/pass_definitions.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/pipeline/pass_definitions.c) |
| **Resolve Semantic Edges** | Resolves imports, calls, and usage relationships | [`pipeline/pass_semantic.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/pipeline/pass_semantic.c) |
| **Post-passes** | Optional analysis: test detection, Git history, HTTP links | `pipeline/pass_*.c` |
| **Dump** | Serializes the graph to `graph.db.zst` via SQLite | [`pipeline/pipeline.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/pipeline/pipeline.c), [`store/store.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/store/store.c) |

## LZ4 Compression in the Bulk Load Phase

During phase 3 (Bulk Load), the pipeline passes every source file through a thin LZ4 wrapper defined in [`internal/cbm/vendored/lz4/lz4hc.h`](https://github.com/DeusData/codebase-memory-mcp/blob/main/internal/cbm/vendored/lz4/lz4hc.h). The wrapper exposes three functions that handle buffer sizing, compression, and decompression:

- **`cbm_lz4_bound()`** — Computes the maximum compressed size required for buffer allocation
- **`cbm_lz4_compress_hc()`** — Applies LZ4 high-compression mode for optimal size reduction
- **`cbm_lz4_decompress()`** — Restores original text with constant-time complexity

The compression sequence implemented in [`src/pipeline/pipeline.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pipeline.c) follows this pattern:

```c
int bound = cbm_lz4_bound(src_len);
int comp_len = cbm_lz4_compress_hc(src, src_len, dst_buf, bound);

```

Downstream passes access the original source by invoking:

```c
int out_len = cbm_lz4_decompress(comp_buf, comp_len, out_buf, src_len);

```

### Memory and Performance Characteristics

The LZ4 HC mode provides a favorable trade-off for the RAM-first architecture. Because LZ4 decompression is **O(1)** and operates at memory-copy speeds, the pipeline can store source text in a compact form within the graph buffer while maintaining near-instant random access for the Extract Definitions and Semantic Edge resolution phases.

## Vendored LZ4 Implementation

The project embeds a vendored copy of the LZ4 library to ensure deterministic behavior across platforms:

- **Header**: [`internal/cbm/vendored/lz4/lz4hc.h`](https://github.com/DeusData/codebase-memory-mcp/blob/main/internal/cbm/vendored/lz4/lz4hc.h) — Declares the `cbm_lz4_*` wrapper API
- **Source**: [`internal/cbm/vendored/lz4/lz4hc.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/internal/cbm/vendored/lz4/lz4hc.c) — Contains the high-compression algorithm derived from upstream LZ4

The test suite in [`tests/test_lz4.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/tests/test_lz4.c) validates round-trip correctness, ensuring that compressed blobs decompress to byte-identical source text and that error conditions (such as insufficient buffer sizes) are handled gracefully.

## Practical Code Examples

### Compressing and Decompressing Source Text

This pattern from [`tests/test_lz4.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/tests/test_lz4.c) demonstrates the basic compression workflow used throughout the pipeline:

```c
#include "pipeline/pipeline.h"

void process_source(const char *src, int src_len) {
    int bound = cbm_lz4_bound(src_len);
    char *compressed = malloc(bound);
    int comp_len = cbm_lz4_compress_hc(src, src_len, compressed, bound);
    if (comp_len < 0) { /* handle compression error */ }

    char *decompressed = malloc(src_len);
    int dec_len = cbm_lz4_decompress(compressed, comp_len, decompressed, src_len);
    
    assert(dec_len == src_len);
    assert(memcmp(src, decompressed, src_len) == 0);
    
    free(compressed);
    free(decompressed);
}

```

### Loading Compressed Blobs into the Graph Buffer

The bulk load implementation in [`src/pipeline/pipeline.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pipeline.c) illustrates how compressed sources enter the in-memory graph:

```c
static int load_source_into_gbuf(cbm_pipeline_t *p, const char *path) {
    char *src = NULL;
    size_t src_len = 0;
    if (!cbm_read_file(path, &src, &src_len))
        return -1;

    int bound = cbm_lz4_bound((int)src_len);
    char *cbuf = malloc(bound);
    int comp_len = cbm_lz4_compress_hc(src, (int)src_len, cbuf, bound);
    free(src);
    
    if (comp_len < 0) { free(cbuf); return -1; }

    cbm_gbuf_add_blob(p->gbuf, path, cbuf, comp_len, (int)src_len);
    return 0;
}

```

### Executing the Full Pipeline

To run the RAM-first indexing process from a client application:

```c
int main(int argc, char **argv) {
    const char *repo = argv[1];
    cbm_pipeline_t *pl = cbm_pipeline_new(repo, NULL, CBM_MODE_FULL);
    if (!pl) return 1;

    cbm_pipeline_set_persistence(pl, true);
    int rc = cbm_pipeline_run(pl);
    
    printf("Indexing finished with status %d, project %s\n",
           rc, cbm_pipeline_project_name(pl));
    
    cbm_pipeline_free(pl);
    return rc;
}

```

## Summary

- The **RAM-first indexing pipeline** processes codebases entirely in memory across seven phases before persisting the final graph to SQLite
- **LZ4 HC compression** is applied during the Bulk Load phase to store source text compactly within the `cbm_gbuf_t` graph buffer
- The wrapper functions `cbm_lz4_bound()`, `cbm_lz4_compress_hc()`, and `cbm_lz4_decompress()` in [`internal/cbm/vendored/lz4/lz4hc.h`](https://github.com/DeusData/codebase-memory-mcp/blob/main/internal/cbm/vendored/lz4/lz4hc.h) provide the compression interface
- Compressed blobs enable O(1) random access decompression, allowing downstream passes to parse source code without disk I/O
- The vendored implementation in [`internal/cbm/vendored/lz4/lz4hc.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/internal/cbm/vendored/lz4/lz4hc.c) is validated by [`tests/test_lz4.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/tests/test_lz4.c) to ensure correctness

## Frequently Asked Questions

### Why does the pipeline use LZ4 HC instead of standard LZ4?

The high-compression (HC) mode trades slightly higher CPU usage during the initial Bulk Load phase for improved compression ratios. Since the **RAM-first** architecture keeps all source text in memory simultaneously, maximizing compression reduces memory pressure for large repositories, while decompression speed remains identical to standard LZ4.

### Where are the compressed source files stored during indexing?

Compressed source blobs reside in the in-memory graph buffer (`cbm_gbuf_t`) allocated during pipeline initialization. They remain in RAM throughout all seven phases and are only written to disk during the final Dump phase when the complete graph serializes to `graph.db.zst`.

### How does the pipeline handle LZ4 compression errors?

The wrapper functions return negative values on failure. For example, `cbm_lz4_compress_hc()` returns a negative integer if compression fails, which the bulk load logic in [`src/pipeline/pipeline.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pipeline.c) checks to abort processing for that specific file while allowing the pipeline to continue indexing remaining sources.

### Can the compression level be tuned in the LZ4 wrapper?

The current vendored implementation in [`internal/cbm/vendored/lz4/lz4hc.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/internal/cbm/vendored/lz4/lz4hc.c) uses default high-compression settings optimized for source code text. The wrapper API (`cbm_lz4_compress_hc`) does not expose compression level parameters, providing instead a fixed balance of compression ratio and speed suitable for the memory-first indexing workload.