How the RAM-First Pipeline Achieves Extreme Indexing Speed in Codebase-Memory-MCP

The RAM-first pipeline in Codebase-Memory-MCP achieves extreme indexing speed by keeping the entire processing flow in memory, eliminating disk I/O through memory-mapped files, and using lock-free parallel processing until a single bulk persistence step.

Codebase-Memory-MCP (CBM) implements a deliberately aggressive RAM-first indexing pipeline designed to process multi-gigabyte codebases in seconds. Unlike traditional tools that repeatedly read from and write to disk during indexing, this architecture loads the entire repository into contiguous memory and performs all computation—parsing, hashing, and graph construction—without intermediate disk access. The result is a system that can index a 12 GB repository in under 15 seconds on modern multi-core hardware.

The Three Stages of the RAM-First Pipeline

The pipeline consists of three tightly coupled stages that work together to minimize latency and maximize throughput.

Stage 1: In-Memory Repository Loading

The process begins in src/store/store.c with the cbm_store_load_mmap function, which memory-maps the entire source tree into a contiguous buffer (cbm_store_t). Instead of using repeated open/read/close system calls, this stage performs a single bulk load of raw byte streams into RAM.

By storing files in a contiguous memory-mapped buffer, subsequent processing passes operate entirely on in-memory pointers. This eliminates disk seeks and context switches, allowing the parser to work directly on RAM-resident data without copying bytes.

Stage 2: Fast Hash-Based Indexing

Once loaded, the pipeline computes 64-bit xxHash values for every file and logical unit (symbols, imports, AST nodes). According to the vendored header in vendored/xxhash/xxhash.h (lines 39-68), xxHash is an "extremely fast non-cryptographic hash algorithm, working at RAM speed" capable of hashing gigabytes per second on modern CPUs.

These hashes populate a flat hash table structure (cbm_hash_table_t) that resides purely in memory. Because the structure is RAM-resident, lookups and insertions operate at O(1) complexity without requiring disk seeks or page faults.

Stage 3: Parallel Lock-Free Processing

The orchestration layer in src/pipeline/pipeline.c spawns a pool of workers defined in src/pipeline/worker_pool.c. Each worker claims a chunk of the in-memory buffer and performs syntax parsing, symbol extraction, and edge creation independently.

Critical to the speed is the absence of global locks. Workers write results into lock-free queues, enabling linear scaling with core count. The pipeline aggregates these results in memory and performs exactly one disk interaction: a sequential bulk write to the compressed store.

Core Optimizations Driving Speed

Beyond the three-stage architecture, specific implementation details in the source code maximize performance.

Zero-Copy Memory Mapping

The cbm_store_load_mmap implementation provides zero-copy reads by mapping files directly into the process address space. Parsers receive pointers directly into the memory-mapped buffer, eliminating heap allocations and byte-copying overhead that would otherwise consume CPU cycles and pollute cache lines.

Cache-Friendly Data Layouts

All intermediate structures—including hash tables, AST nodes, and edge lists—are allocated from large contiguous arenas rather than scattered heap allocations. This cache-friendly layout maximizes L1/L2 cache hits during the symbol resolution phase, where the pipeline traverses millions of graph edges.

Batch Persistence Strategy

After completing the in-memory pass, the pipeline invokes the ZSTD compression layer in internal/cbm/zstd_store.c. Rather than writing incremental updates, it performs a batch persistence operation that dumps the fully compiled index to disk in a single compressed stream. This avoids the performance degradation associated with many small random writes, leveraging sequential write speeds instead.

Implementation Example

Initialize and run the RAM-first pipeline using the C API:

/* Initialise a RAM-first pipeline */
cbm_pipeline_t *pipeline = cbm_pipeline_new(
    "/path/to/repo",          /* repo root */
    "/path/to/db",            /* output store */
    CBM_MODE_FULL);           /* full-index mode */

/* Run the pipeline – everything stays in RAM until the very end */
int rc = cbm_pipeline_run(pipeline);
if (rc != 0) { /* handle error */ }

/* The resulting index is now persisted (once) to the ZSTD store */
cbm_pipeline_free(pipeline);

For Go applications, use the provided wrapper:

// Using the Go wrapper to start the RAM-first pipeline
package main

import (
    "github.com/DeusData/codebase-memory-mcp/pkg/go/cbm"
)

func main() {
    p := cbm.NewPipeline("/my/project", "/my/db", cbm.ModeFull)
    if err := p.Run(); err != nil {
        panic(err)
    }
    // Index is now stored on disk; the whole indexing run never
    // performed intermediate reads/writes.
}

Key Source Files

The following files implement the RAM-first architecture:

• src/store/store.c – Contains cbm_store_load_mmap for memory-mapped repository loading
• src/pipeline/pipeline.c – Core orchestration coordinating in-memory passes and final bulk dump
• src/pipeline/worker_pool.c – Lock-free parallel worker pool implementation
• internal/cbm/zstd_store.c – Single-shot ZSTD compression and persistence layer
• vendored/xxhash/xxhash.h – Ultra-fast hash implementation (lines 39-68)
• tests/test_pipeline.c – Integration benchmarks validating the 12 GB / 15 second performance claim

Summary

• Memory-mapped loading in src/store/store.c eliminates disk I/O during parsing by mapping the entire repository into RAM.
• xxHash provides RAM-speed hashing for O(1) symbol lookups without disk access.
• Lock-free worker pools enable linear scaling across CPU cores without synchronization overhead.
• Batch persistence writes the complete index exactly once using ZSTD compression, avoiding incremental write penalties.
• The pipeline achieves sub-15-second indexing for 12 GB repositories, as verified in tests/test_pipeline.c.

Frequently Asked Questions

What makes xxHash suitable for high-speed indexing in this pipeline?

xxHash is a non-cryptographic hash algorithm optimized for RAM speed rather than collision resistance. As documented in vendored/xxhash/xxhash.h, it processes data at multiple gigabytes per second, making it ideal for generating 64-bit identifiers for millions of symbols without becoming a CPU bottleneck.

How does the pipeline handle multi-core scaling?

The pipeline spawns a worker pool via src/pipeline/worker_pool.c that divides the in-memory buffer into chunks processed in parallel. Because workers use lock-free queues to submit results rather than contending for global locks, throughput scales linearly with available CPU cores.

When does the RAM-first pipeline actually write to disk?

Disk writes occur exactly once per indexing run. After all parsing, hashing, and graph construction complete in memory, internal/cbm/zstd_store.c performs a single sequential bulk write to persist the compressed index. No intermediate writes occur during the processing stages.

How fast is the RAM-first pipeline compared to traditional tools?

According to the benchmark suite in tests/test_pipeline.c, the pipeline indexes a 12 GB repository in under 15 seconds on modern multi-core hardware. Traditional disk-bound tools typically require repeated read/write cycles that extend this process by orders of magnitude.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}