# Codebase-Memory-MCP Memory Model During Indexing: RAM-First Pipeline and Release Mechanism

> Explore the Codebase-Memory-MCP memory model during indexing. Learn how it uses RAM-first pipelining and a release mechanism to optimize memory usage and persist data efficiently.

- Repository: [Martin Vogel/codebase-memory-mcp](https://github.com/DeusData/codebase-memory-mcp)
- Tags: internals
- Published: 2026-07-13

---

**Codebase-Memory-MCP allocates a configurable percentage of system RAM (defaulting to 75%) for in-memory SQLite indexing using custom slab allocators, then explicitly releases all temporary memory via `cbm_mem_release()` and `cbm_graph_buffer_reset()` after persisting the knowledge graph to disk.**

The **memory model during indexing** in the DeusData/codebase-memory-mcp repository is designed for extreme speed: it constructs the entire code knowledge graph in RAM using LZ4-compressed source buffers and an in-memory SQLite database. Understanding how this RAM-first pipeline manages allocation limits and performs cleanup is essential for indexing large repositories such as the Linux kernel without exhausting system resources.

## Memory Budget Resolution and Initialization

The foundation of the memory model lies in [`src/foundation/mem.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/foundation/mem.c), where the system calculates available resources before indexing begins. The global budget structure `g_mem_budget` stores the computed limit, which constrains all subsequent allocation operations.

### Calculating Available RAM

The function `cbm_mem_resolve_budget()` determines the maximum bytes available for indexing. By default, it allocates 75% of total system RAM unless overridden by the `CBM_MEM_BUDGET_MB` environment variable, which accepts values in megabytes.

```c
cbm_mem_budget_t cbm_mem_resolve_budget(size_t total_ram, double ram_fraction, const char *env) {
    cbm_mem_budget_t result = {0};
    double fraction = ram_fraction > 0 ? ram_fraction : 0.75; // default 75% of RAM
    size_t budget = (size_t)(total_ram * fraction);
    
    if (env && *env) {
        long long env_val = atoll(env);
        if (env_val > 0) {
            budget = (size_t)env_val * 1024 * 1024; // MB to bytes
        }
    }
    result.total_ram = total_ram;
    result.budget_bytes = budget;
    result.ram_fraction = fraction;
    return result;
}

```

### Initialization via cbm_mem_init()

During startup, `cbm_mem_init()` detects total RAM via `cbm_platform_total_ram()` and populates the global `g_mem_budget` structure. This function also checks for the `CBM_MEM_BUDGET_MB` environment variable to allow hard limits:

```c
void cbm_mem_init(void) {
    size_t total_ram = cbm_platform_total_ram();
    const char *env = getenv("CBM_MEM_BUDGET_MB");
    double ram_fraction = 0;
    g_mem_budget = cbm_mem_resolve_budget(total_ram, ram_fraction, env);
    cbm_log_info("mem.init", "budget_bytes", itoa_log(g_mem_budget.budget_bytes));
}

```

## The RAM-First Indexing Pipeline

During active indexing, as implemented in [`src/pipeline/pass_parallel.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pass_parallel.c), the system operates entirely within the pre-calculated memory budget. The pipeline uses a custom slab allocator to manage heavy temporary structures while respecting the global limit.

### Slab Allocation and Per-Worker Budgets

The parallel pipeline divides the total budget among worker threads. Each worker allocates parse trees, symbol tables, and temporary edge lists using slab allocators that respect the global limit calculated in `cbm_mem_resolve_budget()`.

```c
void cbm_pass_parallel_run(cbm_store_t *store) {
    size_t per_worker_mb = cbm_mem_budget() / 1024 / 1024;
    cbm_log_info("pipeline.mem", "per_worker_mb", itoa_log((int)per_worker_mb));
    
    size_t buffer_cap = cbm_mem_budget(); // simplistic use of total budget
    
    // ... indexing operations that allocate heavy structures such as parse trees,
    // symbol tables, and temporary edge lists.
    // The allocation is done via custom slab allocator which respects the budget.
}

```

### In-Memory SQLite and LZ4 Compression

As documented in the repository README, the pipeline maintains the knowledge graph in a **pure in-memory SQLite** database with **LZ4 compression** applied to source file buffers. This architecture avoids disk I/O bottlenecks during graph construction, but requires that the entire working set fit within the RAM budget defined by `CBM_MEM_BUDGET_MB` or the default 75% threshold.

## Memory Release After Indexing

Once indexing finishes, the system executes a coordinated release sequence in `cbm_pass_parallel_run()` to free temporary allocations while preserving the persisted database.

### Persisting to Disk

Before releasing memory, the pipeline performs a single dump operation to persist the in-memory graph:

```c
// After indexing, we dump the in‑memory SQLite DB to disk (single dump)
cbm_store_dump_to_disk(store);

```

### Explicit Buffer Reset and Budget Cleanup

After persistence, the code explicitly resets temporary structures and clears the budget:

```c
// Release all temporary structures – the slab allocator can be reset in bulk
cbm_graph_buffer_reset(store->graph_buf);
// Additional per‑worker temporary allocations are freed here.
// The memory budget is released back to the OS when the process exits,
// but we also explicitly free large buffers.
cbm_mem_release();

```

The `cbm_mem_release()` function in [`src/foundation/mem.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/foundation/mem.c) zeros the global budget structure and logs the release event:

```c
void cbm_mem_release(void) {
    memset(&g_mem_budget, 0, sizeof(g_mem_budget));
    cbm_log_info("mem.release", "status", "released");
}

```

## Configuring Memory Limits

Users can control the **memory model during indexing** through environment variables or programmatic APIs.

### Setting Hard Limits via Environment Variables

To constrain memory usage on memory-constrained systems, set the budget before running the indexer:

```bash
export CBM_MEM_BUDGET_MB=2048
./codebase-memory-mcp index /path/to/repository

```

This overrides the default 75% RAM calculation with a fixed 2 GB limit, as parsed by `cbm_mem_resolve_budget()` in [`src/foundation/mem.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/foundation/mem.c).

### Programmatic Integration

Developers integrating the library can check budget status before performing operations:

```c
// Initialize budget detection from system RAM
cbm_mem_init();

// Verify available memory
size_t budget = cbm_mem_budget();
if (cbm_mem_over_budget()) {
    // Handle constraint - allocated exceeds budget_bytes
}

// ... perform indexing via cbm_pass_parallel_run ...

// Explicitly release all temporary allocations
cbm_mem_release();

```

The `cbm_mem_over_budget()` function compares currently allocated memory (tracked via `cbm_platform_memory_allocated()`) against `g_mem_budget.budget_bytes` to prevent OOM conditions during the RAM-first pipeline execution.

## Summary

- The **memory model during indexing** uses a RAM-first approach, defaulting to 75% of total system RAM unless `CBM_MEM_BUDGET_MB` is set.
- Budget calculation occurs in `cbm_mem_resolve_budget()` in [`src/foundation/mem.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/foundation/mem.c), with detection via `cbm_platform_total_ram()`.
- Custom slab allocators manage temporary structures (parse trees, symbol tables, edge lists) within per-worker budgets defined in [`src/pipeline/pass_parallel.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pass_parallel.c).
- Persistence happens via a single `cbm_store_dump_to_disk()` call, writing the in-memory SQLite database to compressed storage.
- Explicit cleanup via `cbm_graph_buffer_reset()` and `cbm_mem_release()` frees large buffers and zeros the budget struct, returning memory to the operating system.

## Frequently Asked Questions

### What happens if indexing exceeds the memory budget?

The system checks `cbm_mem_over_budget()` against the limit set in `cbm_mem_resolve_budget()`. The custom slab allocator respects these constraints during the allocation of heavy structures like parse trees and symbol tables, preventing uncontrolled memory growth that would exhaust system resources.

### How does Codebase-Memory-MCP determine the default memory limit?

By default, `cbm_mem_init()` calls `cbm_mem_resolve_budget()` with a `ram_fraction` of 0.75, allocating 75% of the total RAM detected via `cbm_platform_total_ram()`. Users can override this by setting the `CBM_MEM_BUDGET_MB` environment variable to a specific megabyte value.

### Is the memory released back to the operating system immediately?

Yes. After `cbm_store_dump_to_disk()` persists the graph to disk, the pipeline calls `cbm_graph_buffer_reset()` to bulk-free the slab-allocated graph buffers, followed by `cbm_mem_release()` to zero the `g_mem_budget` structure. Because the process continues running as an MCP server, the OS immediately reclaims the released pages.

### Can I index repositories larger than available RAM?

No. The RAM-first architecture requires the entire knowledge graph to fit within the configured memory budget during the indexing phase. For repositories approaching memory limits, increase the budget via `CBM_MEM_BUDGET_MB` or ensure the system has sufficient physical RAM to accommodate the default 75% allocation.