# How the Multi-Pass Indexing Pipeline Works in Codebase-Memory MCP

> Uncover the multi pass indexing pipeline in Codebase Memory MCP. Learn how it transforms source files into a queryable graph through seven phases, supporting parallel execution and incremental updates.

- Repository: [Martin Vogel/codebase-memory-mcp](https://github.com/DeusData/codebase-memory-mcp)
- Tags: internals
- Published: 2026-07-18

---

**The multi-pass indexing pipeline in Codebase-Memory MCP transforms repository source files into a queryable graph through seven distinct phases—from initial file discovery and LZ4 compression to symbol extraction and SQLite serialization—supporting both parallel and sequential execution modes with optional incremental updates.**

The **Codebase-Memory MCP** **multi-pass indexing pipeline** is the core engine that converts raw source code into a rich, navigable graph database. Implemented primarily in [`src/pipeline/pipeline.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pipeline.c), this deterministic system processes repositories through discrete transformation stages, enabling powerful code intelligence queries across complex codebases.

## The Seven Phases of the Multi-Pass Indexing Pipeline

The pipeline executes a deterministic sequence of transformations, each handled by specialized modules in the `src/pipeline/` directory.

### Phase 1: Discovery (File Walking and Ignore Rules)

The pipeline begins in [`discover/discover.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/discover/discover.c), which walks the repository tree, collects file metadata, and applies ignore rules to filter out irrelevant paths before any graph construction begins.

### Phase 2: Structure (Hierarchy Node Creation)

The `pass_structure()` function (lines 4-11 in [`src/pipeline/pipeline.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pipeline.c)) creates the foundational graph hierarchy. It instantiates **Project**, **Folder**, **Package**, and **File** nodes, establishing `CONTAINS_*` edges that mirror the filesystem structure.

### Phase 3: Bulk Load (Compression and Storage)

During initialization inside `cbm_pipeline_new()`, the pipeline allocates the graph buffer (`cbm_gbuf`) and reads each source file. Content is compressed using **LZ4 HC** before storage, minimizing memory footprint while preserving full source access for downstream passes.

### Phase 4: Definitions (Symbol Extraction)

The [`src/pipeline/pass_definitions.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pass_definitions.c) module parses each file to extract top-level definitions including functions, types, and constants. These become distinct nodes in the graph, invoked from both sequential and parallel execution paths.

### Phase 5: Import Resolution and Semantic Edges

This phase resolves cross-file relationships through three specialized passes:

- **[`pass_calls.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/pass_calls.c)**: Resolves function and method calls, creating `CALLS` edges
- **[`pass_usages.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/pass_usages.c)**: Tracks identifier usages via `USES` edges  
- **[`pass_semantic.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/pass_semantic.c)**: Builds higher-level relationships like `OVERRIDES` and `IMPLEMENTS`

### Phase 6: Post-Passes (Tests, Infrastructure, and Complexity)

Optional enrichment passes add domain-specific metadata:

- **[`pass_tests.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/pass_tests.c)**: Detects test files and links them to implementation code
- **[`pass_infrascan.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/pass_infrascan.c)**: Creates `Route` nodes for cloud infrastructure bindings
- **[`pass_complexity.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/pass_complexity.c)**: Calculates cyclomatic complexity metrics
- **HTTP links** and **Git history** integration

### Phase 7: Dump (Serialization to SQLite)

The `dump_and_persist_hashes()` function (lines 89-121 in [`src/pipeline/pipeline.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pipeline.c)) serializes the in-memory graph buffer to a SQLite database. It optionally persists file hashes to enable incremental re-indexing in subsequent runs.

## Execution Modes: Parallel vs. Sequential

The pipeline supports two execution strategies controlled by the `CBM_MODE_FULL` flag and the `CBM_INDEX_SINGLE_THREAD` environment variable.

**Sequential Mode**: Executed via `run_sequential_pipeline()`, this path processes passes one-by-one on the calling thread. It builds a package map first, then executes definition extraction, K8s detection, LSP cross-file resolution, calls, usages, and semantic passes in strict order.

**Parallel Mode**: Invoked through `run_parallel_pipeline()`, this approach shards heavy computation across a worker pool implemented in [`src/pipeline/worker_pool.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/worker_pool.c). File extraction, registry building, and LSP cross-file resolution run concurrently. A **shared ID counter** (`shared_ids`) guarantees unique node IDs across threads. Both modes execute identical predump passes—including decorator tags, config-linking, and similarity analysis—via `run_predump_passes()` (lines 70-94).

## Incremental Indexing for Large Codebases

For repositories with existing indices, `try_incremental_or_delete_db()` (lines 105-139) checks whether the current file set matches stored hashes. When the delta is small, `cbm_pipeline_run_incremental()` updates only changed files rather than rebuilding the entire graph. If the divergence is too large, the system deletes the old database and triggers a full re-index through the standard pipeline.

## Configuration: Workers and Fast Mode

Worker count is determined by `effective_worker_count()`, which respects the `CBM_INDEX_SINGLE_THREAD` environment variable to force single-threaded execution.

**Fast Mode** (`CBM_MODE_FAST`) optimizes for CI environments by skipping moderate-cost passes—including similarity analysis and certain semantic edges—through modified `run_predump_passes()` logic.

## Key Source Files and Architecture

| File | Role |
|------|------|
| [`src/pipeline/pipeline.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pipeline.c) | Orchestrates creation, mode selection, predump passes, and dumping |
| [`src/pipeline/pipeline.h`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pipeline.h) | Public API: `cbm_pipeline_new`, `cbm_pipeline_run`, `cbm_pipeline_set_persistence` |
| [`src/pipeline/pipeline_incremental.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pipeline_incremental.c) | Handles incremental re-indexing logic |
| [`src/pipeline/worker_pool.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/worker_pool.c) | Thread pool implementation for parallel execution |
| [`src/pipeline/pass_structure.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pass_structure.c) | Builds folder/file hierarchy (inline in [`pipeline.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/pipeline.c)) |
| [`src/pipeline/pass_definitions.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pass_definitions.c) | Extracts top-level symbols |
| [`src/pipeline/pass_calls.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pass_calls.c) | Resolves function and method calls |
| [`src/pipeline/pass_usages.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pass_usages.c) | Resolves identifier usages |
| [`src/pipeline/pass_semantic.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pass_semantic.c) | Generates semantic edges |
| [`src/pipeline/pass_infrascan.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pass_infrascan.c) | Infrastructure route detection |
| [`src/pipeline/pass_tests.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pass_tests.c) | Test file linking |
| [`src/pipeline/pass_complexity.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pass_complexity.c) | Complexity metrics calculation |

## Practical Implementation Examples

Create a full index with persistence:

```c
cbm_pipeline_t *pl = cbm_pipeline_new("/path/to/repo", NULL, CBM_MODE_FULL);
cbm_pipeline_set_persistence(pl, true);
int rc = cbm_pipeline_run(pl);
if (rc == 0) {
    printf("Indexing succeeded – DB written to %s\n",
           cbm_pipeline_db_path(pl));
}
cbm_pipeline_free(pl);

```

Run a fast, single-threaded index for CI environments:

```c
cbm_pipeline_t *pl = cbm_pipeline_new("/path/to/repo", "mygraph.db", CBM_MODE_FAST);
setenv("CBM_INDEX_SINGLE_THREAD", "1", 1);
int rc = cbm_pipeline_run(pl);
cbm_pipeline_free(pl);

```

## Summary

- The multi-pass indexing pipeline processes repositories through seven deterministic phases from discovery to SQLite serialization
- **Structure creation** establishes the hierarchy via `pass_structure()` while **definitions extraction** populates symbol nodes in [`pass_definitions.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/pass_definitions.c)
- **Parallel execution** uses a worker pool with shared ID counters; **sequential mode** runs passes on the main thread via `run_sequential_pipeline()`
- **Incremental indexing** compares file hashes in `try_incremental_or_delete_db()` to avoid full rebuilds when only minor changes occur
- **Fast mode** (`CBM_MODE_FAST`) skips expensive semantic analysis for quicker CI feedback
- All operations are orchestrated through [`src/pipeline/pipeline.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pipeline.c) with modular pass implementations in `src/pipeline/pass_*.c` files

## Frequently Asked Questions

### What is the difference between CBM_MODE_FULL and CBM_MODE_FAST?

`CBM_MODE_FULL` executes all seven phases including semantic edge generation and similarity analysis, producing the most complete graph. `CBM_MODE_FAST` skips moderate-cost passes like similarity calculations and certain semantic edges in `run_predump_passes()`, making it ideal for CI pipelines where speed matters more than deep semantic analysis.

### How does the pipeline handle concurrent file processing?

When `CBM_INDEX_SINGLE_THREAD` is not set and the mode is full, `run_parallel_pipeline()` distributes file extraction and LSP resolution across a worker pool defined in [`src/pipeline/worker_pool.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/worker_pool.c). A `shared_ids` counter ensures unique node IDs across threads, while `run_predump_passes()` synchronizes the final graph enrichment steps before dumping to SQLite.

### Can I resume an interrupted indexing operation?

While the pipeline does not support resuming mid-pass, it supports incremental indexing via `cbm_pipeline_run_incremental()` in [`src/pipeline/pipeline_incremental.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pipeline_incremental.c). If a previous database exists and file hashes match sufficiently, only changed files are re-indexed. Otherwise, the old database is deleted and a full re-index starts fresh.

### Where are the graph files stored after indexing?

By default, the pipeline creates a `.codebase-memory` graph file in the repository root. You can specify a custom path via the second parameter to `cbm_pipeline_new()` and enable persistence with `cbm_pipeline_set_persistence()`. The final SQLite database path is accessible via `cbm_pipeline_db_path()`.