# How the Multi-Pass Indexing Pipeline Works in codebase-memory-mcp: A Deep Dive

> Explore the internal process of the multi-pass indexing pipeline in codebase-memory-mcp. Understand how source trees transform into knowledge graphs via an in-memory buffer and 17+ specialized passes.

- Repository: [Martin Vogel/codebase-memory-mcp](https://github.com/DeusData/codebase-memory-mcp)
- Tags: internals
- Published: 2026-07-17

---

**TLDR:** The multi-pass indexing pipeline in codebase-memory-mcp is a RAM-first, sequential processing system that transforms raw source trees into a rich knowledge graph through 17+ specialized passes, operating entirely on an in-memory buffer with LZ4-HC compression before persisting to SQLite.

The **multi-pass indexing pipeline** is the core engine of the `DeusData/codebase-memory-mcp` repository, responsible for converting source code into a queryable graph database. Unlike traditional single-pass analyzers, this architecture processes files multiple times—each pass focusing on a specific concern—while maintaining all intermediate data in RAM for maximum throughput. This design enables the system to extract definitions, resolve cross-file type dependencies, and detect semantic similarities without repeated disk I/O.

## Pipeline Initialization and Global Locking

Before any processing begins, the pipeline acquires exclusive access to the target database. In [`src/pipeline/pipeline.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pipeline.c) (lines 54–71), a spin-lock named `g_pipeline_busy` guarantees that only one pipeline run touches a given SQLite database at a time. This prevents corruption during concurrent operations.

The orchestration starts with `cbm_pipeline_new()` (lines 66–84), which allocates a `cbm_pipeline_t` structure, records the repository path, resolves Git context, and generates a project name via `cbm_project_name_from_path()`. Immediately after, the discovery phase in [`discover/discover.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/discover/discover.c) walks the filesystem, respecting `.gitignore`, `.cbmignore`, and OS-specific exclusion rules to build a list of `cbm_file_info_t` structures.

## The Sequential Pass Architecture

The **multi-pass indexing pipeline** follows a fixed execution order defined by the `seq_passes` table in [`src/pipeline/pipeline.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pipeline.c) (lines 898–904). Each pass is a function pointer entry that operates on the same in-memory graph buffer, allowing later passes to consume nodes and edges created by earlier ones.

### Structure Pass: Building the Hierarchy

The first pass, `pass_structure` (lines 47–78 in [`pipeline.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/pipeline.c)), establishes the skeleton of the knowledge graph. It creates high-level nodes for **Project**, **Branch**, **Folder**, and **File**, and inserts `CONTAINS_*` edges that mirror the repository's directory structure. This hierarchy serves as the foundation for all subsequent relationship mapping.

### Definitions Pass: Extracting Symbols

Once the file structure is established, `cbm_pipeline_pass_definitions` in [`src/pipeline/pass_definitions.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pass_definitions.c) (lines 4–9) parses each source file using Tree-sitter. This pass extracts **definitions**—functions, classes, methods, variables, and modules—and registers them in a central registry for later lookup. Each definition becomes a node in the graph buffer, tagged with line numbers and source spans.

### Hybrid LSP Cross-File Resolution

Before the pipeline can resolve inter-file relationships, it runs `cbm_pipeline_pass_lsp_cross` from [`src/pipeline/pass_lsp_cross.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pass_lsp_cross.c) (line 785). This "Hybrid LSP" layer re-reads every source file to build type-aware resolution tables, handling imports, generics, inheritance, and other cross-file dependencies. Unlike standalone language servers, this pass operates offline using the same in-memory representation as the rest of the pipeline.

### Relationship and Semantic Passes

With definitions and cross-file type tables available, the pipeline proceeds to map relationships:

- **Calls Pass** (`cbm_pipeline_pass_calls`): Consumes the definition registry and LSP tables to create `CALLS` edges, including macro-mediated dispatch.
- **Usages Pass** (`cbm_pipeline_pass_usages` in [`src/pipeline/pass_usages.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pass_usages.c), lines 2–12): Analyzes variable read/write patterns to emit `READS`, `WRITES`, and `USES_TYPE` edges.
- **Semantic Pass** (`cbm_pipeline_pass_semantic` in [`src/pipeline/pass_semantic.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pass_semantic.c), lines 2–12): Generates high-level edges such as `INHERITS`, `IMPLEMENTS`, and `DECORATES` based on type information gathered by the LSP pass.
- **Tests Pass** (`cbm_pipeline_pass_tests` in [`src/pipeline/pass_tests.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pass_tests.c), lines 2–10): Links test functions to the production code they exercise via `TESTS` edges.

### Metadata, Similarity, and Configuration Passes

The final set of passes enriches the graph with auxiliary data:

- **Decorator-Tags Pass** (`cbm_pipeline_pass_decorator_tags`, called from [`pipeline.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/pipeline.c) line 752): Extracts language-specific decorators (e.g., Python `@dataclass`, TypeScript `@Component`) and stores them as node properties.
- **Similarity Pass** (`cbm_pipeline_pass_similarity` in [`src/pipeline/pass_similarity.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pass_similarity.c), lines 2–9): Computes MinHash fingerprints for each definition and creates `SIMILAR_TO` edges for near-duplicate detection.
- **Semantic-Edges Pass** (`cbm_pipeline_pass_semantic_edges` in [`src/pipeline/pass_semantic_edges.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pass_semantic_edges.c), lines 2–5): Emits `SEMANTICALLY_RELATED` edges that combine structural similarity, type similarity, and MinHash results.
- **Config-Link Pass** (`cbm_pipeline_pass_configlink` in [`src/pipeline/pass_configlink.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pass_configlink.c), called from [`pipeline.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/pipeline.c) line 764): Connects configuration files (e.g., [`package.json`](https://github.com/DeusData/codebase-memory-mcp/blob/main/package.json), [`pyproject.toml`](https://github.com/DeusData/codebase-memory-mcp/blob/main/pyproject.toml)) to the code entities they configure.
- **Complexity Pass** (`cbm_pipeline_pass_complexity`): Calculates cyclomatic complexity metrics and stores them as node properties.

## Memory Management and Persistence

The entire **multi-pass indexing pipeline** operates on a pre-allocated memory budget controlled by `CBM_MEM_BUDGET_MB`. Files are read once, compressed using LZ4-HC, and stored in an in-memory SQLite buffer, giving later passes fast random access without hitting the disk again. This RAM-first approach ensures that even large codebases index quickly while maintaining bounded memory usage.

The pipeline is also **interrupt-aware**: a `p->cancelled` flag is checked between passes, allowing graceful abortion without corrupting the database. After the final pass completes, `cbm_gbuf_dump()` writes the in-memory graph buffer to a compressed SQLite file (`graph.db.zst`), and `cbm_mem_release()` frees the allocated memory.

## Running the Pipeline

You can trigger the multi-pass indexing pipeline via the CLI or programmatically:

```bash

# Index a repository from the command line

codebase-memory-mcp cli index_repository '{"repo_path":"/path/to/my/project"}'

```

To invoke the pipeline programmatically in C:

```c
#include "pipeline/pipeline.h"

cbm_pipeline_t *pl = cbm_pipeline_new(
    "/path/to/my/project",                               /* repo_path */
    "/home/user/.cache/codebase-memory-mcp/graph.db.zst", /* db_path */
    CBM_INDEX_MODE_FULL);                                 /* mode = full re-index */

cbm_pipeline_set_persistence(pl, true); /* write .codebase-memory artifact */

if (cbm_pipeline_try_lock()) {           /* acquire global lock */
    int rc = cbm_pipeline_run(pl);       /* runs all passes in order */
    cbm_pipeline_unlock();
    /* rc == 0 → success, non-zero → error/cancel */
}
cbm_pipeline_free(pl);

```

To query nodes generated by specific passes:

```bash
codebase-memory-mcp cli search_graph '{
  "project":"my-project",
  "label":"Function",
  "name_pattern":"^process_.*"
}'

```

## Summary

- The **multi-pass indexing pipeline** in `DeusData/codebase-memory-mcp` processes source code through 17+ sequential passes, each specializing in a single concern like structure, definitions, or semantics.
- All passes operate on an in-memory graph buffer with **LZ4-HC compression**, ensuring fast random access without repeated disk I/O.
- Critical passes include `pass_structure` for hierarchy, `cbm_pipeline_pass_definitions` for symbol extraction, and `cbm_pipeline_pass_lsp_cross` for cross-file type resolution.
- The pipeline uses a **global spin-lock** (`g_pipeline_busy`) to prevent concurrent database corruption and supports graceful interruption via `p->cancelled`.
- Execution is defined by the `seq_passes` array in [`src/pipeline/pipeline.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pipeline.c), which specifies the exact order of transformation stages.
- Output persists as a compressed SQLite database (`graph.db.zst`) suitable for high-performance code intelligence queries.

## Frequently Asked Questions

### What is the order of passes in the codebase-memory indexing pipeline?

The pass order is hard-coded in the `seq_passes` array in [`src/pipeline/pipeline.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pipeline.c) (lines 898–904). It begins with structure discovery, followed by definitions extraction, LSP cross-file resolution, calls mapping, usages analysis, semantic relationship detection, and finally similarity and complexity calculations. This sequential design ensures that each pass can consume graph nodes created by previous stages.

### How does codebase-memory-mcp handle memory during indexing?

The pipeline operates on a **RAM-first** architecture with a configurable memory budget (`CBM_MEM_BUDGET_MB`). Source files are loaded once, compressed with LZ4-HC, and stored in an in-memory SQLite buffer. All passes read from and write to this same buffer, eliminating redundant disk access and keeping memory usage bounded even for large repositories.

### What is the Hybrid LSP pass and why is it necessary?

The **Hybrid LSP pass** (`cbm_pipeline_pass_lsp_cross` in [`src/pipeline/pass_lsp_cross.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pass_lsp_cross.c)) performs offline type-aware resolution across files, handling imports, generics, and inheritance without requiring a running language server. It is necessary because it populates cross-file resolution tables that subsequent passes (like calls and semantic analysis) depend on to accurately map relationships between distant code entities.

### Can the indexing pipeline be interrupted safely?

Yes. The pipeline checks a `p->cancelled` flag between each pass, allowing it to abort gracefully without corrupting the SQLite database. This design supports responsive user interfaces where indexing might need to stop immediately due to system changes or user input, while the global lock (`g_pipeline_busy`) ensures atomicity.