# Multi-Pass Indexing Pipeline Architecture in Codebase-Memory-MCP

> Explore the multi-pass indexing pipeline architecture in Codebase-Memory-MCP. Discover how this C-based system transforms code into a graph database with parallel analysis and incremental updates.

- Repository: [Martin Vogel/codebase-memory-mcp](https://github.com/DeusData/codebase-memory-mcp)
- Tags: architecture
- Published: 2026-07-13

---

**The Codebase-Memory-MCP multi-pass indexing pipeline is a C-based, staged processing system that transforms source code into an SQLite-backed graph database through parallelized, independent passes—each handling specific analysis tasks like symbol extraction, usage linking, and similarity detection—while supporting incremental updates and fault tolerance via checkpointing.**

The multi-pass indexing pipeline in Codebase-Memory-MCP (MCP) processes repositories into a rich, queryable graph structure by breaking analysis into discrete, composable stages. Implemented in C and organized under `src/pipeline/`, this architecture separates concerns into specialized passes that execute sequentially yet leverage worker pools for parallel file processing, enabling scalable indexing of large monorepos without compromising robustness or data integrity.

## Core Architecture Components

### Pipeline Controller

The central orchestrator resides in [`src/pipeline/pipeline.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pipeline.c). This file implements the main driver that instantiates a `cbm_pipeline_t` object, initializes the graph database connection, and manages the `cbm_pipeline_ctx_t` context structure. The context tracks the current execution phase and maintains state across passes, allowing the pipeline to resume from checkpoints if interrupted.

The controller reads configuration from [`pipeline.yaml`](https://github.com/DeusData/codebase-memory-mcp/blob/main/pipeline.yaml) to set worker counts, timeout thresholds, and verbosity levels. It registers passes via `cbm_pipeline_register_pass` and executes them in order, propagating error codes upward and aborting immediately if any pass returns a non-zero status, thereby ensuring the database remains consistent through transaction boundaries.

### Worker Pool Concurrency

Parallel execution is handled by [`src/pipeline/worker_pool.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/worker_pool.c), which creates a fixed-size thread pool according to the `workers` configuration parameter. Work items are encapsulated in `struct cbm_work_item` objects containing function pointers and payloads (typically file paths or analysis units). The pool distributes tasks across threads, balancing CPU utilization while preventing resource exhaustion. When a pass completes, the pool handles graceful shutdown, ensuring all pending work items finish before the next pass begins.

### Pass Registry

The [`src/pipeline/registry.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/registry.c) file maintains the pass registry, a static array of `cbm_pipeline_pass_t` structures. Each entry stores the pass name along with pointers to its `init`, `run`, and `finalize` callbacks, plus optional metadata declaring required database tables. This registration system enables the pipeline to iterate over passes generically without hardcoding pass-specific logic, supporting the addition of new analysis passes without modifying the core controller.

## The Pass Lifecycle and Interface

Every pass implements a uniform interface defined by the `cbm_pipeline_pass_t` struct:

```c
typedef struct {
    const char *name;
    int (*init)(cbm_pipeline_ctx_t *ctx);
    int (*run)(cbm_pipeline_ctx_t *ctx);
    int (*finalize)(cbm_pipeline_ctx_t *ctx);
} cbm_pipeline_pass_t;

```

The pipeline driver invokes these callbacks in three phases for each registered pass:

1. **Initialize** – Allocates pass-specific data structures and prepares database tables.
2. **Run** – Processes files, queries existing graph data, and produces new nodes or edges.
3. **Finalize** – Flushes batched writes to the graph buffer, updates checkpoint files, and releases resources.

This lifecycle ensures that each pass operates on a consistent view of the graph produced by previous passes while maintaining isolation from subsequent stages.

## Built-in Analysis Passes

MCP ships with specialized passes located in `src/pipeline/`, each handling a distinct aspect of codebase analysis:

### Symbol Extraction ([`src/pipeline/pass_definitions.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pass_definitions.c))

Parses source files to extract **definitions**—functions, classes, constants, and other symbols—and creates corresponding definition nodes in the graph database. This pass establishes the foundational symbol table that downstream passes reference.

### Usage Linking ([`src/pipeline/pass_usages.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pass_usages.c))

Links **usages** to the definitions discovered in the previous pass, creating edges that represent reference relationships. This pass queries the symbol table to resolve identifiers and establish dependency graphs.

### Semantic Analysis ([`src/pipeline/pass_semantic.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pass_semantic.c))

Generates **semantic** relationships such as inheritance hierarchies, interface implementations, and type dependencies using language-specific parsers. These edges enable deep code understanding beyond simple textual references.

### Similarity Detection ([`src/pipeline/pass_similarity.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pass_similarity.c))

Computes **similarity** hashes (MinHash or SimHash) for each file to enable near-duplicate detection and code cloning analysis. This pass writes hash properties to file nodes for later querying.

### Cross-Repository Analysis ([`src/pipeline/pass_cross_repo.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pass_cross_repo.c))

Detects **cross-repo** references, such as imports from external repositories, and creates edges linking local code to external dependencies. This supports monorepo and multi-repo analysis scenarios.

### Git History Integration ([`src/pipeline/pass_githistory.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pass_githistory.c))

Walks the Git history to add **revision** metadata and temporal edges, connecting code entities to their commit history. This pass integrates with [`src/git/git_context.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/git/git_context.c) to extract commit-level information.

### Environment Scanning ([`src/pipeline/pass_envscan.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pass_envscan.c))

Scans for **environment** files including Dockerfiles, `.env` files, and configuration manifests, recording them as artifact nodes. This captures deployment context and infrastructure dependencies.

### Complexity Metrics ([`src/pipeline/pass_complexity.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pass_complexity.c))

Calculates **code complexity** metrics (such as cyclomatic complexity) and stores them as node properties, enabling quality analysis and hot-spot identification.

### Configuration Linking ([`src/pipeline/pass_configlink.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pass_configlink.c))

Resolves configuration linkages, such as mapping `Makefile` targets to the source files they build, creating edges between build artifacts and their inputs.

## Incremental vs. Full Indexing

The pipeline supports two operating modes controlled by [`pipeline_incremental.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/pipeline_incremental.c) ([`src/pipeline/pipeline_incremental.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pipeline_incremental.c)):

- **Full Mode (`CBM_MODE_FULL`)** – Executes all passes and rebuilds the database from scratch, suitable for initial indexing or when the database schema changes.
- **Incremental Mode (`CBM_MODE_INCREMENTAL`)** – Tracks file timestamps and content hashes to determine which files have changed since the last run. Only the *definition*, *usage*, and downstream passes re-execute for modified files, dramatically reducing indexing time for large repositories with small deltas.

The incremental system writes checkpoint files after each successful pass, allowing the pipeline to resume from the last valid state if interrupted.

## Configuration and Operation

The pipeline is configured via [`pipeline.yaml`](https://github.com/DeusData/codebase-memory-mcp/blob/main/pipeline.yaml):

```yaml
pipeline:
  workers: 8          # Number of threads for parallel processing

  verbose: true       # Enable detailed logging

  timeout: 300        # Per-pass timeout in seconds

```

Programmatically, clients instantiate and run the pipeline through the API exposed in [`src/cli/cli.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/cli/cli.c):

```c
/* Create a pipeline for a project */
cbm_pipeline_t *pipeline = cbm_pipeline_new(
    "/path/to/repo",         /* repository root */
    "/tmp/cbm-db.sqlite3",   /* SQLite database path */
    CBM_MODE_FULL            /* indexing mode */
);

/* Execute all registered passes */
int rc = cbm_pipeline_run(pipeline);
if (rc != 0) {
    fprintf(stderr, "Pipeline failed with code %d\n", rc);
}

/* Retrieve indexed project name */
const char *proj_name = cbm_pipeline_project_name(pipeline);

/* Cleanup */
cbm_pipeline_free(pipeline);

```

For debugging or custom workflows, individual passes can be executed manually:

```c
cbm_pipeline_ctx_t ctx = { .pipeline = pipeline };

cbm_pipeline_pass_t *def_pass = cbm_pipeline_find_pass(pipeline, "definitions");
def_pass->init(&ctx);
def_pass->run(&ctx);
def_pass->finalize(&ctx);

```

## Summary

- **Modular Design**: The `cbm_pipeline_pass_t` interface in [`src/pipeline/registry.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/registry.c) allows passes to be developed independently and registered without modifying core logic.
- **Parallel Execution**: [`src/pipeline/worker_pool.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/worker_pool.c) provides thread-pool concurrency within each pass, maximizing CPU utilization during file processing.
- **Fault Tolerance**: Checkpointing in [`src/pipeline/pipeline_incremental.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pipeline_incremental.c) and transaction boundaries ensure that failures in one pass do not corrupt the entire index.
- **Incremental Support**: The pipeline distinguishes between `CBM_MODE_FULL` and `CBM_MODE_INCREMENTAL`, reprocessing only changed files to minimize runtime.
- **Rich Analysis**: Nine specialized passes (from [`pass_definitions.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/pass_definitions.c) to [`pass_configlink.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/pass_configlink.c)) extract definitions, usages, semantics, similarity, history, and environment data into a unified graph.

## Frequently Asked Questions

### How does the multi-pass architecture improve scalability?

The pipeline achieves scalability through **worker pool parallelism** implemented in [`src/pipeline/worker_pool.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/worker_pool.c), which distributes file processing across a configurable number of threads. Additionally, passes are **I/O-bound separately**, allowing the database to batch writes in [`src/graph_buffer/graph_buffer.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/graph_buffer/graph_buffer.c) while workers parse files concurrently. This design prevents any single pass from becoming a bottleneck and enables horizontal scaling across CPU cores.

### What happens when a pass fails during indexing?

If a pass returns a non-zero error code from its `run` callback, the pipeline controller in [`src/pipeline/pipeline.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pipeline.c) immediately aborts execution and propagates the error to the caller. Because each pass maintains its own transaction boundaries and writes checkpoint files via [`pipeline_incremental.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/pipeline_incremental.c), the database remains consistent—the failed pass's changes are discarded, and the system can resume from the last successful pass's checkpoint upon restart.

### Can I run a single pass without executing the entire pipeline?

Yes. The registry API exposes `cbm_pipeline_find_pass`, allowing clients to retrieve a specific `cbm_pipeline_pass_t` and invoke its `init`, `run`, and `finalize` methods manually. This is useful for debugging individual analysis stages or updating only specific graph aspects (such as re-running [`pass_similarity.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/pass_similarity.c) after tuning hash algorithms) without rebuilding the entire index.

### How does incremental indexing determine which files to reprocess?

The incremental logic in [`src/pipeline/pipeline_incremental.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pipeline_incremental.c) maintains a manifest of previously indexed files, recording content hashes and modification timestamps. When running in `CBM_MODE_INCREMENTAL`, the pipeline compares the current filesystem state against this manifest. Only files with changed hashes trigger re-execution of the **definition**, **usage**, and dependent passes, while unchanged files are skipped, significantly reducing indexing time for large repositories.