Multi-Pass Indexing Pipeline Architecture in Codebase-Memory-MCP

The Codebase-Memory-MCP multi-pass indexing pipeline is a C-based, staged processing system that transforms source code into an SQLite-backed graph database through parallelized, independent passes—each handling specific analysis tasks like symbol extraction, usage linking, and similarity detection—while supporting incremental updates and fault tolerance via checkpointing.

The multi-pass indexing pipeline in Codebase-Memory-MCP (MCP) processes repositories into a rich, queryable graph structure by breaking analysis into discrete, composable stages. Implemented in C and organized under src/pipeline/, this architecture separates concerns into specialized passes that execute sequentially yet leverage worker pools for parallel file processing, enabling scalable indexing of large monorepos without compromising robustness or data integrity.

Core Architecture Components

Pipeline Controller

The central orchestrator resides in src/pipeline/pipeline.c. This file implements the main driver that instantiates a cbm_pipeline_t object, initializes the graph database connection, and manages the cbm_pipeline_ctx_t context structure. The context tracks the current execution phase and maintains state across passes, allowing the pipeline to resume from checkpoints if interrupted.

The controller reads configuration from pipeline.yaml to set worker counts, timeout thresholds, and verbosity levels. It registers passes via cbm_pipeline_register_pass and executes them in order, propagating error codes upward and aborting immediately if any pass returns a non-zero status, thereby ensuring the database remains consistent through transaction boundaries.

Worker Pool Concurrency

Parallel execution is handled by src/pipeline/worker_pool.c, which creates a fixed-size thread pool according to the workers configuration parameter. Work items are encapsulated in struct cbm_work_item objects containing function pointers and payloads (typically file paths or analysis units). The pool distributes tasks across threads, balancing CPU utilization while preventing resource exhaustion. When a pass completes, the pool handles graceful shutdown, ensuring all pending work items finish before the next pass begins.

Pass Registry

The src/pipeline/registry.c file maintains the pass registry, a static array of cbm_pipeline_pass_t structures. Each entry stores the pass name along with pointers to its init, run, and finalize callbacks, plus optional metadata declaring required database tables. This registration system enables the pipeline to iterate over passes generically without hardcoding pass-specific logic, supporting the addition of new analysis passes without modifying the core controller.

The Pass Lifecycle and Interface

Every pass implements a uniform interface defined by the cbm_pipeline_pass_t struct:

typedef struct {
    const char *name;
    int (*init)(cbm_pipeline_ctx_t *ctx);
    int (*run)(cbm_pipeline_ctx_t *ctx);
    int (*finalize)(cbm_pipeline_ctx_t *ctx);
} cbm_pipeline_pass_t;

The pipeline driver invokes these callbacks in three phases for each registered pass:

1. Initialize – Allocates pass-specific data structures and prepares database tables.
2. Run – Processes files, queries existing graph data, and produces new nodes or edges.
3. Finalize – Flushes batched writes to the graph buffer, updates checkpoint files, and releases resources.

This lifecycle ensures that each pass operates on a consistent view of the graph produced by previous passes while maintaining isolation from subsequent stages.

Built-in Analysis Passes

MCP ships with specialized passes located in src/pipeline/, each handling a distinct aspect of codebase analysis:

Symbol Extraction (src/pipeline/pass_definitions.c)

Parses source files to extract definitions—functions, classes, constants, and other symbols—and creates corresponding definition nodes in the graph database. This pass establishes the foundational symbol table that downstream passes reference.

Usage Linking (src/pipeline/pass_usages.c)

Links usages to the definitions discovered in the previous pass, creating edges that represent reference relationships. This pass queries the symbol table to resolve identifiers and establish dependency graphs.

Semantic Analysis (src/pipeline/pass_semantic.c)

Generates semantic relationships such as inheritance hierarchies, interface implementations, and type dependencies using language-specific parsers. These edges enable deep code understanding beyond simple textual references.

Similarity Detection (src/pipeline/pass_similarity.c)

Computes similarity hashes (MinHash or SimHash) for each file to enable near-duplicate detection and code cloning analysis. This pass writes hash properties to file nodes for later querying.

Cross-Repository Analysis (src/pipeline/pass_cross_repo.c)

Detects cross-repo references, such as imports from external repositories, and creates edges linking local code to external dependencies. This supports monorepo and multi-repo analysis scenarios.

Git History Integration (src/pipeline/pass_githistory.c)

Walks the Git history to add revision metadata and temporal edges, connecting code entities to their commit history. This pass integrates with src/git/git_context.c to extract commit-level information.

Environment Scanning (src/pipeline/pass_envscan.c)

Scans for environment files including Dockerfiles, .env files, and configuration manifests, recording them as artifact nodes. This captures deployment context and infrastructure dependencies.

Complexity Metrics (src/pipeline/pass_complexity.c)

Calculates code complexity metrics (such as cyclomatic complexity) and stores them as node properties, enabling quality analysis and hot-spot identification.

Configuration Linking (src/pipeline/pass_configlink.c)

Resolves configuration linkages, such as mapping Makefile targets to the source files they build, creating edges between build artifacts and their inputs.

Incremental vs. Full Indexing

The pipeline supports two operating modes controlled by pipeline_incremental.c (src/pipeline/pipeline_incremental.c):

• Full Mode (CBM_MODE_FULL) – Executes all passes and rebuilds the database from scratch, suitable for initial indexing or when the database schema changes.
• Incremental Mode (CBM_MODE_INCREMENTAL) – Tracks file timestamps and content hashes to determine which files have changed since the last run. Only the definition, usage, and downstream passes re-execute for modified files, dramatically reducing indexing time for large repositories with small deltas.

The incremental system writes checkpoint files after each successful pass, allowing the pipeline to resume from the last valid state if interrupted.

Configuration and Operation

The pipeline is configured via pipeline.yaml:

pipeline:
  workers: 8          # Number of threads for parallel processing

  verbose: true       # Enable detailed logging

  timeout: 300        # Per-pass timeout in seconds

Programmatically, clients instantiate and run the pipeline through the API exposed in src/cli/cli.c:

/* Create a pipeline for a project */
cbm_pipeline_t *pipeline = cbm_pipeline_new(
    "/path/to/repo",         /* repository root */
    "/tmp/cbm-db.sqlite3",   /* SQLite database path */
    CBM_MODE_FULL            /* indexing mode */
);

/* Execute all registered passes */
int rc = cbm_pipeline_run(pipeline);
if (rc != 0) {
    fprintf(stderr, "Pipeline failed with code %d\n", rc);
}

/* Retrieve indexed project name */
const char *proj_name = cbm_pipeline_project_name(pipeline);

/* Cleanup */
cbm_pipeline_free(pipeline);

For debugging or custom workflows, individual passes can be executed manually:

cbm_pipeline_ctx_t ctx = { .pipeline = pipeline };

cbm_pipeline_pass_t *def_pass = cbm_pipeline_find_pass(pipeline, "definitions");
def_pass->init(&ctx);
def_pass->run(&ctx);
def_pass->finalize(&ctx);

Summary

• Modular Design: The cbm_pipeline_pass_t interface in src/pipeline/registry.c allows passes to be developed independently and registered without modifying core logic.
• Parallel Execution: src/pipeline/worker_pool.c provides thread-pool concurrency within each pass, maximizing CPU utilization during file processing.
• Fault Tolerance: Checkpointing in src/pipeline/pipeline_incremental.c and transaction boundaries ensure that failures in one pass do not corrupt the entire index.
• Incremental Support: The pipeline distinguishes between CBM_MODE_FULL and CBM_MODE_INCREMENTAL, reprocessing only changed files to minimize runtime.
• Rich Analysis: Nine specialized passes (from pass_definitions.c to pass_configlink.c) extract definitions, usages, semantics, similarity, history, and environment data into a unified graph.

Frequently Asked Questions

How does the multi-pass architecture improve scalability?

The pipeline achieves scalability through worker pool parallelism implemented in src/pipeline/worker_pool.c, which distributes file processing across a configurable number of threads. Additionally, passes are I/O-bound separately, allowing the database to batch writes in src/graph_buffer/graph_buffer.c while workers parse files concurrently. This design prevents any single pass from becoming a bottleneck and enables horizontal scaling across CPU cores.

What happens when a pass fails during indexing?

If a pass returns a non-zero error code from its run callback, the pipeline controller in src/pipeline/pipeline.c immediately aborts execution and propagates the error to the caller. Because each pass maintains its own transaction boundaries and writes checkpoint files via pipeline_incremental.c, the database remains consistent—the failed pass's changes are discarded, and the system can resume from the last successful pass's checkpoint upon restart.

Can I run a single pass without executing the entire pipeline?

Yes. The registry API exposes cbm_pipeline_find_pass, allowing clients to retrieve a specific cbm_pipeline_pass_t and invoke its init, run, and finalize methods manually. This is useful for debugging individual analysis stages or updating only specific graph aspects (such as re-running pass_similarity.c after tuning hash algorithms) without rebuilding the entire index.

How does incremental indexing determine which files to reprocess?

The incremental logic in src/pipeline/pipeline_incremental.c maintains a manifest of previously indexed files, recording content hashes and modification timestamps. When running in CBM_MODE_INCREMENTAL, the pipeline compares the current filesystem state against this manifest. Only files with changed hashes trigger re-execution of the definition, usage, and dependent passes, while unchanged files are skipped, significantly reducing indexing time for large repositories.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}