How the Multi-Pass Indexing Pipeline Works in Codebase-Memory MCP

The multi-pass indexing pipeline in Codebase-Memory MCP transforms repository source files into a queryable graph through seven distinct phases—from initial file discovery and LZ4 compression to symbol extraction and SQLite serialization—supporting both parallel and sequential execution modes with optional incremental updates.

The Codebase-Memory MCP multi-pass indexing pipeline is the core engine that converts raw source code into a rich, navigable graph database. Implemented primarily in src/pipeline/pipeline.c, this deterministic system processes repositories through discrete transformation stages, enabling powerful code intelligence queries across complex codebases.

The Seven Phases of the Multi-Pass Indexing Pipeline

The pipeline executes a deterministic sequence of transformations, each handled by specialized modules in the src/pipeline/ directory.

Phase 1: Discovery (File Walking and Ignore Rules)

The pipeline begins in discover/discover.c, which walks the repository tree, collects file metadata, and applies ignore rules to filter out irrelevant paths before any graph construction begins.

Phase 2: Structure (Hierarchy Node Creation)

The pass_structure() function (lines 4-11 in src/pipeline/pipeline.c) creates the foundational graph hierarchy. It instantiates Project, Folder, Package, and File nodes, establishing CONTAINS_* edges that mirror the filesystem structure.

Phase 3: Bulk Load (Compression and Storage)

During initialization inside cbm_pipeline_new(), the pipeline allocates the graph buffer (cbm_gbuf) and reads each source file. Content is compressed using LZ4 HC before storage, minimizing memory footprint while preserving full source access for downstream passes.

Phase 4: Definitions (Symbol Extraction)

The src/pipeline/pass_definitions.c module parses each file to extract top-level definitions including functions, types, and constants. These become distinct nodes in the graph, invoked from both sequential and parallel execution paths.

Phase 5: Import Resolution and Semantic Edges

This phase resolves cross-file relationships through three specialized passes:

• pass_calls.c: Resolves function and method calls, creating CALLS edges
• pass_usages.c: Tracks identifier usages via USES edges
• pass_semantic.c: Builds higher-level relationships like OVERRIDES and IMPLEMENTS

Phase 6: Post-Passes (Tests, Infrastructure, and Complexity)

Optional enrichment passes add domain-specific metadata:

• pass_tests.c: Detects test files and links them to implementation code
• pass_infrascan.c: Creates Route nodes for cloud infrastructure bindings
• pass_complexity.c: Calculates cyclomatic complexity metrics
• HTTP links and Git history integration

Phase 7: Dump (Serialization to SQLite)

The dump_and_persist_hashes() function (lines 89-121 in src/pipeline/pipeline.c) serializes the in-memory graph buffer to a SQLite database. It optionally persists file hashes to enable incremental re-indexing in subsequent runs.

Execution Modes: Parallel vs. Sequential

The pipeline supports two execution strategies controlled by the CBM_MODE_FULL flag and the CBM_INDEX_SINGLE_THREAD environment variable.

Sequential Mode: Executed via run_sequential_pipeline(), this path processes passes one-by-one on the calling thread. It builds a package map first, then executes definition extraction, K8s detection, LSP cross-file resolution, calls, usages, and semantic passes in strict order.

Parallel Mode: Invoked through run_parallel_pipeline(), this approach shards heavy computation across a worker pool implemented in src/pipeline/worker_pool.c. File extraction, registry building, and LSP cross-file resolution run concurrently. A shared ID counter (shared_ids) guarantees unique node IDs across threads. Both modes execute identical predump passes—including decorator tags, config-linking, and similarity analysis—via run_predump_passes() (lines 70-94).

Incremental Indexing for Large Codebases

For repositories with existing indices, try_incremental_or_delete_db() (lines 105-139) checks whether the current file set matches stored hashes. When the delta is small, cbm_pipeline_run_incremental() updates only changed files rather than rebuilding the entire graph. If the divergence is too large, the system deletes the old database and triggers a full re-index through the standard pipeline.

Configuration: Workers and Fast Mode

Worker count is determined by effective_worker_count(), which respects the CBM_INDEX_SINGLE_THREAD environment variable to force single-threaded execution.

Fast Mode (CBM_MODE_FAST) optimizes for CI environments by skipping moderate-cost passes—including similarity analysis and certain semantic edges—through modified run_predump_passes() logic.

Key Source Files and Architecture

File	Role
src/pipeline/pipeline.c	Orchestrates creation, mode selection, predump passes, and dumping
src/pipeline/pipeline.h	Public API: cbm_pipeline_new, cbm_pipeline_run, cbm_pipeline_set_persistence
src/pipeline/pipeline_incremental.c	Handles incremental re-indexing logic
src/pipeline/worker_pool.c	Thread pool implementation for parallel execution
src/pipeline/pass_structure.c	Builds folder/file hierarchy (inline in pipeline.c)
src/pipeline/pass_definitions.c	Extracts top-level symbols
src/pipeline/pass_calls.c	Resolves function and method calls
src/pipeline/pass_usages.c	Resolves identifier usages
src/pipeline/pass_semantic.c	Generates semantic edges
src/pipeline/pass_infrascan.c	Infrastructure route detection
src/pipeline/pass_tests.c	Test file linking
src/pipeline/pass_complexity.c	Complexity metrics calculation

Practical Implementation Examples

Create a full index with persistence:

cbm_pipeline_t *pl = cbm_pipeline_new("/path/to/repo", NULL, CBM_MODE_FULL);
cbm_pipeline_set_persistence(pl, true);
int rc = cbm_pipeline_run(pl);
if (rc == 0) {
    printf("Indexing succeeded – DB written to %s\n",
           cbm_pipeline_db_path(pl));
}
cbm_pipeline_free(pl);

Run a fast, single-threaded index for CI environments:

cbm_pipeline_t *pl = cbm_pipeline_new("/path/to/repo", "mygraph.db", CBM_MODE_FAST);
setenv("CBM_INDEX_SINGLE_THREAD", "1", 1);
int rc = cbm_pipeline_run(pl);
cbm_pipeline_free(pl);

Summary

• The multi-pass indexing pipeline processes repositories through seven deterministic phases from discovery to SQLite serialization
• Structure creation establishes the hierarchy via pass_structure() while definitions extraction populates symbol nodes in pass_definitions.c
• Parallel execution uses a worker pool with shared ID counters; sequential mode runs passes on the main thread via run_sequential_pipeline()
• Incremental indexing compares file hashes in try_incremental_or_delete_db() to avoid full rebuilds when only minor changes occur
• Fast mode (CBM_MODE_FAST) skips expensive semantic analysis for quicker CI feedback
• All operations are orchestrated through src/pipeline/pipeline.c with modular pass implementations in src/pipeline/pass_*.c files

Frequently Asked Questions

What is the difference between CBM_MODE_FULL and CBM_MODE_FAST?

CBM_MODE_FULL executes all seven phases including semantic edge generation and similarity analysis, producing the most complete graph. CBM_MODE_FAST skips moderate-cost passes like similarity calculations and certain semantic edges in run_predump_passes(), making it ideal for CI pipelines where speed matters more than deep semantic analysis.

How does the pipeline handle concurrent file processing?

When CBM_INDEX_SINGLE_THREAD is not set and the mode is full, run_parallel_pipeline() distributes file extraction and LSP resolution across a worker pool defined in src/pipeline/worker_pool.c. A shared_ids counter ensures unique node IDs across threads, while run_predump_passes() synchronizes the final graph enrichment steps before dumping to SQLite.

Can I resume an interrupted indexing operation?

While the pipeline does not support resuming mid-pass, it supports incremental indexing via cbm_pipeline_run_incremental() in src/pipeline/pipeline_incremental.c. If a previous database exists and file hashes match sufficiently, only changed files are re-indexed. Otherwise, the old database is deleted and a full re-index starts fresh.

Where are the graph files stored after indexing?

By default, the pipeline creates a .codebase-memory graph file in the repository root. You can specify a custom path via the second parameter to cbm_pipeline_new() and enable persistence with cbm_pipeline_set_persistence(). The final SQLite database path is accessible via cbm_pipeline_db_path().

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}