How the Multi-Pass Indexing Pipeline Works in codebase-memory-mcp: A Deep Dive

TLDR: The multi-pass indexing pipeline in codebase-memory-mcp is a RAM-first, sequential processing system that transforms raw source trees into a rich knowledge graph through 17+ specialized passes, operating entirely on an in-memory buffer with LZ4-HC compression before persisting to SQLite.

The multi-pass indexing pipeline is the core engine of the DeusData/codebase-memory-mcp repository, responsible for converting source code into a queryable graph database. Unlike traditional single-pass analyzers, this architecture processes files multiple times—each pass focusing on a specific concern—while maintaining all intermediate data in RAM for maximum throughput. This design enables the system to extract definitions, resolve cross-file type dependencies, and detect semantic similarities without repeated disk I/O.

Pipeline Initialization and Global Locking

Before any processing begins, the pipeline acquires exclusive access to the target database. In src/pipeline/pipeline.c (lines 54–71), a spin-lock named g_pipeline_busy guarantees that only one pipeline run touches a given SQLite database at a time. This prevents corruption during concurrent operations.

The orchestration starts with cbm_pipeline_new() (lines 66–84), which allocates a cbm_pipeline_t structure, records the repository path, resolves Git context, and generates a project name via cbm_project_name_from_path(). Immediately after, the discovery phase in discover/discover.c walks the filesystem, respecting .gitignore, .cbmignore, and OS-specific exclusion rules to build a list of cbm_file_info_t structures.

The Sequential Pass Architecture

The multi-pass indexing pipeline follows a fixed execution order defined by the seq_passes table in src/pipeline/pipeline.c (lines 898–904). Each pass is a function pointer entry that operates on the same in-memory graph buffer, allowing later passes to consume nodes and edges created by earlier ones.

Structure Pass: Building the Hierarchy

The first pass, pass_structure (lines 47–78 in pipeline.c), establishes the skeleton of the knowledge graph. It creates high-level nodes for Project, Branch, Folder, and File, and inserts CONTAINS_* edges that mirror the repository's directory structure. This hierarchy serves as the foundation for all subsequent relationship mapping.

Definitions Pass: Extracting Symbols

Once the file structure is established, cbm_pipeline_pass_definitions in src/pipeline/pass_definitions.c (lines 4–9) parses each source file using Tree-sitter. This pass extracts definitions—functions, classes, methods, variables, and modules—and registers them in a central registry for later lookup. Each definition becomes a node in the graph buffer, tagged with line numbers and source spans.

Hybrid LSP Cross-File Resolution

Before the pipeline can resolve inter-file relationships, it runs cbm_pipeline_pass_lsp_cross from src/pipeline/pass_lsp_cross.c (line 785). This "Hybrid LSP" layer re-reads every source file to build type-aware resolution tables, handling imports, generics, inheritance, and other cross-file dependencies. Unlike standalone language servers, this pass operates offline using the same in-memory representation as the rest of the pipeline.

Relationship and Semantic Passes

With definitions and cross-file type tables available, the pipeline proceeds to map relationships:

• Calls Pass (cbm_pipeline_pass_calls): Consumes the definition registry and LSP tables to create CALLS edges, including macro-mediated dispatch.
• Usages Pass (cbm_pipeline_pass_usages in src/pipeline/pass_usages.c, lines 2–12): Analyzes variable read/write patterns to emit READS, WRITES, and USES_TYPE edges.
• Semantic Pass (cbm_pipeline_pass_semantic in src/pipeline/pass_semantic.c, lines 2–12): Generates high-level edges such as INHERITS, IMPLEMENTS, and DECORATES based on type information gathered by the LSP pass.
• Tests Pass (cbm_pipeline_pass_tests in src/pipeline/pass_tests.c, lines 2–10): Links test functions to the production code they exercise via TESTS edges.

Metadata, Similarity, and Configuration Passes

The final set of passes enriches the graph with auxiliary data:

• Decorator-Tags Pass (cbm_pipeline_pass_decorator_tags, called from pipeline.c line 752): Extracts language-specific decorators (e.g., Python @dataclass, TypeScript @Component) and stores them as node properties.
• Similarity Pass (cbm_pipeline_pass_similarity in src/pipeline/pass_similarity.c, lines 2–9): Computes MinHash fingerprints for each definition and creates SIMILAR_TO edges for near-duplicate detection.
• Semantic-Edges Pass (cbm_pipeline_pass_semantic_edges in src/pipeline/pass_semantic_edges.c, lines 2–5): Emits SEMANTICALLY_RELATED edges that combine structural similarity, type similarity, and MinHash results.
• Config-Link Pass (cbm_pipeline_pass_configlink in src/pipeline/pass_configlink.c, called from pipeline.c line 764): Connects configuration files (e.g., package.json, pyproject.toml) to the code entities they configure.
• Complexity Pass (cbm_pipeline_pass_complexity): Calculates cyclomatic complexity metrics and stores them as node properties.

Memory Management and Persistence

The entire multi-pass indexing pipeline operates on a pre-allocated memory budget controlled by CBM_MEM_BUDGET_MB. Files are read once, compressed using LZ4-HC, and stored in an in-memory SQLite buffer, giving later passes fast random access without hitting the disk again. This RAM-first approach ensures that even large codebases index quickly while maintaining bounded memory usage.

The pipeline is also interrupt-aware: a p->cancelled flag is checked between passes, allowing graceful abortion without corrupting the database. After the final pass completes, cbm_gbuf_dump() writes the in-memory graph buffer to a compressed SQLite file (graph.db.zst), and cbm_mem_release() frees the allocated memory.

Running the Pipeline

You can trigger the multi-pass indexing pipeline via the CLI or programmatically:

# Index a repository from the command line

codebase-memory-mcp cli index_repository '{"repo_path":"/path/to/my/project"}'

To invoke the pipeline programmatically in C:

#include "pipeline/pipeline.h"

cbm_pipeline_t *pl = cbm_pipeline_new(
    "/path/to/my/project",                               /* repo_path */
    "/home/user/.cache/codebase-memory-mcp/graph.db.zst", /* db_path */
    CBM_INDEX_MODE_FULL);                                 /* mode = full re-index */

cbm_pipeline_set_persistence(pl, true); /* write .codebase-memory artifact */

if (cbm_pipeline_try_lock()) {           /* acquire global lock */
    int rc = cbm_pipeline_run(pl);       /* runs all passes in order */
    cbm_pipeline_unlock();
    /* rc == 0 → success, non-zero → error/cancel */
}
cbm_pipeline_free(pl);

To query nodes generated by specific passes:

codebase-memory-mcp cli search_graph '{
  "project":"my-project",
  "label":"Function",
  "name_pattern":"^process_.*"
}'

Summary

• The multi-pass indexing pipeline in DeusData/codebase-memory-mcp processes source code through 17+ sequential passes, each specializing in a single concern like structure, definitions, or semantics.
• All passes operate on an in-memory graph buffer with LZ4-HC compression, ensuring fast random access without repeated disk I/O.
• Critical passes include pass_structure for hierarchy, cbm_pipeline_pass_definitions for symbol extraction, and cbm_pipeline_pass_lsp_cross for cross-file type resolution.
• The pipeline uses a global spin-lock (g_pipeline_busy) to prevent concurrent database corruption and supports graceful interruption via p->cancelled.
• Execution is defined by the seq_passes array in src/pipeline/pipeline.c, which specifies the exact order of transformation stages.
• Output persists as a compressed SQLite database (graph.db.zst) suitable for high-performance code intelligence queries.

Frequently Asked Questions

What is the order of passes in the codebase-memory indexing pipeline?

The pass order is hard-coded in the seq_passes array in src/pipeline/pipeline.c (lines 898–904). It begins with structure discovery, followed by definitions extraction, LSP cross-file resolution, calls mapping, usages analysis, semantic relationship detection, and finally similarity and complexity calculations. This sequential design ensures that each pass can consume graph nodes created by previous stages.

How does codebase-memory-mcp handle memory during indexing?

The pipeline operates on a RAM-first architecture with a configurable memory budget (CBM_MEM_BUDGET_MB). Source files are loaded once, compressed with LZ4-HC, and stored in an in-memory SQLite buffer. All passes read from and write to this same buffer, eliminating redundant disk access and keeping memory usage bounded even for large repositories.

What is the Hybrid LSP pass and why is it necessary?

The Hybrid LSP pass (cbm_pipeline_pass_lsp_cross in src/pipeline/pass_lsp_cross.c) performs offline type-aware resolution across files, handling imports, generics, and inheritance without requiring a running language server. It is necessary because it populates cross-file resolution tables that subsequent passes (like calls and semantic analysis) depend on to accurately map relationships between distant code entities.

Can the indexing pipeline be interrupted safely?

Yes. The pipeline checks a p->cancelled flag between each pass, allowing it to abort gracefully without corrupting the SQLite database. This design supports responsive user interfaces where indexing might need to stop immediately due to system changes or user input, while the global lock (g_pipeline_busy) ensures atomicity.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}