Understanding the Codebase-Memory-MCP Indexing Pipeline Architecture: A Multi-Pass Breakdown

The Codebase-Memory-MCP indexer uses a linear, seven-pass pipeline written in C that transforms raw source files into a deduplicated 3D graph structure without mutating previous results, enabling deterministic and parallelizable code analysis.

The indexing pipeline architecture of Codebase-Memory-MCP is designed to build a searchable, three-dimensional graph of any codebase. Implemented primarily in the C backend under internal/cbm/, this multi-pass system extracts definitions, references, calls, and dependencies before computing spatial coordinates for visualization. Each pass enriches the data model while treating previous outputs as read-only, ensuring stability and enabling parallel execution.

Overview of the Multi-Pass Architecture

The pipeline follows a linear data-flow design where seven specialized passes progressively build a comprehensive code graph. This architecture separates concerns between lexical analysis, semantic extraction, graph unification, and spatial layout.

Stage 1: File Discovery and Parsing

The process begins in internal/cbm/cbm.c, the main driver that walks the target repository to collect every file matching supported grammars. Language-specific parsers generated from grammar_*.c files (supporting C, Rust, Python, and others) tokenize source files and produce AST fragments for downstream processing.

Stage 2: The Seven Extraction Passes

Each pass focuses on a specific semantic layer, outputting intermediate structures that culminate in a unified graph:

• Definitions: Functions, classes, structs, and variables
• Type References: Cross-file type name occurrences
• Call Sites: Function invocation edges
• Imports: Module-level dependencies (import, #include, etc.)
• Environment Accesses: Runtime resource usage (env vars, config files)
• Graph Unification: Deduplication and merging
• 3D Layout: Spatial coordinate assignment

Deep Dive into the Seven Pipeline Passes

Pass 1 – Definition Extraction (extract_defs.c)

The first pass processes AST fragments to identify top-level symbols. In internal/cbm/extract_defs.c, the system generates Def objects that become the primary nodes in the graph. Each definition captures metadata including file paths, line numbers, and qualified names, establishing the foundational node list referenced by all subsequent passes.

Pass 2 – Type-Reference Extraction (extract_type_refs.c)

Once definitions exist, internal/cbm/extract_type_refs.c locates every occurrence of type names across the codebase. This pass creates TypeRef objects that map usage sites to their declarations, enabling cross-file type resolution without modifying the definition nodes created in Pass 1.

Pass 3 – Call-Site Extraction (extract_calls.c)

The third pass builds the functional dependency graph. internal/cbm/extract_calls.c traverses the AST to record call expressions, generating Call edges that link caller and callee IDs. These edges represent the runtime execution flow and module coupling within the codebase.

Pass 4 – Import and Include Extraction (extract_imports.c)

Module dependencies enter the graph in internal/cbm/extract_imports.c. This pass detects import, require, #include, and similar statements, creating Import edges that represent the dependency graph between compilation units or modules.

Pass 5 – Environment-Access Extraction (extract_env_accesses.c)

Runtime behavior analysis occurs in internal/cbm/extract_env_accesses.c. This specialized pass captures accesses to environment variables, configuration files, and external resources, producing EnvAccess annotations that highlight operational dependencies invisible to static type analysis.

Pass 6 – Unified Graph Construction (extract_unified.c)

Aggregation happens in internal/cbm/extract_unified.c, where intermediate structures from passes 1-5 merge into a single deduplicated graph. This pass resolves duplicates (e.g., a function both defined and called) and produces the GraphData structure consumed by the UI. The unification step ensures that nodes referenced across multiple passes receive stable IDs before layout computation.

Pass 7 – 3D Layout Computation (layout3d.c)

The final pass transforms the unified graph into a spatial visualization. internal/cbm/layout3d.c computes x, y, z coordinates for each node, assigns clustering information, and determines NodeStatus values (such as dead, entry, or test). These status values map to colors in the UI, creating the three-dimensional code landscape that distinguishes node types and code health.

Data Model and JSON Contract

The UI consumes a strictly typed JSON output defined in graph-ui/src/lib/types.ts. The interface contract requires specific fields for nodes, edges, and metadata:

export interface GraphNode {
  id: number;
  x: number;
  y: number;
  z: number;
  label: string;
  name: string;
  file_path?: string;
  qualified_name?: string;
  start_line?: number;
  end_line?: number;
  size: number;
  color: string;
  status?: NodeStatus;
  in_calls?: number;
}

export interface GraphEdge {
  source: number;
  target: number;
  type: string;
}

export interface GraphData {
  nodes: GraphNode[];
  edges: GraphEdge[];
  total_nodes: number;
  linked_projects?: LinkedProject[];
  missed_graph?: MissedGraph;
}

The MissedGraph structure captures files the indexer could not fully parse, appearing as a "satellite cluster" in the visualization. This separation ensures partial indexing results remain visible rather than being silently dropped.

Running the Indexing Pipeline

To execute the multi-pass pipeline on a local repository:

# Install dependencies including tree-sitter grammars

./install.sh

# Run the full pipeline (outputs layout3d.json)

./internal/cbm/cbm /path/to/target/repo

# Copy output to UI data directory

cp layout3d.json graph-ui/public/data.json

# Launch the visualization interface

cd graph-ui
npm run dev

The cbm.c driver orchestrates all seven passes sequentially, writing the final JSON payload that graph-ui/src/hooks/useGraphData.ts consumes to populate the React frontend.

Summary

• The Codebase-Memory-MCP indexing pipeline uses a seven-pass, linear architecture implemented in C under internal/cbm/.
• Each pass treats previous results as read-only, enabling deterministic output and parallel execution per file.
• The pipeline progresses from lexical extraction through semantic analysis to spatial layout, outputting JSON matching the TypeScript interfaces in graph-ui/src/lib/types.ts.
• Graph unification in extract_unified.c deduplicates entities before 3D coordinate assignment in layout3d.c.
• The UI renders the complete graph while displaying MissedGraph data for unparseable files.

Frequently Asked Questions

What makes the Codebase-Memory-MCP pipeline architecture deterministic?

The read-only dependency chain between passes ensures determinism. Because extract_type_refs.c, extract_calls.c, and subsequent passes only consume the stable node IDs generated by extract_defs.c without mutating them, the system produces identical graph outputs for identical inputs regardless of execution order or parallelization.

Can the indexing pipeline run in parallel for large codebases?

Yes. While the seven passes execute sequentially in the current cbm.c driver, each pass can process individual files in parallel because they treat previous pass outputs as immutable. The unification pass (extract_unified.c) handles deduplication after parallel file processing completes, making the architecture horizontally scalable.

Why are environment variable accesses tracked separately from imports?

Environment accesses represent runtime operational dependencies rather than compile-time module dependencies. extract_env_accesses.c captures dynamic resource usage (like process.env or std::env) that static import analysis misses. This separation allows the UI to distinguish between structural dependencies (imports) and operational coupling (env vars), critical for understanding deployment and security implications.

How does the UI handle files that fail parsing during the indexing pipeline?

Files that cannot be fully parsed populate the MissedGraph object defined in graph-ui/src/lib/types.ts. Rather than failing the entire pipeline, extract_unified.c isolates these files into a satellite cluster with separate metadata. The React frontend renders this as a distinct visual cluster, alerting users to coverage gaps while preserving the partial graph of successfully indexed files.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}