# Understanding the Codebase-Memory-MCP Indexing Pipeline Architecture: A Multi-Pass Breakdown

> Explore the linear seven-pass indexing pipeline architecture of Codebase-Memory-MCP. Discover how it transforms raw source files into a deduplicated 3D graph for deterministic parallelizable code analysis.

- Repository: [Martin Vogel/codebase-memory-mcp](https://github.com/DeusData/codebase-memory-mcp)
- Tags: architecture
- Published: 2026-07-30

---

**The Codebase-Memory-MCP indexer uses a linear, seven-pass pipeline written in C that transforms raw source files into a deduplicated 3D graph structure without mutating previous results, enabling deterministic and parallelizable code analysis.**

The **indexing pipeline architecture** of Codebase-Memory-MCP is designed to build a searchable, three-dimensional graph of any codebase. Implemented primarily in the C backend under `internal/cbm/`, this multi-pass system extracts definitions, references, calls, and dependencies before computing spatial coordinates for visualization. Each pass enriches the data model while treating previous outputs as read-only, ensuring stability and enabling parallel execution.

## Overview of the Multi-Pass Architecture

The pipeline follows a **linear data-flow design** where seven specialized passes progressively build a comprehensive code graph. This architecture separates concerns between lexical analysis, semantic extraction, graph unification, and spatial layout.

### Stage 1: File Discovery and Parsing

The process begins in [`internal/cbm/cbm.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/internal/cbm/cbm.c), the main driver that walks the target repository to collect every file matching supported grammars. Language-specific parsers generated from `grammar_*.c` files (supporting C, Rust, Python, and others) tokenize source files and produce AST fragments for downstream processing.

### Stage 2: The Seven Extraction Passes

Each pass focuses on a specific semantic layer, outputting intermediate structures that culminate in a unified graph:

- **Definitions**: Functions, classes, structs, and variables
- **Type References**: Cross-file type name occurrences
- **Call Sites**: Function invocation edges
- **Imports**: Module-level dependencies (`import`, `#include`, etc.)
- **Environment Accesses**: Runtime resource usage (env vars, config files)
- **Graph Unification**: Deduplication and merging
- **3D Layout**: Spatial coordinate assignment

## Deep Dive into the Seven Pipeline Passes

### Pass 1 – Definition Extraction (extract_defs.c)

The first pass processes AST fragments to identify top-level symbols. In [`internal/cbm/extract_defs.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/internal/cbm/extract_defs.c), the system generates **Def** objects that become the primary nodes in the graph. Each definition captures metadata including file paths, line numbers, and qualified names, establishing the foundational node list referenced by all subsequent passes.

### Pass 2 – Type-Reference Extraction (extract_type_refs.c)

Once definitions exist, [`internal/cbm/extract_type_refs.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/internal/cbm/extract_type_refs.c) locates every occurrence of type names across the codebase. This pass creates **TypeRef** objects that map usage sites to their declarations, enabling cross-file type resolution without modifying the definition nodes created in Pass 1.

### Pass 3 – Call-Site Extraction (extract_calls.c)

The third pass builds the functional dependency graph. [`internal/cbm/extract_calls.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/internal/cbm/extract_calls.c) traverses the AST to record call expressions, generating **Call** edges that link caller and callee IDs. These edges represent the runtime execution flow and module coupling within the codebase.

### Pass 4 – Import and Include Extraction (extract_imports.c)

Module dependencies enter the graph in [`internal/cbm/extract_imports.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/internal/cbm/extract_imports.c). This pass detects `import`, `require`, `#include`, and similar statements, creating **Import** edges that represent the dependency graph between compilation units or modules.

### Pass 5 – Environment-Access Extraction (extract_env_accesses.c)

Runtime behavior analysis occurs in [`internal/cbm/extract_env_accesses.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/internal/cbm/extract_env_accesses.c). This specialized pass captures accesses to environment variables, configuration files, and external resources, producing **EnvAccess** annotations that highlight operational dependencies invisible to static type analysis.

### Pass 6 – Unified Graph Construction (extract_unified.c)

Aggregation happens in [`internal/cbm/extract_unified.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/internal/cbm/extract_unified.c), where intermediate structures from passes 1-5 merge into a single deduplicated graph. This pass resolves duplicates (e.g., a function both defined and called) and produces the `GraphData` structure consumed by the UI. The unification step ensures that nodes referenced across multiple passes receive stable IDs before layout computation.

### Pass 7 – 3D Layout Computation (layout3d.c)

The final pass transforms the unified graph into a spatial visualization. [`internal/cbm/layout3d.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/internal/cbm/layout3d.c) computes **x, y, z** coordinates for each node, assigns clustering information, and determines **NodeStatus** values (such as `dead`, `entry`, or `test`). These status values map to colors in the UI, creating the three-dimensional code landscape that distinguishes node types and code health.

## Data Model and JSON Contract

The UI consumes a strictly typed JSON output defined in [`graph-ui/src/lib/types.ts`](https://github.com/DeusData/codebase-memory-mcp/blob/main/graph-ui/src/lib/types.ts). The interface contract requires specific fields for nodes, edges, and metadata:

```typescript
export interface GraphNode {
  id: number;
  x: number;
  y: number;
  z: number;
  label: string;
  name: string;
  file_path?: string;
  qualified_name?: string;
  start_line?: number;
  end_line?: number;
  size: number;
  color: string;
  status?: NodeStatus;
  in_calls?: number;
}

export interface GraphEdge {
  source: number;
  target: number;
  type: string;
}

export interface GraphData {
  nodes: GraphNode[];
  edges: GraphEdge[];
  total_nodes: number;
  linked_projects?: LinkedProject[];
  missed_graph?: MissedGraph;
}

```

The `MissedGraph` structure captures files the indexer could not fully parse, appearing as a "satellite cluster" in the visualization. This separation ensures partial indexing results remain visible rather than being silently dropped.

## Running the Indexing Pipeline

To execute the multi-pass pipeline on a local repository:

```bash

# Install dependencies including tree-sitter grammars

./install.sh

# Run the full pipeline (outputs layout3d.json)

./internal/cbm/cbm /path/to/target/repo

# Copy output to UI data directory

cp layout3d.json graph-ui/public/data.json

# Launch the visualization interface

cd graph-ui
npm run dev

```

The [`cbm.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/cbm.c) driver orchestrates all seven passes sequentially, writing the final JSON payload that [`graph-ui/src/hooks/useGraphData.ts`](https://github.com/DeusData/codebase-memory-mcp/blob/main/graph-ui/src/hooks/useGraphData.ts) consumes to populate the React frontend.

## Summary

- The **Codebase-Memory-MCP** indexing pipeline uses a **seven-pass, linear architecture** implemented in C under `internal/cbm/`.
- Each pass treats previous results as read-only, enabling **deterministic output** and **parallel execution** per file.
- The pipeline progresses from **lexical extraction** through **semantic analysis** to **spatial layout**, outputting JSON matching the TypeScript interfaces in [`graph-ui/src/lib/types.ts`](https://github.com/DeusData/codebase-memory-mcp/blob/main/graph-ui/src/lib/types.ts).
- **Graph unification** in [`extract_unified.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/extract_unified.c) deduplicates entities before **3D coordinate assignment** in [`layout3d.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/layout3d.c).
- The UI renders the complete graph while displaying **MissedGraph** data for unparseable files.

## Frequently Asked Questions

### What makes the Codebase-Memory-MCP pipeline architecture deterministic?

The read-only dependency chain between passes ensures determinism. Because [`extract_type_refs.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/extract_type_refs.c), [`extract_calls.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/extract_calls.c), and subsequent passes only consume the stable node IDs generated by [`extract_defs.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/extract_defs.c) without mutating them, the system produces identical graph outputs for identical inputs regardless of execution order or parallelization.

### Can the indexing pipeline run in parallel for large codebases?

Yes. While the seven passes execute sequentially in the current [`cbm.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/cbm.c) driver, each pass can process individual files in parallel because they treat previous pass outputs as immutable. The unification pass ([`extract_unified.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/extract_unified.c)) handles deduplication after parallel file processing completes, making the architecture horizontally scalable.

### Why are environment variable accesses tracked separately from imports?

Environment accesses represent **runtime operational dependencies** rather than **compile-time module dependencies**. [`extract_env_accesses.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/extract_env_accesses.c) captures dynamic resource usage (like `process.env` or `std::env`) that static import analysis misses. This separation allows the UI to distinguish between structural dependencies (imports) and operational coupling (env vars), critical for understanding deployment and security implications.

### How does the UI handle files that fail parsing during the indexing pipeline?

Files that cannot be fully parsed populate the **MissedGraph** object defined in [`graph-ui/src/lib/types.ts`](https://github.com/DeusData/codebase-memory-mcp/blob/main/graph-ui/src/lib/types.ts). Rather than failing the entire pipeline, [`extract_unified.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/extract_unified.c) isolates these files into a satellite cluster with separate metadata. The React frontend renders this as a distinct visual cluster, alerting users to coverage gaps while preserving the partial graph of successfully indexed files.