# Understanding the codebase-memory-mcp Indexing Pipeline: From Source Code to Knowledge Graph

> Learn how the codebase-memory-mcp indexing pipeline transforms source code into a queryable knowledge graph using Tree-sitter and Hybrid-LSP for in-depth code analysis.

- Repository: [Martin Vogel/codebase-memory-mcp](https://github.com/DeusData/codebase-memory-mcp)
- Tags: internals
- Published: 2026-07-15

---

**The `index_repository` pipeline transforms a source code checkout into a persistent, queryable knowledge graph by parsing files with Tree-sitter, resolving types via Hybrid-LSP, and storing the resulting graph structure in SQLite.**

The `codebase-memory-mcp` project provides a Model Context Protocol (MCP) server that enables AI assistants to query codebases with semantic understanding. At its heart lies the **codebase-memory-mcp indexing pipeline**, a multi-stage C implementation that converts raw repositories into structured graph databases that MCP tools can interrogate.

## What the codebase-memory-mcp Indexing Pipeline Does

The primary function of the pipeline is to bridge the gap between text files and machine-queryable code intelligence. According to the DeusData/codebase-memory-mcp source code, the `index_repository` tool orchestrates a series of transformations that extract functions, classes, call graphs, routes, and Architecture Decision Records (ADRs) from source code. It then persists these as nodes and edges in an on-disk SQLite database, creating a structural graph that supports precise semantic queries.

## Multi-Pass Architecture of the codebase-memory-mcp Indexing Pipeline

The pipeline implemented in [`src/pipeline/pipeline.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pipeline.c) follows a multi-pass strategy where each phase refines the graph with specific semantic information before writing to storage.

### Discovery and Definition Extraction

The process begins with file discovery and symbol extraction. The [`src/pipeline/pass_definitions.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pass_definitions.c) module uses vendored Tree-sitter grammars to parse source files and identify functions, classes, and other definable symbols, creating the initial nodes in the knowledge graph.

### Call Graph Construction

Once definitions are established, [`src/pipeline/pass_calls.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pass_calls.c) traverses the abstract syntax trees to build raw `CALLS` edges between symbols, mapping the invocation relationships within the codebase.

### Hybrid-LSP Type Resolution

The pipeline enhances call accuracy through [`src/pipeline/pass_lsp_cross.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pass_lsp_cross.c), which implements the Hybrid-LSP type-resolution layer. This pass resolves cross-file and cross-module references that Tree-sitter alone cannot determine, linking method calls to their actual definitions across package boundaries.

### Package and Import Resolution

For dependency mapping, [`src/pipeline/pass_pkgmap.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pass_pkgmap.c) processes package manager manifests such as [`package.json`](https://github.com/DeusData/codebase-memory-mcp/blob/main/package.json), `go.mod`, and [`requirements.txt`](https://github.com/DeusData/codebase-memory-mcp/blob/main/requirements.txt), integrating external dependency information into the graph structure.

### Infrastructure Scanning

The pipeline also indexes deployment configurations through [`src/pipeline/pass_infrascan.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pass_infrascan.c), which parses infrastructure-as-code files including Dockerfiles and Kubernetes manifests, ensuring the knowledge graph encompasses operational context.

## Running the codebase-memory-mcp Indexing Pipeline

Users invoke the pipeline through the `index_repository` RPC tool, handled by [`src/mcp/index_supervisor.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/mcp/index_supervisor.c). This module validates parameters, executes the orchestration logic, and reports status to the MCP client.

Choose between incremental and full indexing modes:

```bash

# Fast, incremental index of the current project

codebase-memory-mcp cli index_repository '{"repo_path":"." ,"mode":"fast"}'

# Full (slow) index – useful for a fresh database

codebase-memory-mcp cli index_repository '{"repo_path":"/path/to/project","mode":"full"}'

```

The **fast mode** performs incremental updates by processing only changed files, while **full mode** rebuilds the entire graph from scratch, useful for initializing a new database or recovering from corruption.

## Data Persistence and Storage

The final stage writes the complete knowledge graph to disk via [`src/store/store.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/store/store.c). This module serializes nodes and edges into SQLite databases stored in `~/.cache/codebase-memory-mcp/`, providing ACID-compliant storage that MCP tools query via SQL for semantic code search and navigation.

## Summary

- The **codebase-memory-mcp indexing pipeline** transforms raw source code into a queryable knowledge graph through the `index_repository` entry point.
- The multi-pass architecture in [`src/pipeline/pipeline.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pipeline.c) sequentially extracts definitions, builds call graphs, resolves types via Hybrid-LSP, and maps package dependencies.
- Each specialized pass is implemented in dedicated C files: [`pass_definitions.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/pass_definitions.c), [`pass_calls.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/pass_calls.c), [`pass_lsp_cross.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/pass_lsp_cross.c), [`pass_pkgmap.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/pass_pkgmap.c), and [`pass_infrascan.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/pass_infrascan.c).
- Results persist to SQLite via [`src/store/store.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/store/store.c), enabling fast semantic queries by MCP clients.
- Users trigger indexing via [`src/mcp/index_supervisor.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/mcp/index_supervisor.c) with either fast incremental or full rebuild modes.

## Frequently Asked Questions

### What data formats does the codebase-memory-mcp indexing pipeline support?

The pipeline uses vendored Tree-sitter grammars to parse most mainstream programming languages. Additionally, [`src/pipeline/pass_pkgmap.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pass_pkgmap.c) handles package manifests like [`package.json`](https://github.com/DeusData/codebase-memory-mcp/blob/main/package.json) and `go.mod`, while [`src/pipeline/pass_infrascan.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pass_infrascan.c) processes Dockerfiles and Kubernetes YAML files, ensuring comprehensive coverage of both application code and deployment configuration.

### Where does the codebase-memory-mcp indexing pipeline store its data?

According to the source implementation in [`src/store/store.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/store/store.c), the pipeline persists the knowledge graph to SQLite databases located in `~/.cache/codebase-memory-mcp/`. This provides durable, file-based storage that survives between MCP server restarts while maintaining transactional integrity during the indexing process.

### How does the Hybrid-LSP resolution improve the indexing pipeline?

The [`src/pipeline/pass_lsp_cross.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/pipeline/pass_lsp_cross.c) module implements Hybrid-LSP type-resolution, which refines raw call edges by resolving cross-file and cross-module symbol references that Tree-sitter parsing alone cannot determine. This enables accurate call graph construction across package boundaries, distinguishing between identically named methods in different files.

### Can I run the indexing pipeline incrementally?

Yes. The `index_repository` tool supports a `"mode":"fast"` parameter for incremental updates that only processes changed files, while `"mode":"full"` triggers a complete rebuild via the orchestration logic in [`src/mcp/index_supervisor.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/mcp/index_supervisor.c). Fast mode is ideal for continuous integration workflows, whereas full mode ensures database consistency after significant structural changes.