Understanding the codebase-memory-mcp Indexing Pipeline: From Source Code to Knowledge Graph

The index_repository pipeline transforms a source code checkout into a persistent, queryable knowledge graph by parsing files with Tree-sitter, resolving types via Hybrid-LSP, and storing the resulting graph structure in SQLite.

The codebase-memory-mcp project provides a Model Context Protocol (MCP) server that enables AI assistants to query codebases with semantic understanding. At its heart lies the codebase-memory-mcp indexing pipeline, a multi-stage C implementation that converts raw repositories into structured graph databases that MCP tools can interrogate.

What the codebase-memory-mcp Indexing Pipeline Does

The primary function of the pipeline is to bridge the gap between text files and machine-queryable code intelligence. According to the DeusData/codebase-memory-mcp source code, the index_repository tool orchestrates a series of transformations that extract functions, classes, call graphs, routes, and Architecture Decision Records (ADRs) from source code. It then persists these as nodes and edges in an on-disk SQLite database, creating a structural graph that supports precise semantic queries.

Multi-Pass Architecture of the codebase-memory-mcp Indexing Pipeline

The pipeline implemented in src/pipeline/pipeline.c follows a multi-pass strategy where each phase refines the graph with specific semantic information before writing to storage.

Discovery and Definition Extraction

The process begins with file discovery and symbol extraction. The src/pipeline/pass_definitions.c module uses vendored Tree-sitter grammars to parse source files and identify functions, classes, and other definable symbols, creating the initial nodes in the knowledge graph.

Call Graph Construction

Once definitions are established, src/pipeline/pass_calls.c traverses the abstract syntax trees to build raw CALLS edges between symbols, mapping the invocation relationships within the codebase.

Hybrid-LSP Type Resolution

The pipeline enhances call accuracy through src/pipeline/pass_lsp_cross.c, which implements the Hybrid-LSP type-resolution layer. This pass resolves cross-file and cross-module references that Tree-sitter alone cannot determine, linking method calls to their actual definitions across package boundaries.

Package and Import Resolution

For dependency mapping, src/pipeline/pass_pkgmap.c processes package manager manifests such as package.json, go.mod, and requirements.txt, integrating external dependency information into the graph structure.

Infrastructure Scanning

The pipeline also indexes deployment configurations through src/pipeline/pass_infrascan.c, which parses infrastructure-as-code files including Dockerfiles and Kubernetes manifests, ensuring the knowledge graph encompasses operational context.

Running the codebase-memory-mcp Indexing Pipeline

Users invoke the pipeline through the index_repository RPC tool, handled by src/mcp/index_supervisor.c. This module validates parameters, executes the orchestration logic, and reports status to the MCP client.

Choose between incremental and full indexing modes:

# Fast, incremental index of the current project

codebase-memory-mcp cli index_repository '{"repo_path":"." ,"mode":"fast"}'

# Full (slow) index – useful for a fresh database

codebase-memory-mcp cli index_repository '{"repo_path":"/path/to/project","mode":"full"}'

The fast mode performs incremental updates by processing only changed files, while full mode rebuilds the entire graph from scratch, useful for initializing a new database or recovering from corruption.

Data Persistence and Storage

The final stage writes the complete knowledge graph to disk via src/store/store.c. This module serializes nodes and edges into SQLite databases stored in ~/.cache/codebase-memory-mcp/, providing ACID-compliant storage that MCP tools query via SQL for semantic code search and navigation.

Summary

• The codebase-memory-mcp indexing pipeline transforms raw source code into a queryable knowledge graph through the index_repository entry point.
• The multi-pass architecture in src/pipeline/pipeline.c sequentially extracts definitions, builds call graphs, resolves types via Hybrid-LSP, and maps package dependencies.
• Each specialized pass is implemented in dedicated C files: pass_definitions.c, pass_calls.c, pass_lsp_cross.c, pass_pkgmap.c, and pass_infrascan.c.
• Results persist to SQLite via src/store/store.c, enabling fast semantic queries by MCP clients.
• Users trigger indexing via src/mcp/index_supervisor.c with either fast incremental or full rebuild modes.

Frequently Asked Questions

What data formats does the codebase-memory-mcp indexing pipeline support?

The pipeline uses vendored Tree-sitter grammars to parse most mainstream programming languages. Additionally, src/pipeline/pass_pkgmap.c handles package manifests like package.json and go.mod, while src/pipeline/pass_infrascan.c processes Dockerfiles and Kubernetes YAML files, ensuring comprehensive coverage of both application code and deployment configuration.

Where does the codebase-memory-mcp indexing pipeline store its data?

According to the source implementation in src/store/store.c, the pipeline persists the knowledge graph to SQLite databases located in ~/.cache/codebase-memory-mcp/. This provides durable, file-based storage that survives between MCP server restarts while maintaining transactional integrity during the indexing process.

How does the Hybrid-LSP resolution improve the indexing pipeline?

The src/pipeline/pass_lsp_cross.c module implements Hybrid-LSP type-resolution, which refines raw call edges by resolving cross-file and cross-module symbol references that Tree-sitter parsing alone cannot determine. This enables accurate call graph construction across package boundaries, distinguishing between identically named methods in different files.

Can I run the indexing pipeline incrementally?

Yes. The index_repository tool supports a "mode":"fast" parameter for incremental updates that only processes changed files, while "mode":"full" triggers a complete rebuild via the orchestration logic in src/mcp/index_supervisor.c. Fast mode is ideal for continuous integration workflows, whereas full mode ensures database consistency after significant structural changes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}