How codebase-memory-mcp Builds a Knowledge Graph from Source Code: A Deep Dive into the Three-Stage Pipeline

The codebase-memory-mcp tool constructs a persistent knowledge graph by parsing source code with Tree-sitter, creating an in-memory graph buffer of symbols and relationships, and flushing the results to a SQLite database for fast structured queries.

The codebase-memory-mcp project transforms raw source trees into queryable graph databases that power AI-assisted code understanding. By combining Tree-sitter parsing with a custom two-phase extraction pipeline, this Model Context Protocol (MCP) server enables developers to build a knowledge graph from source code that persists in SQLite and supports complex structural queries without re-parsing files.

The Three-Stage Knowledge Graph Construction Pipeline

The architecture follows a strict three-stage process: extraction, in-memory graph construction, and SQLite persistence. Each stage is decoupled yet tightly integrated to handle repositories containing millions of lines of code.

Stage 1: Source Code Extraction with Tree-sitter

The pipeline begins in src/discover/discover.c, which walks the repository tree and filters files by supported languages (158 bundled Tree-sitter grammars). For each eligible file, the system calls cbm_extract_file to parse the source into a CBMFileResult structure.

This result container captures:

• Definitions: functions, classes, methods, variables
• Call sites: function invocations, HTTP routes, async boundaries
• Imports: module dependencies and package references
• Channels: event emissions and listeners
• Environment accesses: configuration reads

The extraction logic spans src/pipeline/pass_definitions.c (for symbol definitions) and src/pipeline/pass_calls.c (for call relationships), utilizing Tree-sitter ASTs enhanced by optional LSP semantic data from src/semantic/semantic.c.

Stage 2: Node and Edge Creation

Once extracted, the system builds an in-memory representation using the graph buffer implemented in src/graph_buffer/graph_buffer.c. This stage operates in two distinct phases orchestrated by src/pipeline/pipeline.c:

Phase 1: Definitions Pass

• cbm_pipeline_pass_definitions processes each CBMFileResult
• process_def constructs JSON property blobs via build_def_props
• cbm_gbuf_upsert_node creates nodes (cbm_node_t) for every symbol with qualified names
• cbm_registry_add registers each symbol in a central registry for later resolution
• "DEFINES" edges connect file nodes (__file__) to their contained symbols

Phase 2: Resolution Pass

• After caching results (result_cache), the pipeline builds a namespace map via cbm_pipeline_namespace_map_build
• create_import_edges_for_file resolves imports using cbm_pipeline_resolve_import_node and creates "IMPORTS" edges
• cbm_pipeline_pass_calls.c creates "CALLS", "HTTP_CALLS", and "ASYNC_CALLS" edges linking call sites to definitions
• Channel edges ("EMITS", "LISTENS_ON") and configuration edges ("CONFIGURES") bind runtime behavior to static symbols

Stage 3: Persistence to SQLite

The final stage converts the mutable graph buffer into a durable SQLite database stored in ~/.cache/codebase-memory-mcp. The storage layer in src/store/store.c implements bulk write operations optimized for large graphs:

• cbm_store_begin_bulk initiates a transaction
• cbm_store_upsert_node_batch persists nodes (cbm_node_t)
• cbm_store_insert_edge_batch persists relationships (cbm_edge_t)
• cbm_store_check_integrity validates the database; corrupted files are renamed to *.corrupt

The schema defined in src/store/store.h supports compressed artifacts (.zst) for fast team bootstrapping, allowing new team members to download pre-indexed graphs rather than re-parsing entire repositories.

From MCP Request to Structured Graph

When an AI agent invokes the index_repository tool, the request flows through src/mcp/mcp.c where cbm_jsonrpc_parse handles JSON-RPC dispatch. The server validates arguments via cbm_mcp_get_string_arg and initializes a cbm_mcp_server_t instance holding the project store.

The indexing workflow proceeds as follows:

1. Discovery: discover.c enumerates source files and builds a cbm_file_info_t array
2. Extraction: cbm_extract_file generates CBMFileResult structures containing AST-derived metadata
3. Node Creation: The definitions pass upserts nodes into the graph buffer and populates the symbol registry
4. Edge Resolution: Import and call edges are resolved against the fully populated registry
5. Persistence: The graph buffer flushes to SQLite via batched upsert operations

Querying the Knowledge Graph

Once persisted, the graph supports high-performance queries without re-parsing source files. MCP tools exposed in src/mcp/mcp.c include:

• search_graph: Executes BM25 full-text search via cbm_store_search with optional vector similarity (cbm_store_vector_search)
• trace_path: Performs breadth-first traversal (cbm_store_bfs) over "CALLS" edges to map dependency chains
• query_graph: Supports Cypher graph queries (cbm_cypher_execute) against the SQLite-backed schema

The result is a single-file persistent graph that enables AI agents to answer structural questions with approximately 120× token reduction compared to naive text searching.

Code Examples

Index a Repository via CLI

# Index the current repo and write a compressed artifact for teammates

codebase-memory-mcp index_repository \
    --repo_path . \
    --persistence true \
    --name my-project

This command routes through src/cli/cli.c to issue an MCP index_repository request handled in src/mcp/mcp.c.

Search the Graph via JSON-RPC

{
  "jsonrpc": "2.0",
  "method": "search_graph",
  "id": 1,
  "params": {
    "project": "my-project",
    "query": "publish event",
    "limit": 5
  }
}

The server executes cbm_store_search with full-text ranking and returns relevance-scored nodes.

Trace a Call Chain

{
  "jsonrpc": "2.0",
  "method": "trace_path",
  "id": 2,
  "params": {
    "function_name": "UserService.sendMessage",
    "project": "my-project",
    "direction": "outbound",
    "depth": 3,
    "mode": "calls"
  }
}

The handler invokes cbm_store_bfs to traverse "CALLS" edges and returns cbm_node_hop_t structures showing the execution path.

Summary

• Tree-sitter parsing extracts definitions, calls, and dependencies from source files into CBMFileResult structures
• Two-phase pipeline first creates symbol nodes and registers them, then resolves import and call edges using the registry
• Graph buffer (src/graph_buffer/graph_buffer.c) provides fast mutable storage before persistence
• SQLite backend (src/store/store.c) enables compressed, portable graph files with full-text and vector search capabilities
• MCP integration exposes graph construction and querying via standardized JSON-RPC tools for AI agent integration

Frequently Asked Questions

What database does codebase-memory-mcp use for the knowledge graph?

The system uses SQLite as its persistence layer, storing nodes (cbm_node_t) and edges (cbm_edge_t) in a relational schema defined in src/store/store.h. The database supports optional Zstandard compression (.zst) and is typically cached under ~/.cache/codebase-memory-mcp.

How does the tool handle multiple programming languages?

The extraction layer utilizes Tree-sitter with 158 bundled grammars to parse diverse languages. File discovery in src/discover/discover.c filters repositories by language support, routing each file to the appropriate parser while normalizing outputs into the common CBMFileResult format.

What types of relationships does the graph capture?

Beyond standard "DEFINES" (file-to-symbol) and "CALLS" edges, the graph captures "IMPORTS" for module dependencies, "HTTP_CALLS" for API endpoints, "ASYNC_CALLS" for asynchronous boundaries, "EMITS"/"LISTENS_ON" for event-driven channels, and "CONFIGURES" for environment variable access.

Can the knowledge graph be shared among team members?

Yes. The SQLite database can be compressed using Zstandard (.zst) to create portable artifacts. Teams can distribute these compressed graph files rather than requiring each developer to re-parse large repositories, significantly reducing bootstrapping time for new team members.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}