How Tree-Sitter Syntactic Parsing Compares to Hybrid LSP in codebase-memory-mcp

codebase-memory-mcp employs a two-layer indexing system where Tree-sitter generates fast syntactic ASTs and Hybrid LSP resolves types to create semantic cross-references.

Understanding how tree-sitter syntactic parsing compares to hybrid LSP reveals the architectural foundation of the codebase-memory-mcp knowledge graph. Rather than choosing between these technologies, the project uses them sequentially—first building a language-agnostic syntax tree, then enriching it with type-aware semantic links. This dual-pass approach balances parsing speed with deep code intelligence.

The Two-Pass Architecture

The codebase-memory-mcp indexer processes source code in two distinct phases that build upon each other. The first pass handles structure; the second pass handles meaning.

Pass 1: Tree-Sitter Syntactic Parsing

The initial pass uses vendored Tree-sitter grammars supporting 158 languages to generate a purely syntactic Abstract Syntax Tree (AST). According to the source code in internal/cbm/grammar_*.c (e.g., grammar_python.c, grammar_cpp.c), these compiled grammars identify nodes for definitions, calls, imports, and literals.

Strengths:

• Extreme speed: Processes millions of lines in hundreds of milliseconds
• Language agnostic: Same parser code works across all grammars
• Precise boundaries: Provides exact source locations and token ranges

Limitations:

• Contains only syntax—it cannot determine what a name actually refers to across files or type hierarchies
• Cross-file references remain unresolved at this stage

The internal/cbm/extract_calls.c file extracts raw CALLS edges from this AST, but these edges represent syntactic appearances rather than semantic relationships.

Pass 2: Hybrid LSP Semantic Resolution

The second pass runs a lightweight, hand-written C implementation of type-resolution algorithms located in internal/cbm/lsp/. This layer operates on top of the Tree-sitter AST to resolve imports, generics, inheritance, and standard-library types for nine supported language families: Python, TypeScript/JS/JSX/TSX, PHP, C#, Go, C/C++, Java, Kotlin, Rust, and Perl.

Capabilities:

• Produces type-aware RESOLVED_CALLS edges that mirror IDE "Go to Definition" behavior
• Enables cross-file, cross-module resolution (e.g., resolving user.profile.display_name() to Profile.display_name three modules away)
• Runs entirely in-process—no external language server, no API keys, no network round-trips

Constraints:

• Only the nine language families have full Hybrid LSP support; others fall back to textual resolution
• Hand-crafted resolvers may approximate exotic language features rather than fully spec-compliant parsing

Performance and Capability Comparison

When evaluating tree-sitter syntactic parsing compared to hybrid LSP, consider these architectural trade-offs:

Dimension	Tree-Sitter Pass	Hybrid LSP Pass
Speed	Hundreds of milliseconds for millions of lines	Modest overhead (sub-second for typical projects)
Accuracy	Syntactic only—knows what was written	Semantic—knows what it refers to
Scope	158 languages supported	9 language families with deep resolution
Dependencies	Zero external processes	Self-contained C implementation
Determinism	Fully deterministic offline	Fully deterministic offline

The Hybrid LSP layer therefore augments the Tree-sitter AST rather than replacing it. The pipeline looks like this:

1. Tree-sitter parsing → generates function_declaration, call_expression, and import_statement nodes
2. Hybrid LSP resolution → walks the AST, builds per-language symbol tables, follows import graphs, performs generic substitution, and rewrites call edges to point at the resolved target

Practical Code Examples

You can observe the difference between syntactic and semantic indexing by toggling the LSP layer:

# Index with ONLY Tree-sitter (LSP disabled)

CBM_LSP_DISABLED=1 codebase-memory-mcp cli index_repository \
    '{"repo_path":"/path/to/myproj"}'

# Query raw syntactic call edges

codebase-memory-mcp cli search_graph \
    '{"project":"myproj","label":"Call","name_pattern":".*"}' | jq '.results[]'

# Index with Hybrid LSP enabled (default behavior)

codebase-memory-mcp cli index_repository \
    '{"repo_path":"/path/to/myproj"}'

# Query resolved calls—now edges reflect true definitions

codebase-memory-mcp cli search_graph \
    '{"project":"myproj","label":"ResolvedCall","name_pattern":"^UserService"}' \
    | jq '.results[]'

In the first query, expect to see calls pointing to ERROR or MISSING nodes when the target resides in another file. After enabling Hybrid LSP, the ResolvedCall query returns only statically resolved cross-module references.

Implementation Details in the Source Code

The distinction between these two parsing strategies is evident in the repository structure:

• internal/cbm/grammar_*.c: Contains the vendored Tree-sitter grammars compiled into the binary. These drive the first parsing pass for all 158 supported languages.

• internal/cbm/extract_calls.c: Implements the extraction of raw CALLS edges from the Tree-sitter AST before semantic resolution occurs.

• internal/cbm/lsp/: Houses the Hybrid LSP resolvers. This directory contains the C implementations that build symbol tables, traverse import graphs, and perform type substitution for the nine supported language families.

As documented in the repository's README under the Hybrid LSP section, this architecture ensures that a call from foo() links directly to the actual implementation of foo, even when it lives in a different package or module—something a pure Tree-sitter graph cannot guarantee.

Summary

• Tree-sitter provides speed: It generates hierarchical ASTs for 158 languages in milliseconds, but only captures syntactic relationships.
• Hybrid LSP provides depth: It adds semantic resolution for 9 language families, creating type-aware edges that cross file and module boundaries.
• They work sequentially: The Hybrid LSP layer consumes the Tree-sitter AST to produce resolved edges, not as a replacement but as an augmentation.
• Both are deterministic: Neither requires external daemons, network connections, or per-project configuration; the entire stack lives in a single static binary.

Frequently Asked Questions

Does Hybrid LSP replace Tree-sitter parsing?

No. Hybrid LSP augments the Tree-sitter AST rather than replacing it. The Tree-sitter pass always runs first to generate the syntactic structure, and the Hybrid LSP pass subsequently walks that AST to resolve types and cross-references. Even when Hybrid LSP is disabled, the Tree-sitter graph remains available.

Which languages support Hybrid LSP resolution?

Only nine language families have full Hybrid LSP support: Python, TypeScript/JavaScript (including JSX/TSX), PHP, C#, Go, C/C++, Java, Kotlin, Rust, and Perl. All other languages among the 158 Tree-sitter grammars fall back to purely syntactic analysis with textual resolution only.

How can I disable Hybrid LSP for faster indexing?

Set the environment variable CBM_LSP_DISABLED=1 before running the index command. This skips the semantic resolution phase, reducing overhead to the minimum required for Tree-sitter parsing. Use this mode when you need only structural information or when working with unsupported languages.

Why is it called "Hybrid" LSP?

The term reflects that this layer combines the static analysis capabilities of a Language Server Protocol (LSP) implementation with a self-contained, offline architecture. Unlike traditional LSP servers that run as external processes requiring configuration and network communication, the Hybrid LSP is a hand-written C resolver embedded directly in the codebase-memory-mcp binary, providing IDE-quality cross-references without the infrastructure overhead.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}