How Code-Graph-RAG Uses Tree-sitter for Multi-Language Code Analysis

Code-Graph-RAG leverages Tree-sitter to parse Python, JavaScript, TypeScript, Rust, Java, and other languages into abstract syntax trees (ASTs), enabling query-driven semantic extraction and incremental code-graph construction.

Code-Graph-RAG is an open-source retrieval-augmented generation (RAG) framework that builds rich code graphs from diverse programming languages. At its core, the system uses Tree-sitter—GitHub's parsing library—to create language-agnostic ASTs that power cross-language analysis, type inference, and call-graph resolution.

Grammar Discovery and Submodule Management

The framework dynamically discovers language grammars through Git submodules. When you run cgr init or cgr add-language, the tool executes _add_git_submodule in codebase_rag/tools/language.py to pull the appropriate Tree-sitter grammar repository.

After downloading, the _parse_tree_sitter_json function (lines 31-55 in codebase_rag/tools/language.py) reads the grammar's grammar.json file to extract the canonical language name and supported file extensions. This automated discovery eliminates hard-coded language lists and allows the system to support new languages by simply adding their Tree-sitter grammar as a submodule.

# Add a new language grammar as a submodule

cgr add-language --url https://github.com/tree-sitter/tree-sitter-go --name go

Lazy Parser Initialization and AST Caching

Code-Graph-RAG creates Tree-sitter Parser instances lazily and caches them globally to avoid the overhead of repeated initialization. When indexing a Python file for the first time, the system loads the compiled grammar and constructs a Language object from tree_sitter_python.language().

The ASTCacheProtocol implementation in codebase_rag/ast_cache.py stores parsed trees keyed by file path. This cache enables incremental re-indexing and prevents redundant parsing when the same file is visited from multiple call-graph paths. The initial imports and cache setup appear in codebase_rag/parsers/py/ast_analyzer.py (lines 1-14).

from tree_sitter import Language, Parser
import tree_sitter_python

# Build Language from compiled grammar and create parser

py_lang = Language(tree_sitter_python.language())
parser = Parser(py_lang)

# Parse source and cache the tree

source_bytes = Path("example.py").read_bytes()
tree = parser.parse(source_bytes)

Query-Driven AST Traversal

The core analysis logic resides in PythonAstAnalyzerMixin (lines 54-92 in codebase_rag/parsers/py/ast_analyzer.py). This mixin uses Tree-sitter Query objects to capture semantic nodes like assignments, function definitions, and import statements.

The workflow follows three steps:

1. Query Compilation: Language-specific query strings (defined in codebase_rag/constants/ast_python.py) are compiled once and cached via get_cached_query.
2. Cursor Execution: A QueryCursor iterates over the AST root node to find capture groups (e.g., @assignment, @for_stmt).
3. Node Processing: Helper methods like _process_assignment_simple and _process_assignment_complex extract type information from captured nodes, building a local variable type map for accurate call-graph construction.

If a query fails due to missing grammars, the system falls back to manual stack-based traversal.

from codebase_rag.parsers.py.ast_analyzer import _PY_TRAVERSE_QUERY, get_cached_query

# Obtain cached query and create cursor

query = get_cached_query(py_lang, _PY_TRAVERSE_QUERY)
cursor = query.Cursor()

# Capture semantic nodes from the AST root

captures = cursor.captures(tree.root_node)
assignments = captures.get("assignment", [])

for node in assignments:
    print(f"Assignment found at line {node.start_point[0]}")

Cross-Language Semantic Resolution

Code-Graph-RAG provides a unified interface for cross-language analysis through shared utilities. The Node and Query abstractions from Tree-sitter are reused across language parsers, as seen in codebase_rag/parsers/js_ts/utils.py (lines 4-7).

The fully-qualified name (FQN) resolver in codebase_rag/utils/fqn_resolver.py (lines 12-20) walks cached ASTs using Tree-sitter Node objects to locate exact definitions across language boundaries. This enables the system to resolve method calls in JavaScript that reference Python backends, or track Rust functions called from TypeScript bindings.

Language-specific query patterns are centralized in codebase_rag/constants/ (e.g., ast_python.py, ast_rust.py), ensuring consistent capture group naming across different grammars.

from codebase_rag.utils.fqn_resolver import resolve_fqn

# Resolve a symbol across cached ASTs

node = resolve_fqn("my_pkg.module.MyClass.method", ast_cache, parser_cache)
print(node.type)  # "function_definition" or "method_declaration"

Summary

• Grammar Discovery: The _parse_tree_sitter_json function in codebase_rag/tools/language.py automates language support by parsing Tree-sitter grammar metadata from Git submodules.
• Performance Optimization: Parser instances and AST trees are cached globally via ASTCacheProtocol to eliminate redundant parsing during incremental indexing.
• Semantic Extraction: PythonAstAnalyzerMixin uses compiled Tree-sitter queries and QueryCursor captures to extract functions, classes, and type information from AST nodes.
• Cross-Language Support: Shared utilities like fqn_resolver.py and constants files provide a unified interface for multi-language code-graph construction.

Frequently Asked Questions

What programming languages does Code-Graph-RAG support through Tree-sitter?

The framework supports Python, JavaScript, TypeScript, Rust, Java, C/C++, Go, Lua, PHP, and C#, with the ability to add new languages by registering their Tree-sitter grammars as Git submodules. The _parse_tree_sitter_json function automatically detects file extensions and language names from each grammar's metadata.

How does Code-Graph-RAG cache Tree-sitter parsers for performance?

The system implements lazy initialization combined with the ASTCacheProtocol. Parser objects are created once per language and reused across the codebase, while parsed ASTs are stored in an in-memory cache keyed by file path. This architecture prevents re-parsing files during incremental updates or repeated cross-reference lookups.

What happens when a Tree-sitter query fails or encounters a missing grammar?

When a query compilation fails or a grammar is unavailable, Code-Graph-RAG falls back to manual stack-based AST traversal. This ensures robustness while maintaining the efficiency of query-driven capture groups for supported languages. The fallback logic is integrated into the mixin classes like PythonAstAnalyzerMixin.

How does the system resolve symbols across different programming languages?

The resolve_fqn utility in codebase_rag/utils/fqn_resolver.py traverses cached ASTs using Tree-sitter Node objects to locate symbol definitions. By using language-agnostic Node APIs and standardized query patterns from constants/ast_*.py files, the resolver can trace calls from TypeScript to Python or Rust, enabling true cross-language code-graph analysis.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}