Tree-sitter AST Extraction Process for Languages in GitNexus: A 9-Step Technical Pipeline

GitNexus converts source files into Tree-sitter ASTs and executes handcrafted queries to extract definitions, imports, calls, and inheritance relationships through a unified nine-step pipeline that operates identically across JavaScript, TypeScript, Python, Java, and C.

GitNexus implements a robust Tree-sitter AST extraction process for languages in GitNexus to power its code understanding engine. The system treats every supported language uniformly, varying only the Tree-sitter query strings while maintaining consistent parsing, caching, and graph construction logic across the codebase.

Phase 1: Language Detection and AST Generation

The extraction begins with file analysis and Tree-sitter initialization. These initial steps transform raw source code into a traversable AST structure while enforcing memory safety limits.

1. Language Detection via File Extensions

GitNexus identifies the programming language by inspecting file extensions through getLanguageFromFilename(file.path) in gitnexus/src/core/ingestion/utils.ts. This function maps extensions like .ts, .py, .java, and .c to entries in the SupportedLanguages enum, ensuring the correct grammar is selected for parsing.

2. Parser and Grammar Loading

The system creates a single Parser instance via loadParser() and binds the appropriate Tree-sitter grammar using loadLanguage(language, filePath) from gitnexus/src/core/ingestion/parser-loader.ts. This module dynamically loads language-specific grammars such as tree-sitter-typescript or tree-sitter-python and invokes parser.setLanguage(...) to configure the parser instance.

3. File Parsing with Memory Constraints

In gitnexus/src/core/ingestion/parsing-processor.ts, the source is parsed using parser.parse(file.content, undefined, { bufferSize: 1024 * 256 }), which produces a Tree object containing the rootNode AST. To prevent out-of-memory errors, GitNexus skips files exceeding 512KB, prioritizing stability over completeness for large generated files.

Phase 2: Query Execution and Capture Processing

Once the AST exists, GitNexus executes language-specific Tree-sitter queries to identify semantic constructs. This phase transforms AST nodes into structured capture data ready for relationship analysis.

4. Language-Specific Query Selection

The system retrieves multi-line query strings from LANGUAGE_QUERIES[language] defined in gitnexus/src/core/ingestion/tree-sitter-queries.ts. These handcrafted queries match grammar-specific patterns for function definitions, import statements, method calls, and class inheritance across all supported languages.

5. Compiled Query Execution

GitNexus compiles the query using new Parser.Query(language, queryString) and executes it against the AST via query.matches(tree.rootNode). This returns an array of match objects, each containing captures such as @definition.function, @import.source, @call.name, and heritage patterns that map directly to AST node ranges.

6. Capture Map Construction

For each match, processors build a captureMap (Record<string, any>) that associates capture names with concrete AST nodes. This mapping occurs across parsing-processor.ts, import-processor.ts, call-processor.ts, and heritage-processor.ts, normalizing node access for downstream extraction logic.

Phase 3: Semantic Analysis and Knowledge Graph Construction

The final phase extracts domain-specific relationships, caches ASTs for reuse, and populates the knowledge graph with nodes and edges.

7. Semantic Extraction and Node Creation

Definitions: The system creates GraphNode objects with properties including name, filePath, startLine, and isExported to represent functions, classes, and variables.

Imports: import-processor.ts resolves raw import strings to absolute file paths and creates IMPORTS edges linking dependent files to their dependencies.

Calls: call-processor.ts resolves callees via symbol tables and import maps, then adds CALLS edges connecting call sites to their target definitions.

Heritage: heritage-processor.ts captures extends and implements relationships, mapping class hierarchies into the graph structure.

8. AST Caching for Multi-Phase Reuse

To avoid redundant parsing, GitNexus stores parsed Tree objects in ASTCache—an LRU cache implemented in gitnexus/src/core/ingestion/ast-cache.ts. Subsequent phases (imports, calls, heritage) retrieve the cached AST instead of reparsing, significantly improving performance when analyzing complex dependency chains.

9. Knowledge Graph Population

All extracted nodes, edges, and auxiliary symbols are collected into a KnowledgeGraph structure. This graph serves as the foundation for downstream indexing, complex queries, and agent reasoning capabilities within the GitNexus ecosystem.

Parallel Processing with Worker Pools

For large codebases, GitNexus distributes the extraction pipeline across worker threads via gitnexus/src/core/ingestion/workers/parse-worker.ts. This parallel execution model processes multiple files simultaneously while maintaining the same nine-step sequence, maximizing CPU utilization during repository ingestion.

Practical Implementation: Parsing TypeScript

The following example demonstrates the complete Tree-sitter AST extraction process for languages in GitNexus using TypeScript:

import { loadParser, loadLanguage } from './parser-loader.js';
import { getLanguageFromFilename } from './utils.js';
import { LANGUAGE_QUERIES } from './tree-sitter-queries.js';
import Parser from 'tree-sitter';

// Step 1: Detect language from filename
const language = getLanguageFromFilename('src/utils.ts');

// Step 2: Initialize parser and load TypeScript grammar
const parser = await loadParser();
await loadLanguage(language, 'src/utils.ts');

// Step 3: Parse source file with 256KB buffer
const source = await Deno.readTextFile('src/utils.ts');
const tree = parser.parse(source, undefined, { bufferSize: 1024 * 256 });

// Step 4: Retrieve TypeScript-specific query
const queryString = LANGUAGE_QUERIES[language];

// Step 5: Execute query against AST root
const query = new Parser.Query(parser.getLanguage(), queryString);
const matches = query.matches(tree.rootNode);

// Step 6: Build capture map and extract definitions
const definitions: string[] = [];
for (const match of matches) {
  const captureMap: Record<string, any> = {};
  for (const cap of match.captures) {
    captureMap[cap.name] = cap.node;
  }
  
  // Step 7: Process function definitions
  if (captureMap['definition.function']) {
    const nameNode = captureMap['name'];
    definitions.push(nameNode.text);
  }
}

console.log('Functions defined:', definitions);

This skeleton remains identical for Python, Java, or C; only the LANGUAGE_QUERIES[language] string and resulting capture names differ.

Core Source Files in the Extraction Pipeline

• gitnexus/src/core/ingestion/tree-sitter-queries.ts — Central repository of language-specific query strings (TYPESCRIPT_QUERIES, PYTHON_QUERIES, etc.) and the LANGUAGE_QUERIES export map.
• gitnexus/src/core/ingestion/parsing-processor.ts — Orchestrates parsing, query execution, definition node creation, AST caching, and DEFINES edge generation.
• gitnexus/src/core/ingestion/parser-loader.ts — Manages Tree-sitter parser instantiation and dynamic grammar loading for detected languages.
• gitnexus/src/core/ingestion/utils.ts — Provides getLanguageFromFilename and helper utilities for file analysis.
• gitnexus/src/core/ingestion/ast-cache.ts — Implements the LRU cache for parsed Tree objects to enable cross-phase AST reuse.
• gitnexus/src/core/ingestion/import-processor.ts — Post-parsing phase that extracts import relationships and creates IMPORTS edges.
• gitnexus/src/core/ingestion/call-processor.ts — Resolves method invocations and constructs CALLS edges via symbol table lookups.
• gitnexus/src/core/ingestion/heritage-processor.ts — Captures class inheritance and interface implementation relationships.
• gitnexus/src/core/ingestion/workers/parse-worker.ts — Worker-thread implementation enabling parallel file processing.

Summary

• GitNexus employs a unified nine-step pipeline for Tree-sitter AST extraction that works identically across JavaScript, TypeScript, Python, Java, and C.
• The system uses handcrafted Tree-sitter queries stored in tree-sitter-queries.ts to capture language-specific constructs like @definition.function and @call.name.
• Memory safety is enforced through a 256KB parse buffer and automatic skipping of files larger than 512KB.
• An LRU AST cache eliminates redundant parsing across the definition, import, call, and heritage processing phases.
• Parallel worker pools via parse-worker.ts enable scalable repository analysis without blocking the main thread.

Frequently Asked Questions

How does GitNexus handle unsupported file types during AST extraction?

GitNexus relies on getLanguageFromFilename() in utils.ts to validate file extensions against the SupportedLanguages enum. Files with unrecognized extensions are excluded from the Tree-sitter AST extraction process for languages in GitNexus before parsing begins, preventing parser errors and reducing processing overhead.

What specific AST node captures does GitNexus extract from source files?

According to the query definitions in tree-sitter-queries.ts, GitNexus extracts captures including @definition.function for function declarations, @import.source for module dependencies, @call.name for method invocations, and heritage patterns for extends and implements relationships. These captures populate the captureMap used by specialized processors to build graph nodes and edges.

Why does GitNexus skip files larger than 512KB during parsing?

The 512KB size limit in parsing-processor.ts prevents out-of-memory (OOM) errors when processing generated files, minified bundles, or accidentally included binaries. This safety threshold ensures that the parser.parse() operation with its 256KB buffer size (1024 * 256) remains stable during large-scale repository ingestion, prioritizing system reliability over analyzing potentially low-value large files.

Can the AST extraction pipeline process multiple files simultaneously?

Yes. GitNexus implements parallel processing through parse-worker.ts, which executes the complete nine-step Tree-sitter AST extraction pipeline within worker threads. This architecture allows multiple files to be parsed, queried, and analyzed concurrently while maintaining thread-safe access to the shared ASTCache for optimal performance on multi-core systems.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}