# Tree-sitter AST Extraction Process for Languages in GitNexus: A 9-Step Technical Pipeline

> Explore the 9-step Tree-sitter AST extraction process in GitNexus. Extract definitions, imports calls and inheritance across JS TS Python Java C with a unified pipeline. Learn more!

- Repository: [Abhigyan Patwari/GitNexus](https://github.com/abhigyanpatwari/GitNexus)
- Tags: how-to-guide
- Published: 2026-03-08

---

**GitNexus converts source files into Tree-sitter ASTs and executes handcrafted queries to extract definitions, imports, calls, and inheritance relationships through a unified nine-step pipeline that operates identically across JavaScript, TypeScript, Python, Java, and C.**

GitNexus implements a robust Tree-sitter AST extraction process for languages in GitNexus to power its code understanding engine. The system treats every supported language uniformly, varying only the Tree-sitter query strings while maintaining consistent parsing, caching, and graph construction logic across the codebase.

## Phase 1: Language Detection and AST Generation

The extraction begins with file analysis and Tree-sitter initialization. These initial steps transform raw source code into a traversable AST structure while enforcing memory safety limits.

### 1. Language Detection via File Extensions

GitNexus identifies the programming language by inspecting file extensions through `getLanguageFromFilename(file.path)` in [`gitnexus/src/core/ingestion/utils.ts`](https://github.com/abhigyanpatwari/GitNexus/blob/main/gitnexus/src/core/ingestion/utils.ts). This function maps extensions like `.ts`, `.py`, `.java`, and `.c` to entries in the `SupportedLanguages` enum, ensuring the correct grammar is selected for parsing.

### 2. Parser and Grammar Loading

The system creates a single `Parser` instance via `loadParser()` and binds the appropriate Tree-sitter grammar using `loadLanguage(language, filePath)` from [`gitnexus/src/core/ingestion/parser-loader.ts`](https://github.com/abhigyanpatwari/GitNexus/blob/main/gitnexus/src/core/ingestion/parser-loader.ts). This module dynamically loads language-specific grammars such as `tree-sitter-typescript` or `tree-sitter-python` and invokes `parser.setLanguage(...)` to configure the parser instance.

### 3. File Parsing with Memory Constraints

In [`gitnexus/src/core/ingestion/parsing-processor.ts`](https://github.com/abhigyanpatwari/GitNexus/blob/main/gitnexus/src/core/ingestion/parsing-processor.ts), the source is parsed using `parser.parse(file.content, undefined, { bufferSize: 1024 * 256 })`, which produces a `Tree` object containing the `rootNode` AST. To prevent out-of-memory errors, GitNexus skips files exceeding 512KB, prioritizing stability over completeness for large generated files.

## Phase 2: Query Execution and Capture Processing

Once the AST exists, GitNexus executes language-specific Tree-sitter queries to identify semantic constructs. This phase transforms AST nodes into structured capture data ready for relationship analysis.

### 4. Language-Specific Query Selection

The system retrieves multi-line query strings from `LANGUAGE_QUERIES[language]` defined in [`gitnexus/src/core/ingestion/tree-sitter-queries.ts`](https://github.com/abhigyanpatwari/GitNexus/blob/main/gitnexus/src/core/ingestion/tree-sitter-queries.ts). These handcrafted queries match grammar-specific patterns for function definitions, import statements, method calls, and class inheritance across all supported languages.

### 5. Compiled Query Execution

GitNexus compiles the query using `new Parser.Query(language, queryString)` and executes it against the AST via `query.matches(tree.rootNode)`. This returns an array of match objects, each containing captures such as `@definition.function`, `@import.source`, `@call.name`, and heritage patterns that map directly to AST node ranges.

### 6. Capture Map Construction

For each match, processors build a `captureMap` (`Record<string, any>`) that associates capture names with concrete AST nodes. This mapping occurs across [`parsing-processor.ts`](https://github.com/abhigyanpatwari/GitNexus/blob/main/parsing-processor.ts), [`import-processor.ts`](https://github.com/abhigyanpatwari/GitNexus/blob/main/import-processor.ts), [`call-processor.ts`](https://github.com/abhigyanpatwari/GitNexus/blob/main/call-processor.ts), and [`heritage-processor.ts`](https://github.com/abhigyanpatwari/GitNexus/blob/main/heritage-processor.ts), normalizing node access for downstream extraction logic.

## Phase 3: Semantic Analysis and Knowledge Graph Construction

The final phase extracts domain-specific relationships, caches ASTs for reuse, and populates the knowledge graph with nodes and edges.

### 7. Semantic Extraction and Node Creation

**Definitions:** The system creates `GraphNode` objects with properties including `name`, `filePath`, `startLine`, and `isExported` to represent functions, classes, and variables.

**Imports:** [`import-processor.ts`](https://github.com/abhigyanpatwari/GitNexus/blob/main/import-processor.ts) resolves raw import strings to absolute file paths and creates `IMPORTS` edges linking dependent files to their dependencies.

**Calls:** [`call-processor.ts`](https://github.com/abhigyanpatwari/GitNexus/blob/main/call-processor.ts) resolves callees via symbol tables and import maps, then adds `CALLS` edges connecting call sites to their target definitions.

**Heritage:** [`heritage-processor.ts`](https://github.com/abhigyanpatwari/GitNexus/blob/main/heritage-processor.ts) captures `extends` and `implements` relationships, mapping class hierarchies into the graph structure.

### 8. AST Caching for Multi-Phase Reuse

To avoid redundant parsing, GitNexus stores parsed `Tree` objects in `ASTCache`—an LRU cache implemented in [`gitnexus/src/core/ingestion/ast-cache.ts`](https://github.com/abhigyanpatwari/GitNexus/blob/main/gitnexus/src/core/ingestion/ast-cache.ts). Subsequent phases (imports, calls, heritage) retrieve the cached AST instead of reparsing, significantly improving performance when analyzing complex dependency chains.

### 9. Knowledge Graph Population

All extracted nodes, edges, and auxiliary symbols are collected into a `KnowledgeGraph` structure. This graph serves as the foundation for downstream indexing, complex queries, and agent reasoning capabilities within the GitNexus ecosystem.

## Parallel Processing with Worker Pools

For large codebases, GitNexus distributes the extraction pipeline across worker threads via [`gitnexus/src/core/ingestion/workers/parse-worker.ts`](https://github.com/abhigyanpatwari/GitNexus/blob/main/gitnexus/src/core/ingestion/workers/parse-worker.ts). This parallel execution model processes multiple files simultaneously while maintaining the same nine-step sequence, maximizing CPU utilization during repository ingestion.

## Practical Implementation: Parsing TypeScript

The following example demonstrates the complete Tree-sitter AST extraction process for languages in GitNexus using TypeScript:

```typescript
import { loadParser, loadLanguage } from './parser-loader.js';
import { getLanguageFromFilename } from './utils.js';
import { LANGUAGE_QUERIES } from './tree-sitter-queries.js';
import Parser from 'tree-sitter';

// Step 1: Detect language from filename
const language = getLanguageFromFilename('src/utils.ts');

// Step 2: Initialize parser and load TypeScript grammar
const parser = await loadParser();
await loadLanguage(language, 'src/utils.ts');

// Step 3: Parse source file with 256KB buffer
const source = await Deno.readTextFile('src/utils.ts');
const tree = parser.parse(source, undefined, { bufferSize: 1024 * 256 });

// Step 4: Retrieve TypeScript-specific query
const queryString = LANGUAGE_QUERIES[language];

// Step 5: Execute query against AST root
const query = new Parser.Query(parser.getLanguage(), queryString);
const matches = query.matches(tree.rootNode);

// Step 6: Build capture map and extract definitions
const definitions: string[] = [];
for (const match of matches) {
  const captureMap: Record<string, any> = {};
  for (const cap of match.captures) {
    captureMap[cap.name] = cap.node;
  }
  
  // Step 7: Process function definitions
  if (captureMap['definition.function']) {
    const nameNode = captureMap['name'];
    definitions.push(nameNode.text);
  }
}

console.log('Functions defined:', definitions);

```

This skeleton remains identical for Python, Java, or C; only the `LANGUAGE_QUERIES[language]` string and resulting capture names differ.

## Core Source Files in the Extraction Pipeline

- **[`gitnexus/src/core/ingestion/tree-sitter-queries.ts`](https://github.com/abhigyanpatwari/GitNexus/blob/main/gitnexus/src/core/ingestion/tree-sitter-queries.ts)** — Central repository of language-specific query strings (`TYPESCRIPT_QUERIES`, `PYTHON_QUERIES`, etc.) and the `LANGUAGE_QUERIES` export map.
- **[`gitnexus/src/core/ingestion/parsing-processor.ts`](https://github.com/abhigyanpatwari/GitNexus/blob/main/gitnexus/src/core/ingestion/parsing-processor.ts)** — Orchestrates parsing, query execution, definition node creation, AST caching, and `DEFINES` edge generation.
- **[`gitnexus/src/core/ingestion/parser-loader.ts`](https://github.com/abhigyanpatwari/GitNexus/blob/main/gitnexus/src/core/ingestion/parser-loader.ts)** — Manages Tree-sitter parser instantiation and dynamic grammar loading for detected languages.
- **[`gitnexus/src/core/ingestion/utils.ts`](https://github.com/abhigyanpatwari/GitNexus/blob/main/gitnexus/src/core/ingestion/utils.ts)** — Provides `getLanguageFromFilename` and helper utilities for file analysis.
- **[`gitnexus/src/core/ingestion/ast-cache.ts`](https://github.com/abhigyanpatwari/GitNexus/blob/main/gitnexus/src/core/ingestion/ast-cache.ts)** — Implements the LRU cache for parsed `Tree` objects to enable cross-phase AST reuse.
- **[`gitnexus/src/core/ingestion/import-processor.ts`](https://github.com/abhigyanpatwari/GitNexus/blob/main/gitnexus/src/core/ingestion/import-processor.ts)** — Post-parsing phase that extracts import relationships and creates `IMPORTS` edges.
- **[`gitnexus/src/core/ingestion/call-processor.ts`](https://github.com/abhigyanpatwari/GitNexus/blob/main/gitnexus/src/core/ingestion/call-processor.ts)** — Resolves method invocations and constructs `CALLS` edges via symbol table lookups.
- **[`gitnexus/src/core/ingestion/heritage-processor.ts`](https://github.com/abhigyanpatwari/GitNexus/blob/main/gitnexus/src/core/ingestion/heritage-processor.ts)** — Captures class inheritance and interface implementation relationships.
- **[`gitnexus/src/core/ingestion/workers/parse-worker.ts`](https://github.com/abhigyanpatwari/GitNexus/blob/main/gitnexus/src/core/ingestion/workers/parse-worker.ts)** — Worker-thread implementation enabling parallel file processing.

## Summary

- GitNexus employs a **unified nine-step pipeline** for Tree-sitter AST extraction that works identically across JavaScript, TypeScript, Python, Java, and C.
- The system uses **handcrafted Tree-sitter queries** stored in [`tree-sitter-queries.ts`](https://github.com/abhigyanpatwari/GitNexus/blob/main/tree-sitter-queries.ts) to capture language-specific constructs like `@definition.function` and `@call.name`.
- **Memory safety** is enforced through a 256KB parse buffer and automatic skipping of files larger than 512KB.
- An **LRU AST cache** eliminates redundant parsing across the definition, import, call, and heritage processing phases.
- **Parallel worker pools** via [`parse-worker.ts`](https://github.com/abhigyanpatwari/GitNexus/blob/main/parse-worker.ts) enable scalable repository analysis without blocking the main thread.

## Frequently Asked Questions

### How does GitNexus handle unsupported file types during AST extraction?

GitNexus relies on `getLanguageFromFilename()` in [`utils.ts`](https://github.com/abhigyanpatwari/GitNexus/blob/main/utils.ts) to validate file extensions against the `SupportedLanguages` enum. Files with unrecognized extensions are excluded from the Tree-sitter AST extraction process for languages in GitNexus before parsing begins, preventing parser errors and reducing processing overhead.

### What specific AST node captures does GitNexus extract from source files?

According to the query definitions in [`tree-sitter-queries.ts`](https://github.com/abhigyanpatwari/GitNexus/blob/main/tree-sitter-queries.ts), GitNexus extracts captures including `@definition.function` for function declarations, `@import.source` for module dependencies, `@call.name` for method invocations, and heritage patterns for `extends` and `implements` relationships. These captures populate the `captureMap` used by specialized processors to build graph nodes and edges.

### Why does GitNexus skip files larger than 512KB during parsing?

The 512KB size limit in [`parsing-processor.ts`](https://github.com/abhigyanpatwari/GitNexus/blob/main/parsing-processor.ts) prevents out-of-memory (OOM) errors when processing generated files, minified bundles, or accidentally included binaries. This safety threshold ensures that the `parser.parse()` operation with its 256KB buffer size (`1024 * 256`) remains stable during large-scale repository ingestion, prioritizing system reliability over analyzing potentially low-value large files.

### Can the AST extraction pipeline process multiple files simultaneously?

Yes. GitNexus implements parallel processing through [`parse-worker.ts`](https://github.com/abhigyanpatwari/GitNexus/blob/main/parse-worker.ts), which executes the complete nine-step Tree-sitter AST extraction pipeline within worker threads. This architecture allows multiple files to be parsed, queried, and analyzed concurrently while maintaining thread-safe access to the shared `ASTCache` for optimal performance on multi-core systems.