How Understand Anything Language Extractors Process TypeScript, Python, Go, Rust, and Java Code

Understand Anything uses dedicated Tree-Sitter-based language extractors for each supported language to perform structural analysis and call-graph extraction by traversing AST nodes specific to each grammar.

The Egonex-AI/Understand-Anything repository implements a modular extraction system where language-specific plugins parse source code into abstract syntax trees. Each extractor implements the common LanguageExtractor interface defined in packages/core/src/plugins/extractors/types.ts, providing two core methods: extractStructure for building structural representations and extractCallGraph for mapping function relationships. These extractors power the knowledge graph by converting Tree-Sitter AST nodes into normalized StructuralAnalysis and CallGraphEntry objects.

The LanguageExtractor Interface and Common Pattern

All language extractors in Understand Anything follow a consistent architectural pattern defined by the LanguageExtractor interface. Each concrete implementation specifies accepted languageIds and provides two primary entry points for AST traversal.

export class XxxExtractor implements LanguageExtractor {
  readonly languageIds = ["xxx"];

  extractStructure(rootNode: TreeSitterNode): StructuralAnalysis {
    // Walks top-level children to populate functions, classes, imports, and exports
  }

  extractCallGraph(rootNode: TreeSitterNode): CallGraphEntry[] {
    // Depth-first traversal maintaining a functionStack to record call relationships
  }
}

The extractStructure method identifies module boundaries, type definitions, and visibility markers, while extractCallGraph performs a recursive descent to build caller-callee relationships. Shared utilities from packages/core/src/plugins/extractors/base-extractor.ts handle common tasks like string unquoting and child node iteration.

TypeScript and JavaScript Extraction

The TypeScript extractor in packages/core/src/plugins/extractors/typescript-extractor.ts handles both TypeScript and JavaScript via languageIds = ["typescript", "javascript"]. It processes the Tree-Sitter grammar specific to ECMAScript-compatible languages.

Structural Analysis

The processTopLevelNode function dispatches on node types including function_declaration, class_declaration, lexical_declaration, import_statement, and export_statement. For classes, it collects method_definition nodes as methods and public_field_definition or property_definition nodes as properties.

Parameter extraction handles required_parameter, optional_parameter, rest_pattern, and rest_element nodes. Return types are extracted from type_annotation fields by stripping the leading colon. Exports are tracked via export_statement nodes, distinguishing default exports from named export_clause specifiers.

Call Graph Construction

During call-graph extraction, the extractor detects function boundaries (function_declaration, method_definition, arrow_function) and maintains a stack of current enclosing functions. When encountering a call_expression node, it records the relationship between the current function and the callee.

Python Code Analysis

The Python extractor in packages/core/src/plugins/extractors/python-extractor.ts handles Python-specific constructs like decorators and dynamic typing. Unlike languages with explicit export keywords, Python treats all top-level definitions as exports.

Handling Decorated Definitions

The extractor unwraps decorated_definition nodes to reach underlying function_definition or class_definition nodes. Parameters are extracted from identifier, typed_parameter, default_parameter, and typed_default_parameter nodes, including splat patterns (list_splat_pattern for *args and dictionary_splat_pattern for **kwargs). Return types are read from the return_type field.

Call Graph Implementation

For call-graph extraction, the extractor tracks entry into function_definition nodes and records any call node encountered during traversal. This captures both direct function calls and method invocations through attribute access.

Go Structural Analysis

The Go extractor in packages/core/src/plugins/extractors/go-extractor.ts implements Go-specific visibility rules and receiver-based method dispatch.

Export Detection and Types

Go uses capitalization for export visibility, so the extractor implements isExported to check if a name starts with an uppercase letter. For methods, the receiver field provides the receiver type through parameter_declaration and type_identifier nodes. Both functions and methods are processed, with methods attached to their receiver types via a methodsByReceiver map.

Parameter extraction iterates through parameter_declaration children, collecting identifier nodes. Return types are read from the result field, supporting single, pointer, and tuple returns.

Struct and Interface Processing

type_declaration nodes containing struct_type or interface_type become class entries. For structs, field_declaration nodes populate the properties list. For interfaces, method_elem nodes define method signatures.

Rust Module Processing

The Rust extractor in packages/core/src/plugins/extractors/rust-extractor.ts handles visibility modifiers, complex import syntax, and implementation blocks.

Visibility and Structs

Rust exports are determined by visibility_modifier nodes starting with pub. The extractor processes struct_item, enum_item, and trait_item as classes, mapping struct fields and enum variants to properties. Method extraction occurs within impl_item nodes, with methods stored in a methodsByType map for attachment to their corresponding types.

Import Handling

The use_declaration parser supports simple paths, scoped identifiers (std::collections::HashMap), grouped lists (std::io::{Read, Write}), and wildcards (std::prelude::*).

Call Expression Resolution

Call-graph extraction identifies call_expression nodes and resolves callee names handling plain identifiers, field expressions (self.validate), and scoped identifiers (Vec::new).

Java Class and Method Extraction

The Java extractor in packages/core/src/plugins/extractors/java-extractor.ts processes class-based object-oriented structures with explicit visibility modifiers.

Class and Member Analysis

The extractor handles import_declaration, class_declaration, and interface_declaration as top-level nodes. Both classes and interfaces are stored as class entries. Methods and constructors are extracted from method_declaration and constructor_declaration nodes, added to both the class's method list and the global functions list. Fields from field_declaration nodes become properties.

Visibility and Parameters

Export status is determined by checking modifiers children for the public keyword via hasModifier. Parameters are extracted from formal_parameter and spread_parameter nodes. Return types come from the type field of method declarations.

Call Graph Edges

The extractor tracks method declarations on a stack and generates call relationships from method_invocation nodes (capturing object.method patterns when present) and object_creation_expression nodes (recording new <type> callees).

Integration in the Analysis Pipeline

Language extractors fit into a four-stage pipeline orchestrated by the TreeSitterPlugin in packages/core/src/plugins/tree-sitter-plugin.ts:

1. Language Detection: The plugin matches file extensions to languageIds arrays to select the appropriate extractor.
2. AST Creation: Tree-Sitter parsers generate language-specific ASTs from source files.
3. Extraction: The selected extractor runs extractStructure and extractCallGraph, producing normalized data structures.
4. Graph Assembly: The PluginRegistry merges results across files, deduplicates exports, and resolves import relationships into the final knowledge graph.

Practical Usage Examples

Direct Extractor Usage

Analyze TypeScript code directly using the extractor:

import { TypeScriptExtractor } from "./extractors/typescript-extractor.js";
import { Parser } from "web-tree-sitter";

async function analyzeTs(source: string) {
  await Parser.init();
  const parser = new Parser();
  parser.setLanguage(await Parser.Language.load(
    "node_modules/tree-sitter-typescript/tsx.wasm"
  ));

  const tree = parser.parse(source);
  const extractor = new TypeScriptExtractor();
  
  const structure = extractor.extractStructure(tree.rootNode);
  const callGraph = extractor.extractCallGraph(tree.rootNode);

  console.log("Functions:", structure.functions);
  console.log("Calls:", callGraph);
}

Full Pipeline Execution

Run the complete analysis pipeline across a project:

import { TreeSitterPlugin } from "./plugins/tree-sitter-plugin.js";
import { PluginRegistry } from "./plugins/registry.js";

async function scanProject(projectPath: string) {
  const registry = new PluginRegistry();
  const treeSitter = new TreeSitterPlugin();

  await registry.register(treeSitter);
  const graph = await registry.analyzeDirectory(projectPath);

  console.log("Graph nodes:", graph.nodes.length);
  console.log("Call edges:", graph.edges.filter(e => e.type === "call").length);
}

Summary

• LanguageExtractor Interface: All extractors implement extractStructure and extractCallGraph methods operating on Tree-Sitter ASTs.
• TypeScript/JavaScript: Handles export_statement and call_expression nodes, supporting both languages via the same extractor in typescript-extractor.ts.
• Python: Treats all top-level definitions as exports and unwraps decorated_definition nodes to process functions and classes.
• Go: Uses capitalization-based export detection and links methods to receiver types via methodsByReceiver mapping.
• Rust: Checks visibility_modifier for pub keywords and supports complex use_declaration import patterns.
• Java: Detects public modifiers in modifiers nodes and handles method_invocation and object_creation_expression for call graphs.

Frequently Asked Questions

How does Understand Anything handle language-specific visibility rules?

Each extractor implements language-specific logic: Go checks for uppercase names in go-extractor.ts, Rust looks for visibility_modifier nodes starting with pub in rust-extractor.ts, and Java checks modifiers children for the public keyword in java-extractor.ts. Python lacks explicit exports, so the extractor treats all top-level definitions as exported.

Can I use the extractors independently of the full Understand Anything pipeline?

Yes, as shown in the direct usage example. Each extractor can be instantiated and called directly with a Tree-Sitter rootNode, allowing you to obtain StructuralAnalysis and CallGraphEntry objects without running the complete PluginRegistry pipeline.

How are method calls resolved in languages with receivers like Go and Java?

The Go extractor in go-extractor.ts extracts receiver types from the receiver field and links methods to their types via a methodsByReceiver map. The Java extractor tracks method declarations on a stack and resolves method_invocation nodes to object.method callees when an object is present, while handling object_creation_expression for constructor calls.

What Tree-Sitter grammars are required for the extractors to function?

The TreeSitterPlugin registers grammars for TypeScript/JavaScript, Python, Go, Rust, and Java. Each extractor references language-specific WASM files (e.g., tree-sitter-typescript/tsx.wasm for TypeScript) that must be available in the node_modules or specified load path for the parser to generate valid ASTs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}