How GitNexus's Multi-Phase Indexing Pipeline Works: From Files to Knowledge Graphs

GitNexus indexes repositories through a deterministic six-phase pipeline that scans files, builds structure, parses source code in memory-bounded chunks, detects communities using the Leiden algorithm, and infers execution processes, orchestrated by runPipelineFromRepo in gitnexus/src/core/ingestion/pipeline.ts.

GitNexus transforms raw Git repositories into queryable knowledge graphs using a sophisticated multi-phase indexing pipeline. This deterministic ingestion system processes codebases of any size—from small libraries to massive monorepos like the Linux kernel—while maintaining strict memory bounds. The pipeline is implemented in the GitNexus open-source repository and exposed through the gn analyze CLI command.

The Six Phases of the GitNexus Multi-Phase Indexing Pipeline

The pipeline orchestrated by runPipelineFromRepo proceeds through six logical phases, each with distinct responsibilities and progress reporting.

Phase	Purpose	Key Operations	Progress Reporting
1 – Scan	Walk the filesystem without opening files.	walkRepositoryPaths collects every file path and size.	Emits extracting progress 0 → 15 %.
2 – Structure	Record the project's directory hierarchy.	processStructure adds File nodes and CONTAINS relationships for every path.	Emits structure progress 15 → 20 %.
3-4 – Chunked Read & Parse	Load source files in memory-friendly byte-budget chunks, parse them, and extract ASTs.	• Split parse-able files into chunks limited by CHUNK_BYTE_BUDGET (20 MiB).• Create a worker pool (createWorkerPool) that runs parse-worker.js in parallel.• For each chunk: readFileContents → processParsing → produce imports, calls, heritage, routes.	Emits parsing progress 20 → 82 % (dynamic based on files parsed).
5 – Communities	Detect clusters of related symbols using the Leiden algorithm.	processCommunities runs on the partially built graph and adds Community nodes and MEMBER_OF relationships.	Emits communities progress 82 → 94 %.
6 – Processes	Infer high-level execution flows (processes) from the graph.	processProcesses uses community memberships to trace possible call-chains, creating Process nodes and STEP_IN_PROCESS edges.	Emits processes progress 94 → 100 %.

Phase 1: Scan

The Scan phase walks the filesystem without opening files. The walkRepositoryPaths function collects every file path and size, emitting extracting progress from 0% to 15%. This initial pass provides the complete inventory needed for subsequent memory budgeting.

Phase 2: Structure

The Structure phase records the project's directory hierarchy. The processStructure function adds File nodes and CONTAINS relationships for every path discovered during the scan. This phase emits structure progress from 15% to 20%, establishing the foundational graph topology before code analysis begins.

Phase 3-4: Chunked Read and Parse

Phases 3 and 4 handle Chunked Read and Parse, the most computationally intensive stage. This phase loads source files in memory-friendly byte-budget chunks and extracts Abstract Syntax Trees (ASTs).

Key operations include:

• Memory Budgeting: Parse-able files are split into chunks limited by CHUNK_BYTE_BUDGET (20 MiB), ensuring memory usage stays bounded even for massive repositories.
• Worker Pool: A createWorkerPool instance runs parse-worker.js in parallel for dramatic speedup, with automatic fallback to sequential mode if worker creation fails.
• Per-Chunk Processing: For each chunk, readFileContents reads only the files in that chunk, then processParsing parses them to produce imports, calls, heritage, and routes.

This phase emits parsing progress from 20% to 82%, dynamically calculated based on files parsed.

Phase 5: Communities

The Communities phase detects clusters of related symbols using the Leiden algorithm. The processCommunities function runs on the partially built graph, adding Community nodes and MEMBER_OF relationships. This phase emits communities progress from 82% to 94%.

Phase 6: Processes

The Processes phase infers high-level execution flows from the graph. The processProcesses function uses community memberships to trace possible call-chains, creating Process nodes and STEP_IN_PROCESS edges. This phase emits processes progress from 94% to 100%.

Upon completion, the pipeline clears the AST cache and returns { graph, repoPath, totalFileCount, communityResult, processResult }.

Memory-Constrained Architecture and Performance Optimizations

GitNexus implements several engineering strategies in gitnexus/src/core/ingestion/pipeline.ts to handle repositories of arbitrary size without exhausting system memory.

Memory-Constrained Chunking

The CHUNK_BYTE_BUDGET constant (set to 20 MiB) limits the total source size kept in RAM for a single parse batch. Files are grouped into chunks (lines 123-138) so memory usage stays bounded even for massive repos like the Linux kernel.

Worker Pool and Fallback

A WorkerPool is created once (lines 52-66) to run parse-worker.js in parallel, dramatically speeding up parsing. If worker creation fails (e.g., during tests), the pipeline falls back to a sequential parser, guaranteeing correctness without parallelization.

AST Cache

createASTCache(AST_CACHE_CAP) (line 28) stores parsed ASTs for reuse within a chunk. The cache is cleared after each chunk (lines 36-38) to free memory before processing the next batch.

Import Resolution Context

Built once via buildImportResolutionContext (lines 76-78) and shared across all chunks, this context avoids repeated O(files × pathDepth) work. After the main phases, the context is explicitly cleared (lines 56-62) to release approximately 100 MiB for large repositories.

Incremental Graph Building

The pipeline builds the graph incrementally:

• Structure creates file nodes early, providing a stable foundation.
• Parsing adds symbols (Function, Class, etc.) via processParsing.
• Import/Call/Heritage processors operate on extracted data immediately for each chunk, eliminating the need for a second pass.

Running the Pipeline Programmatically

You can execute the full ingestion pipeline from TypeScript code using the same entry point as the gn analyze CLI command.

import { runPipelineFromRepo } from './src/cli/analyze.ts';

// Path to a local Git repository
const repoPath = '/path/to/project';

// Progress callback that prints a simple text-based progress bar
function onProgress(p) {
  console.log(`[${p.phase}] ${p.percent}% – ${p.message}`);
}

// Execute the full ingestion pipeline once
runPipelineFromRepo(repoPath, onProgress)
  .then(result => {
    console.log('✅ Pipeline finished');
    console.log(`Graph contains ${result.graph.nodeCount} nodes`);
    // You can now query the graph, export it, or run downstream tools.
  })
  .catch(err => {
    console.error('❌ Pipeline failed:', err);
  });

The runPipelineFromRepo function defined in gitnexus/src/core/ingestion/pipeline.ts returns a promise that resolves with an object containing the constructed graph, repository path, total file count, community detection results, and process inference results.

Key Source Files

The multi-phase indexing pipeline is implemented across the following files in the GitNexus repository:

File	Role
gitnexus/src/core/ingestion/pipeline.ts	Orchestrates all six phases of the ingestion pipeline.
gitnexus/src/core/graph/graph.ts	Creates the Neo4j-style in-memory graph structure.
gitnexus/src/core/ingestion/structure-processor.ts	Builds file-node hierarchy and CONTAINS relationships.
gitnexus/src/core/ingestion/parsing-processor.ts	Handles chunked parsing using worker pools.
gitnexus/src/core/ingestion/import-processor.ts	Resolves imports and builds the import map.
gitnexus/src/core/ingestion/call-processor.ts	Extracts and links function calls between symbols.
gitnexus/src/core/ingestion/heritage-processor.ts	Resolves class inheritance and interface implementation.
gitnexus/src/core/ingestion/community-processor.ts	Implements Leiden community detection.
gitnexus/src/core/ingestion/process-processor.ts	Infers high-level execution processes and call chains.
gitnexus/src/cli/analyze.ts	CLI wrapper that invokes runPipelineFromRepo.
gitnexus/src/core/ingestion/workers/parse-worker.js	Worker script for parallel AST parsing.
gitnexus/src/types/pipeline.ts	TypeScript definitions for progress and result objects.

Summary

• GitNexus uses a deterministic six-phase pipeline orchestrated by runPipelineFromRepo in gitnexus/src/core/ingestion/pipeline.ts to transform repositories into knowledge graphs.
• The pipeline progresses through Scan, Structure, Chunked Read & Parse, Communities, and Processes, with granular progress reporting at each stage.
• Memory safety is enforced via a 20 MiB CHUNK_BYTE_BUDGET, AST caching with periodic clearing, and explicit cleanup of import resolution contexts.
• Parallel processing utilizes a WorkerPool running parse-worker.js for multi-core parsing, with automatic fallback to sequential mode.
• The architecture supports incremental graph building, where file nodes are created early and symbol relationships (imports, calls, heritage) are attached in subsequent phases.

Frequently Asked Questions

How does GitNexus handle repositories too large to fit in memory?

GitNexus implements memory-constrained chunking with a CHUNK_BYTE_BUDGET of 20 MiB. The processParsing function in gitnexus/src/core/ingestion/parsing-processor.ts groups files into chunks that fit within this budget, processing each batch independently while clearing the AST cache between chunks. This allows GitNexus to index massive codebases like the Linux kernel without exhausting RAM.

What happens if the worker pool fails to initialize during parsing?

If the WorkerPool creation fails (lines 52-66 in gitnexus/src/core/ingestion/pipeline.ts), the pipeline automatically falls back to sequential parsing mode. This fallback ensures the ingestion completes correctly even in environments that restrict worker threads, such as certain CI/CD pipelines or test environments, without requiring manual configuration changes.

How does the pipeline determine progress percentages?

Progress reporting is calculated dynamically based on phase weights defined in gitnexus/src/core/ingestion/pipeline.ts. The Scan phase occupies 0-15%, Structure 15-20%, Parsing 20-82% (scaled by globalCurrent / totalParseable), Communities 82-94%, and Processes 94-100%. The onProgress callback receives these values along with phase names and status messages to render progress bars in the CLI.

What is the difference between the Communities and Processes phases?

The Communities phase (phase 5) uses the Leiden algorithm in gitnexus/src/core/ingestion/community-processor.ts to detect clusters of tightly coupled symbols, creating Community nodes and MEMBER_OF relationships. The Processes phase (phase 6) then uses these community memberships in gitnexus/src/core/ingestion/process-processor.ts to trace possible execution call-chains, creating Process nodes and STEP_IN_PROCESS edges that represent high-level application workflows.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}