# How GitNexus's Multi-Phase Indexing Pipeline Works: From Files to Knowledge Graphs

> Discover how GitNexus's multi-phase indexing pipeline transforms repository files into actionable knowledge. Learn about its six-phase process including scanning, parsing, and community detection.

- Repository: [Abhigyan Patwari/GitNexus](https://github.com/abhigyanpatwari/GitNexus)
- Tags: internals
- Published: 2026-03-08

---

**GitNexus indexes repositories through a deterministic six-phase pipeline that scans files, builds structure, parses source code in memory-bounded chunks, detects communities using the Leiden algorithm, and infers execution processes, orchestrated by `runPipelineFromRepo` in [`gitnexus/src/core/ingestion/pipeline.ts`](https://github.com/abhigyanpatwari/GitNexus/blob/main/gitnexus/src/core/ingestion/pipeline.ts).**

GitNexus transforms raw Git repositories into queryable knowledge graphs using a sophisticated multi-phase indexing pipeline. This deterministic ingestion system processes codebases of any size—from small libraries to massive monorepos like the Linux kernel—while maintaining strict memory bounds. The pipeline is implemented in the GitNexus open-source repository and exposed through the `gn analyze` CLI command.

## The Six Phases of the GitNexus Multi-Phase Indexing Pipeline

The pipeline orchestrated by `runPipelineFromRepo` proceeds through six logical phases, each with distinct responsibilities and progress reporting.

| Phase | Purpose | Key Operations | Progress Reporting |
|-------|---------|----------------|--------------------|
| **1 – Scan** | Walk the filesystem without opening files. | `walkRepositoryPaths` collects every file path and size. | Emits *extracting* progress 0 → 15 %. |
| **2 – Structure** | Record the project's directory hierarchy. | `processStructure` adds `File` nodes and `CONTAINS` relationships for every path. | Emits *structure* progress 15 → 20 %. |
| **3-4 – Chunked Read & Parse** | Load source files in memory-friendly *byte-budget* chunks, parse them, and extract ASTs. | • Split parse-able files into chunks limited by `CHUNK_BYTE_BUDGET` (20 MiB).<br>• Create a **worker pool** (`createWorkerPool`) that runs [`parse-worker.js`](https://github.com/abhigyanpatwari/GitNexus/blob/main/parse-worker.js) in parallel.<br>• For each chunk: `readFileContents` → `processParsing` → produce imports, calls, heritage, routes. | Emits *parsing* progress 20 → 82 % (dynamic based on files parsed). |
| **5 – Communities** | Detect clusters of related symbols using the Leiden algorithm. | `processCommunities` runs on the partially built graph and adds `Community` nodes and `MEMBER_OF` relationships. | Emits *communities* progress 82 → 94 %. |
| **6 – Processes** | Infer high-level execution flows (processes) from the graph. | `processProcesses` uses community memberships to trace possible call-chains, creating `Process` nodes and `STEP_IN_PROCESS` edges. | Emits *processes* progress 94 → 100 %. |

### Phase 1: Scan

The **Scan** phase walks the filesystem without opening files. The `walkRepositoryPaths` function collects every file path and size, emitting *extracting* progress from 0% to 15%. This initial pass provides the complete inventory needed for subsequent memory budgeting.

### Phase 2: Structure

The **Structure** phase records the project's directory hierarchy. The `processStructure` function adds `File` nodes and `CONTAINS` relationships for every path discovered during the scan. This phase emits *structure* progress from 15% to 20%, establishing the foundational graph topology before code analysis begins.

### Phase 3-4: Chunked Read and Parse

Phases 3 and 4 handle **Chunked Read and Parse**, the most computationally intensive stage. This phase loads source files in memory-friendly *byte-budget* chunks and extracts Abstract Syntax Trees (ASTs).

Key operations include:

- **Memory Budgeting**: Parse-able files are split into chunks limited by `CHUNK_BYTE_BUDGET` (20 MiB), ensuring memory usage stays bounded even for massive repositories.
- **Worker Pool**: A `createWorkerPool` instance runs [`parse-worker.js`](https://github.com/abhigyanpatwari/GitNexus/blob/main/parse-worker.js) in parallel for dramatic speedup, with automatic fallback to sequential mode if worker creation fails.
- **Per-Chunk Processing**: For each chunk, `readFileContents` reads only the files in that chunk, then `processParsing` parses them to produce imports, calls, heritage, and routes.

This phase emits *parsing* progress from 20% to 82%, dynamically calculated based on files parsed.

### Phase 5: Communities

The **Communities** phase detects clusters of related symbols using the **Leiden algorithm**. The `processCommunities` function runs on the partially built graph, adding `Community` nodes and `MEMBER_OF` relationships. This phase emits *communities* progress from 82% to 94%.

### Phase 6: Processes

The **Processes** phase infers high-level execution flows from the graph. The `processProcesses` function uses community memberships to trace possible call-chains, creating `Process` nodes and `STEP_IN_PROCESS` edges. This phase emits *processes* progress from 94% to 100%.

Upon completion, the pipeline clears the AST cache and returns `{ graph, repoPath, totalFileCount, communityResult, processResult }`.

## Memory-Constrained Architecture and Performance Optimizations

GitNexus implements several engineering strategies in [`gitnexus/src/core/ingestion/pipeline.ts`](https://github.com/abhigyanpatwari/GitNexus/blob/main/gitnexus/src/core/ingestion/pipeline.ts) to handle repositories of arbitrary size without exhausting system memory.

### Memory-Constrained Chunking

The `CHUNK_BYTE_BUDGET` constant (set to 20 MiB) limits the total source size kept in RAM for a single parse batch. Files are grouped into chunks (lines 123-138) so memory usage stays bounded even for massive repos like the Linux kernel.

### Worker Pool and Fallback

A `WorkerPool` is created once (lines 52-66) to run [`parse-worker.js`](https://github.com/abhigyanpatwari/GitNexus/blob/main/parse-worker.js) in parallel, dramatically speeding up parsing. If worker creation fails (e.g., during tests), the pipeline falls back to a sequential parser, guaranteeing correctness without parallelization.

### AST Cache

`createASTCache(AST_CACHE_CAP)` (line 28) stores parsed ASTs for reuse within a chunk. The cache is cleared after each chunk (lines 36-38) to free memory before processing the next batch.

### Import Resolution Context

Built once via `buildImportResolutionContext` (lines 76-78) and shared across all chunks, this context avoids repeated O(files × pathDepth) work. After the main phases, the context is explicitly cleared (lines 56-62) to release approximately 100 MiB for large repositories.

### Incremental Graph Building

The pipeline builds the graph incrementally:

- **Structure** creates file nodes early, providing a stable foundation.
- **Parsing** adds symbols (`Function`, `Class`, etc.) via `processParsing`.
- **Import/Call/Heritage** processors operate on extracted data immediately for each chunk, eliminating the need for a second pass.

## Running the Pipeline Programmatically

You can execute the full ingestion pipeline from TypeScript code using the same entry point as the `gn analyze` CLI command.

```typescript
import { runPipelineFromRepo } from './src/cli/analyze.ts';

// Path to a local Git repository
const repoPath = '/path/to/project';

// Progress callback that prints a simple text-based progress bar
function onProgress(p) {
  console.log(`[${p.phase}] ${p.percent}% – ${p.message}`);
}

// Execute the full ingestion pipeline once
runPipelineFromRepo(repoPath, onProgress)
  .then(result => {
    console.log('✅ Pipeline finished');
    console.log(`Graph contains ${result.graph.nodeCount} nodes`);
    // You can now query the graph, export it, or run downstream tools.
  })
  .catch(err => {
    console.error('❌ Pipeline failed:', err);
  });

```

The `runPipelineFromRepo` function defined in [`gitnexus/src/core/ingestion/pipeline.ts`](https://github.com/abhigyanpatwari/GitNexus/blob/main/gitnexus/src/core/ingestion/pipeline.ts) returns a promise that resolves with an object containing the constructed graph, repository path, total file count, community detection results, and process inference results.

## Key Source Files

The multi-phase indexing pipeline is implemented across the following files in the GitNexus repository:

| File | Role |
|------|------|
| [`gitnexus/src/core/ingestion/pipeline.ts`](https://github.com/abhigyanpatwari/GitNexus/blob/main/gitnexus/src/core/ingestion/pipeline.ts) | Orchestrates all six phases of the ingestion pipeline. |
| [`gitnexus/src/core/graph/graph.ts`](https://github.com/abhigyanpatwari/GitNexus/blob/main/gitnexus/src/core/graph/graph.ts) | Creates the Neo4j-style in-memory graph structure. |
| [`gitnexus/src/core/ingestion/structure-processor.ts`](https://github.com/abhigyanpatwari/GitNexus/blob/main/gitnexus/src/core/ingestion/structure-processor.ts) | Builds file-node hierarchy and `CONTAINS` relationships. |
| [`gitnexus/src/core/ingestion/parsing-processor.ts`](https://github.com/abhigyanpatwari/GitNexus/blob/main/gitnexus/src/core/ingestion/parsing-processor.ts) | Handles chunked parsing using worker pools. |
| [`gitnexus/src/core/ingestion/import-processor.ts`](https://github.com/abhigyanpatwari/GitNexus/blob/main/gitnexus/src/core/ingestion/import-processor.ts) | Resolves imports and builds the import map. |
| [`gitnexus/src/core/ingestion/call-processor.ts`](https://github.com/abhigyanpatwari/GitNexus/blob/main/gitnexus/src/core/ingestion/call-processor.ts) | Extracts and links function calls between symbols. |
| [`gitnexus/src/core/ingestion/heritage-processor.ts`](https://github.com/abhigyanpatwari/GitNexus/blob/main/gitnexus/src/core/ingestion/heritage-processor.ts) | Resolves class inheritance and interface implementation. |
| [`gitnexus/src/core/ingestion/community-processor.ts`](https://github.com/abhigyanpatwari/GitNexus/blob/main/gitnexus/src/core/ingestion/community-processor.ts) | Implements Leiden community detection. |
| [`gitnexus/src/core/ingestion/process-processor.ts`](https://github.com/abhigyanpatwari/GitNexus/blob/main/gitnexus/src/core/ingestion/process-processor.ts) | Infers high-level execution processes and call chains. |
| [`gitnexus/src/cli/analyze.ts`](https://github.com/abhigyanpatwari/GitNexus/blob/main/gitnexus/src/cli/analyze.ts) | CLI wrapper that invokes `runPipelineFromRepo`. |
| [`gitnexus/src/core/ingestion/workers/parse-worker.js`](https://github.com/abhigyanpatwari/GitNexus/blob/main/gitnexus/src/core/ingestion/workers/parse-worker.js) | Worker script for parallel AST parsing. |
| [`gitnexus/src/types/pipeline.ts`](https://github.com/abhigyanpatwari/GitNexus/blob/main/gitnexus/src/types/pipeline.ts) | TypeScript definitions for progress and result objects. |

## Summary

- GitNexus uses a **deterministic six-phase pipeline** orchestrated by `runPipelineFromRepo` in [`gitnexus/src/core/ingestion/pipeline.ts`](https://github.com/abhigyanpatwari/GitNexus/blob/main/gitnexus/src/core/ingestion/pipeline.ts) to transform repositories into knowledge graphs.
- The pipeline progresses through **Scan**, **Structure**, **Chunked Read & Parse**, **Communities**, and **Processes**, with granular progress reporting at each stage.
- **Memory safety** is enforced via a 20 MiB `CHUNK_BYTE_BUDGET`, AST caching with periodic clearing, and explicit cleanup of import resolution contexts.
- **Parallel processing** utilizes a `WorkerPool` running [`parse-worker.js`](https://github.com/abhigyanpatwari/GitNexus/blob/main/parse-worker.js) for multi-core parsing, with automatic fallback to sequential mode.
- The architecture supports **incremental graph building**, where file nodes are created early and symbol relationships (imports, calls, heritage) are attached in subsequent phases.

## Frequently Asked Questions

### How does GitNexus handle repositories too large to fit in memory?

GitNexus implements **memory-constrained chunking** with a `CHUNK_BYTE_BUDGET` of 20 MiB. The `processParsing` function in [`gitnexus/src/core/ingestion/parsing-processor.ts`](https://github.com/abhigyanpatwari/GitNexus/blob/main/gitnexus/src/core/ingestion/parsing-processor.ts) groups files into chunks that fit within this budget, processing each batch independently while clearing the AST cache between chunks. This allows GitNexus to index massive codebases like the Linux kernel without exhausting RAM.

### What happens if the worker pool fails to initialize during parsing?

If the `WorkerPool` creation fails (lines 52-66 in [`gitnexus/src/core/ingestion/pipeline.ts`](https://github.com/abhigyanpatwari/GitNexus/blob/main/gitnexus/src/core/ingestion/pipeline.ts)), the pipeline automatically falls back to **sequential parsing mode**. This fallback ensures the ingestion completes correctly even in environments that restrict worker threads, such as certain CI/CD pipelines or test environments, without requiring manual configuration changes.

### How does the pipeline determine progress percentages?

Progress reporting is calculated dynamically based on phase weights defined in [`gitnexus/src/core/ingestion/pipeline.ts`](https://github.com/abhigyanpatwari/GitNexus/blob/main/gitnexus/src/core/ingestion/pipeline.ts). The **Scan** phase occupies 0-15%, **Structure** 15-20%, **Parsing** 20-82% (scaled by `globalCurrent / totalParseable`), **Communities** 82-94%, and **Processes** 94-100%. The `onProgress` callback receives these values along with phase names and status messages to render progress bars in the CLI.

### What is the difference between the Communities and Processes phases?

The **Communities** phase (phase 5) uses the **Leiden algorithm** in [`gitnexus/src/core/ingestion/community-processor.ts`](https://github.com/abhigyanpatwari/GitNexus/blob/main/gitnexus/src/core/ingestion/community-processor.ts) to detect clusters of tightly coupled symbols, creating `Community` nodes and `MEMBER_OF` relationships. The **Processes** phase (phase 6) then uses these community memberships in [`gitnexus/src/core/ingestion/process-processor.ts`](https://github.com/abhigyanpatwari/GitNexus/blob/main/gitnexus/src/core/ingestion/process-processor.ts) to trace possible execution call-chains, creating `Process` nodes and `STEP_IN_PROCESS` edges that represent high-level application workflows.