How Worker Pools in GitNexus Parallelize Parsing for Indexing Performance

GitNexus accelerates repository indexing by distributing Tree-sitter parsing across a configurable pool of Node.js worker threads that chunk files into sub-batches, enforce 30-second timeouts, and stream results back to the main thread for aggregation.

GitNexus is an open-source code intelligence platform that indexes large repositories by extracting symbols, imports, and call graphs from source files. To handle the CPU-intensive work of parsing millions of lines of code without blocking the main thread, GitNexus implements a sophisticated worker pool architecture that parallelizes parsing across multiple CPU cores while maintaining strict memory and timeout constraints.

Core Architecture of the GitNexus Worker Pool

The parallel parsing system is built around three specialized modules that handle orchestration, execution, and integration.

worker-pool.ts: The Orchestrator

Located at src/core/ingestion/workers/worker-pool.ts, this module exports createWorkerPool(), which manages the lifecycle of Node.js Worker instances. It handles chunking logic, sub-batch streaming, progress tracking, timeout enforcement, and result aggregation.

parse-worker.ts: The Parser Thread

The src/core/ingestion/workers/parse-worker.ts script runs inside each worker thread. It receives ParseWorkerInput sub-batches, parses files using Tree-sitter, extracts symbols, imports, and call graphs, then streams partial results back to the main thread via parentPort.postMessage().

parsing-processor.ts: The Integration Layer

Found at src/core/ingestion/parsing-processor.ts, this high-level module bridges the worker pool with the ingestion pipeline. It attempts to create a WorkerPool, falls back to sequential parsing if workers are unavailable, and merges the final WorkerExtractedData into the index.

How Worker Pools Parallelize Parsing in GitNexus

The system achieves performance through four coordinated strategies: dynamic worker creation, memory-bounded sub-batch streaming, strict timeout enforcement, and ordered result aggregation.

Dynamic Worker Creation and Chunking Strategy

When createWorkerPool(workerUrl, poolSize?) is invoked, it calculates the optimal pool size defaulting to min(8, cpus‑1) to avoid overwhelming the system:

// src/core/ingestion/workers/worker-pool.ts
const size = poolSize ?? Math.min(8, Math.max(1, os.cpus().length - 1));
for (let i = 0; i < size; i++) {
  workers.push(new Worker(workerUrl));
}

The dispatch method then splits the incoming file array into equal chunks using Math.ceil(items.length / size), ensuring each worker receives a balanced workload.

Sub-Batch Streaming for Memory Efficiency

To prevent memory exhaustion when processing massive repositories, GitNexus implements sub-batch streaming with a fixed SUB_BATCH_SIZE of 1500 files. Instead of sending an entire chunk at once, the main thread streams sub-batches to each worker:

// src/core/ingestion/workers/worker-pool.ts
const subBatch = chunk.slice(start, start + SUB_BATCH_SIZE);
worker.postMessage({ type: 'sub-batch', files: subBatch });

The worker processes each sub-batch, accumulates results locally, and signals completion with sub-batch-done. This pattern keeps the structured-clone payload small and bounds memory usage per worker.

Timeout Handling and Fault Isolation

Each sub-batch operation is guarded by a SUB_BATCH_TIMEOUT_MS of 30 seconds. If a worker hangs—perhaps due to a pathological 50 MB minified file—the timer fires, rejecting the promise and aborting that specific worker without crashing the main thread:

// src/core/ingestion/workers/worker-pool.ts
subBatchTimer = setTimeout(() => {
  if (!settled) {
    settled = true;
    cleanup();
    reject(new Error(`Worker ${i} sub-batch timed out after ${SUB_BATCH_TIMEOUT_MS / 1000}s`));
  }
}, SUB_BATCH_TIMEOUT_MS);

This ensures that a single malformed file cannot stall the entire indexing pipeline.

Result Aggregation and Ordering

As workers complete sub-batches, the main thread merges partial results into the final output array. The dispatch method maintains order by resolving worker promises in sequence, ensuring the concatenated results align with the original file list order:

// src/core/ingestion/workers/worker-pool.ts
if (msg && msg.type === 'result') {
  settled = true;
  cleanup();
  resolve(msg.data);
}

Once all workers resolve, the function returns the complete WorkerExtractedData array to the ingestion pipeline.

Implementation Examples

Manual Worker Pool Creation

For custom parsing tasks, you can instantiate the worker pool directly:

import { createWorkerPool, WorkerPool } from '@/core/ingestion/workers/worker-pool.js';
import { pathToFileURL } from 'node:url';
import * as path from 'node:path';

const workerPath = path.resolve('dist', 'core', 'ingestion', 'workers', 'parse-worker.js');
const pool: WorkerPool = createWorkerPool(pathToFileURL(workerPath));

const files = [{ path: 'src/index.ts', content: '...' }];
const results = await pool.dispatch<typeof files[0], ParseWorkerResult>(
  files,
  (processed) => console.log(`Processed ${processed} files`)
);

await pool.terminate();

High-Level Parsing Processor

For standard repository indexing, use the built-in processor:

import { processParsing } from '@/core/ingestion/parsing-processor.js';
import { readRepositoryFiles } from '@/utils/file-utils.js';

const repoPath = '/path/to/repo';
const files = await readRepositoryFiles(repoPath);
const { nodes, imports, calls } = await processParsing(files);

console.log(`Indexed ${nodes.length} symbols from ${files.length} source files`);

Performance Benefits of Parallel Parsing

The worker pool architecture delivers measurable improvements for large-scale indexing:

• CPU Utilization: Distributes Tree-sitter parsing across min(8, cpus-1) cores, saturating available CPU without overwhelming the system.
• Memory Bounds: Sub-batch streaming caps each worker's memory footprint at 1500 files per iteration, preventing heap exhaustion on repositories with millions of lines.
• Fault Isolation: 30-second timeouts per sub-batch ensure pathological files cannot stall the entire indexing pipeline.
• Scalability: Linear throughput scaling up to the configured pool size, with automatic fallback to sequential processing in resource-constrained environments.

Summary

GitNexus leverages a sophisticated worker pool system to parallelize CPU-intensive parsing operations across multiple Node.js worker threads. The architecture centers on three core modules: worker-pool.ts for orchestration, parse-worker.ts for execution, and parsing-processor.ts for integration. By implementing dynamic chunking, sub-batch streaming with a 1500-file limit, and 30-second timeout guards, the system achieves high-throughput indexing while maintaining memory bounds and fault isolation. Developers can interact with this system through either low-level pool management or the high-level processParsing API.

Frequently Asked Questions

What is the default worker pool size in GitNexus?

The default pool size is calculated as min(8, cpus - 1), where cpus represents the number of logical CPU cores available on the system. This default is implemented in worker-pool.ts to balance parallel throughput with system stability, ensuring the main thread and other system processes retain adequate CPU resources.

How does GitNexus handle memory usage during parallel parsing?

GitNexus implements sub-batch streaming to bound memory consumption. Rather than sending an entire workload chunk to a worker at once, the system streams sub-batches of maximum 1500 files (SUB_BATCH_SIZE). This approach keeps structured-clone payloads small and ensures each worker processes files iteratively, preventing heap exhaustion when indexing massive repositories.

What happens if a worker thread times out?

If a worker fails to process a sub-batch within the 30-second timeout window (SUB_BATCH_TIMEOUT_MS), the main thread rejects the promise for that specific worker, aborts the hung worker, and continues processing with the remaining workers. This fault-isolation mechanism ensures that pathological files—such as minified JavaScript bundles—cannot stall the entire indexing pipeline.

Can I use the worker pool for custom parsing tasks?

Yes, the worker-pool.ts module exports createWorkerPool() for direct instantiation, allowing developers to implement custom parsing logic by providing their own worker script URL. Alternatively, for standard repository indexing, the high-level processParsing() function in parsing-processor.ts abstracts the pool management entirely, automatically handling worker creation, dispatch, and result aggregation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}