# CodeGraph Concurrent Indexing Strategies: Parallel Processing Without Blocking the UI

> Discover CodeGraph's concurrent indexing strategies, including batched I/O parallelism and isolated workers, ensuring efficient background processing without UI blocking.

- Repository: [Colby Mchenry/codegraph](https://github.com/colbymchenry/codegraph)
- Tags: internals
- Published: 2026-05-17

---

**CodeGraph combines batched I/O parallelism, isolated worker-thread parsing, automatic worker recycling every 250 operations, and git-aware change detection to execute concurrent indexing efficiently while keeping the main thread responsive.**

CodeGraph, the open-source codebase analysis tool from `colbymchenry/codegraph`, implements a sophisticated pipeline for handling concurrent indexing operations in large repositories. By separating disk I/O from CPU-intensive parsing and isolating tree-sitter operations in dedicated workers, the system maximizes throughput without freezing the user interface. The following sections examine the specific source-level implementations that enable this efficient parallel processing.

## Parallel I/O Through Batch-Size Limiting

To prevent disk bottlenecks from stalling the CPU, CodeGraph implements **batch-size-limited parallel file reads**. The orchestrator defined in [`src/extraction/index.ts`](https://github.com/colbymchenry/codegraph/blob/main/src/extraction/index.ts) reads up to `FILE_IO_BATCH_SIZE` files simultaneously—set to **10** at line 32—creating an overlap between disk operations and parsing work.

This batching occurs within the `indexAll` method (lines 6023–6039), where the system queues file reads while previously fetched content undergoes processing. By maintaining this steady pipeline, the indexer keeps both the disk and CPU saturated without overwhelming memory through unbounded concurrency.

## Worker Thread Isolation for Tree-Sitter Parsing

CPU-intensive parsing is offloaded from the main thread using Node.js `worker_threads`. In [`src/extraction/index.ts`](https://github.com/colbymchenry/codegraph/blob/main/src/extraction/index.ts), the system spawns a dedicated worker via `new WorkerClass!(parseWorkerPath)` at lines 60–61, wrapping the tree-sitter WASM parser in an isolated execution context.

The `ensureWorker` function (lines 63–70) manages this worker's lifecycle, ensuring that message passing between the main thread and the parser stays asynchronous and non-blocking. This architecture guarantees that long-running parse operations or pathological edge cases cannot freeze the UI thread's progress updates.

## Defensive Timeouts and Worker Recycling

Because WebAssembly linear memory cannot shrink once grown, CodeGraph implements aggressive memory management through **timeouts and forced recycling**. Each parse request receives a dynamic timeout based on `PARSE_TIMEOUT_MS` (defined at lines 38–41) plus a size-based duration bump, implemented in `requestParse` (lines 86–94) to abort hanging parsers.

Additionally, workers are terminated and respawned after processing `WORKER_RECYCLE_INTERVAL` files—set to **250** at lines 49–50. This recycling check occurs inside `requestParse` (lines 75–79), preventing cumulative memory fragmentation from large files or memory leaks in the WASM boundary from crashing the indexing process.

## Graceful Fallbacks and In-Process Recovery

When worker threads are unavailable—such as in testing environments where the compiled worker might be missing—the system gracefully degrades. The orchestrator checks for worker existence (lines 62–64) and, if absent, branches to an in-process parsing path (lines 65–73) that loads grammars locally via [`src/extraction/grammars.ts`](https://github.com/colbymchenry/codegraph/blob/main/src/extraction/grammars.ts).

For resilience against WASM memory corruption, CodeGraph implements a retry loop starting at line 53054. Files causing worker crashes are automatically retried on a fresh worker instance; if parsing fails again, the system attempts a secondary pass after stripping comment-only lines (lines 53090–53124) to reduce memory pressure and eliminate potentially problematic syntax patterns.

## Git-Aware Change Detection

To minimize redundant work, the sync mechanism leverages repository history through `getGitChangedFiles` (lines 22–30 in [`src/sync/index.ts`](https://github.com/colbymchenry/codegraph/blob/main/src/sync/index.ts)). By executing `git status --porcelain`, the indexer identifies only added, modified, or removed files, limiting the indexing scope to actual changes rather than performing full filesystem walks.

For projects without Git initialization or when commands fail, the system falls back to a recursive directory scan (lines 68–71 in [`src/sync/index.ts`](https://github.com/colbymchenry/codegraph/blob/main/src/sync/index.ts)), still applying the configured include/exclude glob patterns to maintain efficiency.

## Responsive Progress Reporting Across Phases

The concurrent pipeline reports granular progress through the `IndexProgress` interface (lines 52–59), enabling UI components to render responsive progress bars. The `indexAll` method fires callbacks across distinct phases—`scanning`, `parsing`, `storing`, and `resolving`—allowing the interface to update while background operations continue.

This phase-aware reporting ensures users receive immediate feedback on indexing status even when the underlying worker thread is occupied with CPU-intensive tree-sitter operations.

## Using the Indexer

The following TypeScript example demonstrates triggering the complete concurrent pipeline:

```typescript
import { CodeGraph } from 'codegraph';
import { CodeGraphConfig } from 'codegraph/src/types';

const config: CodeGraphConfig = {
  include: ['**/*.ts', '**/*.tsx'],
  exclude: ['node_modules/**'],
  maxFileSize: 5_000_000, // 5 MB
};

const cg = new CodeGraph('/path/to/project', config, queries);
await cg.indexAll(
  (p) => console.log(`[${p.phase}] ${p.current}/${p.total} ${p.currentFile ?? ''}`),
);

```

This invocation automatically batches file reads, spawns the parse worker, recycles it after 250 parses, and handles the git-fast-path detection, all while invoking the progress callback asynchronously.

## Key Implementation Files

The concurrent indexing architecture spans several key modules:

- **[`src/extraction/index.ts`](https://github.com/colbymchenry/codegraph/blob/main/src/extraction/index.ts)** – Orchestrates scanning, parallel I/O, worker management, and retry logic.
- **[`src/extraction/tree-sitter.ts`](https://github.com/colbymchenry/codegraph/blob/main/src/extraction/tree-sitter.ts)** – Core tree-sitter parsing logic consumed by the worker thread.
- **[`src/extraction/grammars.ts`](https://github.com/colbymchenry/codegraph/blob/main/src/extraction/grammars.ts)** – Lazy loading of language grammars and WASM runtime initialization.
- **[`src/sync/index.ts`](https://github.com/colbymchenry/codegraph/blob/main/src/sync/index.ts)** – Implements fast-path git change detection (`getGitChangedFiles`) and fallback scanning.
- **[`src/db/queries.ts`](https://github.com/colbymchenry/codegraph/blob/main/src/db/queries.ts)** – Thread-safe SQLite write helpers for storing parsed results.
- **[`src/utils.ts`](https://github.com/colbymchenry/codegraph/blob/main/src/utils.ts)** and **[`src/types.ts`](https://github.com/colbymchenry/codegraph/blob/main/src/types.ts)** – Progress reporting utilities and type definitions.

## Summary

CodeGraph achieves efficient concurrent indexing through these core strategies:

- **Batched I/O parallelism** (`FILE_IO_BATCH_SIZE = 10`) overlaps disk reads with CPU processing to maximize resource utilization.
- **Worker thread isolation** prevents parse operations from blocking the main thread, keeping the UI responsive.
- **Defensive timeouts and recycling** (`PARSE_TIMEOUT_MS`, `WORKER_RECYCLE_INTERVAL = 250`) guard against memory exhaustion and hung WASM parsers.
- **Git-aware incremental sync** (`getGitChangedFiles`) limits indexing to changed files, avoiding redundant work.
- **Resilience patterns** including in-process fallback and comment-stripping retry logic ensure robustness against edge cases.
- **Phase-based progress reporting** provides real-time feedback without interrupting background processing.

## Frequently Asked Questions

### How does CodeGraph prevent the UI from freezing during large indexing operations?

CodeGraph offloads all tree-sitter parsing to a dedicated `worker_threads` worker spawned via `new WorkerClass!(parseWorkerPath)` in [`src/extraction/index.ts`](https://github.com/colbymchenry/codegraph/blob/main/src/extraction/index.ts) (lines 60–61). This isolates CPU-intensive WASM operations from the main thread, allowing the UI to continue processing events and rendering progress updates while parsing proceeds in the background.

### Why does CodeGraph recycle worker threads after exactly 250 parses?

Because WebAssembly linear memory cannot shrink once allocated, cumulative memory growth from parsing large files could eventually exhaust the heap. CodeGraph forces a worker restart every `WORKER_RECYCLE_INTERVAL` (250) parses as defined at lines 49–50 of [`src/extraction/index.ts`](https://github.com/colbymchenry/codegraph/blob/main/src/extraction/index.ts). This periodic recycling releases the entire WASM memory space and spawns a fresh worker, preventing slow memory leaks and fragmentation.

### What happens if the tree-sitter worker crashes while parsing a file?

The system implements a retry mechanism beginning at line 53054 of [`src/extraction/index.ts`](https://github.com/colbymchenry/codegraph/blob/main/src/extraction/index.ts). Files causing crashes are automatically retried on a freshly spawned worker. If the parse fails again, CodeGraph attempts a final fallback by stripping comment-only lines (lines 53090–53124) to reduce memory pressure and eliminate potentially malformed syntax before attempting one last parse.

### How does CodeGraph handle repositories that are not initialized with Git?

When `git status --porcelain` fails or returns no results, the sync mechanism falls back to a recursive filesystem scan (lines 68–71 in [`src/sync/index.ts`](https://github.com/colbymchenry/codegraph/blob/main/src/sync/index.ts)). This fallback still respects the include/exclude glob patterns defined in the configuration, ensuring that only relevant files are indexed even without version control metadata.