How Egonex-AI Manages Batch Processing for Very Large Codebases

Egonex-AI splits massive repositories into small batches of 20–30 files, processes up to five batches concurrently, and merges the partial results into a unified knowledge graph using a deterministic Python merger.

The Understand-Anything repository implements a sophisticated batch processing architecture designed to handle millions of lines of code across gigantic monorepos. Instead of attempting single-pass analysis that would overwhelm memory and compute resources, the system divides the workload into manageable units. This approach enables horizontal scaling through parallel agent execution while maintaining structural integrity during final assembly.

The Five-Stage Batch Processing Pipeline

The batch processing implementation follows a strict pipeline defined in the core TypeScript and Python modules. Each stage handles a specific transformation from raw source files to the final knowledge graph.

Stage 1: Project Scanning and Batch Configuration

The process begins in packages/core/src/languages/configs/batch.ts, where the batchConfig object defines the batch identifier (batchConfig.id === "batch"). The project scanner enumerates every source file in the repository and groups them into logical batches, typically containing 20 to 30 files per unit. This configuration ensures that memory consumption remains bounded regardless of total repository size.

Stage 2: Parallel File Analysis with Concurrent Agents

Once batches are defined, the system initiates parallel processing. The extract-structure.mjs entry point receives the list of batch files (batchFiles) and distributes them to file-analyzer agents. Up to 5 concurrent agents process each batch independently, producing individual JSON artifacts (batch-0.json, batch-1.json, etc.) containing nodes and edges discovered for that specific file group.

Stage 3: Intermediate JSON Storage

Each agent writes its results to .understand-anything/intermediate/ using a simple, deterministic naming scheme (batch-<N>.json). This deliberate naming convention allows the merger to locate partial results reliably without complex indexing. The intermediate storage acts as a checkpoint system, enabling recovery and incremental updates without reprocessing unchanged batches.

Stage 4: Merging and Normalization

The skills/understand/merge-batch-graphs.py script handles the critical consolidation phase. Its merge_and_normalize function reads all batch-*.json files from the intermediate directory, normalizes node IDs to prevent collisions, deduplicates edges that span multiple batches, and assembles a single coherent graph. The script also emits validation reports flagging any missing or malformed batches.

Stage 5: Final Knowledge Graph Assembly

The merged result is written to .understand-anything/knowledge-graph.json. From this point, the dashboard, search engine, and tour-builder operate on the complete graph regardless of the original segmentation. The packages/core/src/analyzer/graph-builder.ts module performs final ID normalization and resolves cross-batch edge references.

Why This Architecture Scales

The batch processing design in Egonex-AI addresses four critical scalability challenges:

• Memory Efficiency: Processing 20–30 files per worker keeps heap usage bounded even on repositories containing millions of files.
• Parallel Execution: Five workers running simultaneously on a typical 8-core machine deliver approximately 5× speedup compared to single-threaded analysis.
• Incremental Updates: Because each batch persists to disk, subsequent runs only reprocess batches containing changed files, reusing unchanged intermediate results.
• Deterministic Output: The static analysis performed by Tree-sitter is pure and reproducible; merging only reindexes IDs, guaranteeing that identical source code always produces identical structural graphs.

Practical Implementation

The following examples demonstrate how to execute and debug the batch processing pipeline in real-world scenarios.

Running a Full Scan on Large Projects

Execute the main analysis command to trigger automatic batch creation and parallel processing:

# Navigate to the skill directory and run the analysis

understand-anything-plugin/skills/understand/understand.mjs

# Or use the shorthand alias

understand

The pipeline produces log output showing batch creation and processing:

[info] Created batch 0 (27 files)
[info] Created batch 1 (30 files)
...
[info] Finished file-analysis for batch 0 → .understand-anything/intermediate/batch-0.json

Manual Batch Merging for Debugging

For inspection or troubleshooting, manually invoke the Python merger:

cd understand-anything-plugin/skills/understand
python merge-batch-graphs.py /path/to/your/project

Typical output shows batch discovery and consolidation statistics:

Found 42 batch files (42 logical batches, 0 multi-part):
✔ Loaded batch 0 (112 nodes, 210 edges)
✔ Loaded batch 1 (95 nodes, 178 edges)
...
Assembled graph: 5,210 nodes, 10,340 edges

The final knowledge-graph.json is written to the project's .understand-anything/ directory.

Incremental Re-Processing

When source files change, the system optimizes by reprocessing only affected batches:

# Modify some source files

git touch src/new-feature.ts

# Re-run to trigger selective batch updates

understand

The logs indicate selective processing:

[info] Detected changes in batch 3 → re-creating batch-3.json
[info] Skipping unchanged batches 0,1,2,4

Working with Batch Configuration in TypeScript

Access the batch configuration programmatically to build custom tooling:

import { batchConfig } from '@understand-anything/core/languages/configs';

function isBatchLanguage(lang: string): boolean {
  return batchConfig.id === lang;
}

// Verify batch processing is enabled
console.log(isBatchLanguage("batch")); // true

Key Source Files and Their Roles

Understanding the codebase requires familiarity with these specific modules:

• packages/core/src/languages/configs/batch.ts: Declares the batch language identifier and scanner configuration settings.
• skills/understand/merge-batch-graphs.py: Python helper that validates, normalizes, and merges all intermediate batch files into the final knowledge graph.
• skills/understand/extract-structure.mjs: Entry point that receives batch file lists and orchestrates parallel agent execution.
• packages/core/src/analyzer/graph-builder.ts: Post-merge normalization and cross-batch edge resolution.

Summary

Egonex-AI handles massive codebases through a disciplined batch processing strategy that prioritizes memory safety and parallel execution. The key architectural decisions include:

• Splitting repositories into 20–30 file batches defined in batch.ts
• Running up to 5 concurrent file-analyzer agents to maximize CPU utilization
• Writing intermediate results to .understand-anything/intermediate/ for durability and incremental updates
• Merging partial graphs using merge-batch-graphs.py with deterministic ID normalization
• Producing a final knowledge-graph.json that consolidates all structural data

Frequently Asked Questions

How does Egonex-AI determine the number of files per batch?

The project scanner groups approximately 20 to 30 files into each batch based on the batchConfig definition in packages/core/src/languages/configs/batch.ts. This size balances parallelism efficiency with memory constraints, ensuring each worker process remains lightweight while minimizing coordination overhead.

What happens if one batch fails during processing?

The merge-batch-graphs.py script detects missing or malformed batches during the consolidation phase and reports them in the validation output. Because intermediate results are written to disk independently, failed batches can be reprocessed individually without restarting the entire pipeline, and the merger only includes successfully completed batch files in the final graph.

Can I adjust the number of concurrent agents processing batches?

The current implementation supports up to 5 concurrent file-analyzer agents as documented in the README's Multi-Agent Pipeline section. This default is optimized for typical 8-core development machines, providing approximately 5× speedup over single-threaded analysis while leaving headroom for system operations.

Where are the intermediate batch results stored?

Each file-analyzer agent writes its output to .understand-anything/intermediate/batch-<N>.json, where <N> represents the batch index. This directory serves as a persistent checkpoint, enabling incremental reprocessing where only batches containing changed files are regenerated on subsequent runs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}