How Semantic Batching Optimizes Token Usage in Understand Anything's Phase 2 File Analysis

TLDR: During Phase 2 analysis, Understand Anything clusters files into semantic batches—grouping by architectural layer or topological order—to send multiple files in single LLM requests, eliminating redundant system-prompt tokens and keeping total token usage manageable for large codebases.

During Phase 2 of the Understand Anything pipeline, the LLM enriches each file's structural data with semantic information such as summaries, tags, and layer assignments. Sending individual requests for every file would exhaust token budgets on large projects. The solution implemented in Egonex-AI/Understand-Anything uses semantic batching to group related files before calling the LLM, significantly reducing overhead while maintaining analysis quality.

The Token Challenge in Phase 2 File Analysis

In Phase 2, the system processes raw structural data to generate semantic metadata. Invoking the LLM separately for each file repeats the system prompt and instruction boilerplate for every request, consuming tokens that could otherwise process additional files. For large codebases, this overhead becomes prohibitive, as noted in the README's token usage guidelines.

How Semantic Batching Works

Semantic batching clusters files by semantic similarity—such as shared architectural layers or related concepts—before invoking the LLM. This strategy allows a single prompt to process multiple files simultaneously, amortizing the fixed prompt cost across the batch.

Batch Configuration and Types

The language configuration defines the batch behavior through the batchConfig type located in understand-anything-plugin/packages/core/src/languages/configs/batch.ts. This configuration establishes how files are grouped based on their semantic characteristics, ensuring that files within the same batch share enough context to be processed together effectively.

Layer-Based Grouping

When the graph-builder detects layer assignments, it groups all files sharing the same layer into a single batch. This approach, implemented in understand-anything-plugin/packages/core/src/analyzer/llm-analyzer.ts, sends one comprehensive request per layer, allowing the model to leverage shared architectural context while minimizing redundant tokens.

Topological Fallback Batching

If no layers are detected, the system falls back to a deterministic three-node batching strategy. According to the source code in understand-anything-plugin/packages/core/src/analyzer/tour-generator.ts (lines 255-259), the analyzer processes files in topological order, slicing them into batches of three:

// From tour-generator.ts - no layers: batch by 3 nodes
if (!layersFound) {
  const batch = topoOrder.slice(i, i + 3);
  const nodeSummaries = batch.map(id => summarizeNode(id));
  // Send nodeSummaries as a single LLM request
}

This fallback ensures consistent batch boundaries even without explicit layer metadata.

Implementation in the LLM Analyzer

The core orchestration logic resides in understand-anything-plugin/packages/core/src/analyzer/llm-analyzer.ts. When the graph-builder reaches the LLM enrichment step, it first checks for layer assignments. If layers exist, it groups by layer; otherwise, it applies the three-node fallback.

Layer-based batching example:

// Group nodes by their assigned layer
const batches = groupBy(layerMap, node => node.layer);
for (const [layer, nodes] of Object.entries(batches)) {
  const prompt = buildPrompt(nodes); // One request per layer
  const result = await llm.call(prompt);
}

Fallback batching without layers:

// Process in topological order, 3 nodes at a time
for (let i = 0; i < topoOrder.length; i += 3) {
  const batch = topoOrder.slice(i, i + 3);
  const prompt = buildBatchPrompt(batch);
  await llm.call(prompt);
}

Token Optimization and Incremental Updates

By sending a single prompt containing several semantically related files, the model reuses the same system instructions and shared context for the entire batch. This eliminates the overhead tokens that would otherwise duplicate across individual file requests, significantly lowering the total token count per run.

The deterministic nature of these batches—whether layer-based or topological—also enables incremental updates. Subsequent runs reuse the same batch composition, limiting token consumption to only newly changed files rather than reprocessing the entire codebase.

Summary

• Semantic batching groups files by architectural layers or topological order before LLM invocation, reducing redundant token overhead.
• The batch configuration in understand-anything-plugin/packages/core/src/languages/configs/batch.ts defines how files cluster by semantic similarity.
• Layer-based batching sends all files in the same layer as one request, while the three-node fallback in understand-anything-plugin/packages/core/src/analyzer/tour-generator.ts handles files without layer assignments.
• Deterministic batch boundaries enable efficient incremental re-analysis, consuming tokens only for changed files.

Frequently Asked Questions

What is semantic batching in Understand Anything?

Semantic batching is a token optimization strategy that clusters files by semantic similarity—such as architectural layers or topological relationships—before sending them to the LLM. This allows multiple files to be processed in a single request, eliminating the repeated system-prompt overhead that would occur with individual file requests.

How does the three-node fallback work when no layers are detected?

When the analyzer cannot detect architectural layers, it falls back to processing files in topological order, grouping them into fixed batches of three nodes. This logic appears in understand-anything-plugin/packages/core/src/analyzer/tour-generator.ts at lines 255-259, ensuring deterministic batch boundaries even without explicit layer metadata.

Where is the batching logic configured in the codebase?

The batch behavior is configured in understand-anything-plugin/packages/core/src/languages/configs/batch.ts, which defines the batchConfig type and semantic grouping rules. The orchestration logic resides in understand-anything-plugin/packages/core/src/analyzer/llm-analyzer.ts, while the fallback batching implementation appears in understand-anything-plugin/packages/core/src/analyzer/tour-generator.ts.

How does semantic batching support incremental analysis?

Because batches are deterministic—grouping by consistent layer assignments or fixed topological slices—the same batch composition is reused across subsequent runs. This allows the system to identify and reprocess only the batches containing changed files, significantly limiting token usage during incremental updates of large codebases.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}