How SkillSpector Batches Large Files for LLM Analysis: A Technical Deep Dive

SkillSpector automatically splits oversized source files into overlapping, token-bounded chunks and processes them in parallel batches to stay within model context limits while preserving accurate line numbers for findings.

When analyzing software repositories with large language models (LLMs), context window constraints pose a significant challenge. NVIDIA's SkillSpector solves this through an intelligent batching system that segments massive files into manageable pieces without losing semantic context. This architecture ensures that every line of code receives LLM attention, even when individual files exceed the model's input token budget.

The Token Budget Challenge

LLM-based code analysis in SkillSpector operates on a per-file basis, but source files can easily exceed the model's maximum token limit. When this happens, the system must either truncate the input (losing valuable context) or implement a chunking strategy. SkillSpector chooses the latter, automatically partitioning large files while maintaining 50 lines of overlap between consecutive chunks to preserve code context that spans boundaries.

The batching logic is centralized in src/skillspector/llm_analyzer_base.py and serves as the foundation for downstream analysis nodes, including the meta-analyzer that performs LLM-based filtering and enrichment.

Token Estimation and Budget Calculation

Before processing begins, the analyzer calculates exactly how much room remains for actual code content. The get_batches method first estimates the token overhead of the base prompt using estimate_tokens.

base_overhead = estimate_tokens(self.base_prompt)   # LLMAnalyzerBase.get_batches

This overhead includes static prompt text and any additional context, such as previously discovered findings. The remaining budget determines whether the file can be processed as a single unit or requires segmentation. This calculation accounts for model-specific limits defined in src/skillspector/constants.py, ensuring compatibility across different LLM providers.

The Chunking Algorithm

When a file exceeds the available token budget, chunk_file_by_lines performs the actual segmentation. This function walks through the file line-by-line, accumulating tokens until the limit is reached, then emits a chunk.

chunks = chunk_file_by_lines(content, chunk_budget)   # src/skillspector/llm_analyzer_base.py#L27

The algorithm adds 50 lines of overlap between consecutive chunks by default. This overlap ensures that functions, classes, or logic blocks that span chunk boundaries maintain sufficient context in at least one of the segments. Without this overlap, findings that reference code near a split point might lack the necessary surrounding context for accurate LLM interpretation.

The Batch Dataclass Structure

Each unit of work is encapsulated in a Batch object defined in llm_analyzer_base.py. This dataclass contains:

• The file path being analyzed
• The sub-content (specific line range)
• Start and end line numbers for accurate reporting
• Any pre-existing findings relevant to that line range

The batch structure ensures that line numbers remain accurate relative to the original file, even when the LLM only sees a subset of the content. This is critical for mapping LLM-generated findings back to the correct locations in the source repository.

Building Batch Objects with get_batches

The LLMAnalyzerBase.get_batches method orchestrates the batch creation logic located at line 63 of llm_analyzer_base.py. It handles three distinct scenarios:

1. Single batch: If the entire file fits within the token budget, it creates one Batch containing the full content.
2. Multiple chunks: If the file is oversized, it invokes chunk_file_by_lines and creates a separate Batch for each chunk.
3. Finding filtering: For each chunk, it filters pre-existing findings to include only those within the chunk's line range using findings_in_range.

This approach allows the system to process massive files as a series of independent, parallelizable units while maintaining the relationship between static analysis findings and specific code regions.

Concurrent Processing with arun_batches

Once batches are constructed, arun_batches (line 64) executes them concurrently using a semaphore-backed throttling mechanism. By default, SkillSpector limits concurrency to 10 parallel LLM calls to prevent API rate limit violations and manage costs.

For each batch, the system:

1. Constructs a prompt including line-numbered content
2. Invokes the LLM via the LangChain chat model (configured in llm_utils.py)
3. Parses the response into standardized Finding objects

The concurrent execution dramatically reduces analysis time for large repositories, as multiple file chunks (or different files) process simultaneously rather than sequentially.

Integration in the Meta-Analyzer Pipeline

The batching infrastructure powers the meta-analyzer node located in src/skillspector/nodes/meta_analyzer.py. Here, LLMMetaAnalyzer (a subclass of LLMAnalyzerBase) applies the batching logic to files that have already passed through static analysis.

The integration at line 85 of meta_analyzer.py follows this flow:

analyzer = LLMMetaAnalyzer(model=state["model_config"]["meta_analyzer"])
batches = analyzer.get_batches(files_with_findings,
                               file_cache,
                               findings)               # Splits large files automatically

batch_results = asyncio.run(
    analyzer.arun_batches(batches, metadata_text=_format_metadata(manifest))
)
filtered_findings = analyzer.apply_filter(findings, batch_results)

This node specifically processes files that contain at least one static finding, using the LLM to filter false positives and enrich findings with additional context. The batching ensures that even 10,000-line files receive full LLM attention without hitting token limits.

Practical Implementation Examples

You can leverage the same batching machinery for custom analysis tasks. Below is a minimal implementation that mirrors the meta-analyzer's approach:

from skillspector.llm_analyzer_base import LLMAnalyzerBase, Batch
from skillspector.models import Finding
from skillspector.logging_config import get_logger

logger = get_logger(__name__)

class SimpleAnalyzer(LLMAnalyzerBase):
    def __init__(self, model: str):
        super().__init__(base_prompt="You are a security analyst.", model=model)

# Load a large file for demonstration

large_file_content = open("very_big.py", "r").read()
file_cache = {"very_big.py": large_file_content}
findings = []  # No prior static findings

analyzer = SimpleAnalyzer(model="gpt-4")
batches = analyzer.get_batches(["very_big.py"], file_cache, findings)

print(f"Created {len(batches)} batch(es) for the file.")
for i, b in enumerate(batches, 1):
    print(f"Batch {i}: lines {b.start_line}–{b.end_line or 'end'} "
          f"({len(b.content)} chars)")

# Execute with reduced concurrency for API safety

results = await analyzer.arun_batches(batches, max_concurrency=5)

for batch, batch_findings in results:
    print(f"LLM returned {len(batch_findings)} finding(s) for {batch.file_path}")

For integration within the SkillSpector pipeline, the meta-analyzer demonstrates production usage:

from skillspector.nodes.meta_analyzer import LLMMetaAnalyzer

# Assuming state contains file_cache, manifest, and static findings

analyzer = LLMMetaAnalyzer(model=state["model_config"]["meta_analyzer"])
batches = analyzer.get_batches(
    files_with_findings,
    state["file_cache"],
    state["findings"]
)

# Process all batches concurrently

batch_results = await analyzer.arun_batches(batches)
filtered = analyzer.apply_filter(state["findings"], batch_results)

Summary

• Token-aware budgeting: SkillSpector calculates base prompt overhead before processing to determine exact available capacity for code content.
• Line-based chunking with overlap: The chunk_file_by_lines function splits oversized files using a 50-line overlap buffer to maintain cross-boundary context.
• Structured batch objects: Each chunk becomes a Batch instance preserving file paths and accurate line numbers for precise finding attribution.
• Parallel execution: arun_batches processes up to 10 chunks concurrently by default, optimizing throughput while respecting API limits.
• Pipeline integration: The meta_analyzer node in meta_analyzer.py leverages LLMAnalyzerBase to apply LLM filtering to arbitrarily large files without manual segmentation.

Frequently Asked Questions

How does SkillSpector handle files that exceed the LLM's token limit?

SkillSpector automatically detects when a file exceeds the remaining token budget after accounting for the base prompt. It then invokes chunk_file_by_lines to split the file into smaller segments, each wrapped in a Batch object with accurate line numbers. This ensures every line receives analysis without truncation.

What is the purpose of the 50-line overlap between chunks?

The 50-line overlap prevents context loss when code elements like functions or classes span chunk boundaries. By duplicating these boundary lines in consecutive batches, SkillSpector guarantees that the LLM has sufficient context to understand code that might be split across segments, ensuring accurate findings near chunk edges.

Can I adjust the concurrency level when running batches?

Yes. While arun_batches defaults to 10 parallel calls, you can specify a custom max_concurrency parameter. This is useful when working with strict API rate limits or when you need to reduce costs by throttling requests. The semaphore-based implementation in llm_analyzer_base.py manages this throttling automatically.

Where does the batching logic fit into the overall SkillSpector analysis pipeline?

The batching logic resides in LLMAnalyzerBase and is consumed by specialized nodes like the meta-analyzer in src/skillspector/nodes/meta_analyzer.py. This separation allows any analysis node to inherit token-aware chunking and parallel execution capabilities simply by subclassing LLMAnalyzerBase and calling get_batches followed by arun_batches.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}