# How SkillSpector Batches Large Files for LLM Analysis: A Technical Deep Dive

> Discover how SkillSpector batches large files for LLM analysis. Learn how it splits files into chunks, processes them in parallel, and maintains accurate line numbers for precise findings.

- Repository: [NVIDIA Corporation/SkillSpector](https://github.com/NVIDIA/SkillSpector)
- Tags: deep-dive
- Published: 2026-06-16

---

**SkillSpector automatically splits oversized source files into overlapping, token-bounded chunks and processes them in parallel batches to stay within model context limits while preserving accurate line numbers for findings.**

When analyzing software repositories with large language models (LLMs), context window constraints pose a significant challenge. NVIDIA's SkillSpector solves this through an intelligent batching system that segments massive files into manageable pieces without losing semantic context. This architecture ensures that every line of code receives LLM attention, even when individual files exceed the model's input token budget.

## The Token Budget Challenge

LLM-based code analysis in SkillSpector operates on a **per-file basis**, but source files can easily exceed the model's maximum token limit. When this happens, the system must either truncate the input (losing valuable context) or implement a chunking strategy. SkillSpector chooses the latter, automatically partitioning large files while maintaining **50 lines of overlap** between consecutive chunks to preserve code context that spans boundaries.

The batching logic is centralized in [`src/skillspector/llm_analyzer_base.py`](https://github.com/NVIDIA/SkillSpector/blob/main/src/skillspector/llm_analyzer_base.py) and serves as the foundation for downstream analysis nodes, including the meta-analyzer that performs LLM-based filtering and enrichment.

## Token Estimation and Budget Calculation

Before processing begins, the analyzer calculates exactly how much room remains for actual code content. The `get_batches` method first estimates the token overhead of the base prompt using `estimate_tokens`.

```python
base_overhead = estimate_tokens(self.base_prompt)   # LLMAnalyzerBase.get_batches

```

This overhead includes static prompt text and any additional context, such as previously discovered findings. The remaining budget determines whether the file can be processed as a single unit or requires segmentation. This calculation accounts for model-specific limits defined in [`src/skillspector/constants.py`](https://github.com/NVIDIA/SkillSpector/blob/main/src/skillspector/constants.py), ensuring compatibility across different LLM providers.

## The Chunking Algorithm

When a file exceeds the available token budget, `chunk_file_by_lines` performs the actual segmentation. This function walks through the file **line-by-line**, accumulating tokens until the limit is reached, then emits a chunk.

```python
chunks = chunk_file_by_lines(content, chunk_budget)   # src/skillspector/llm_analyzer_base.py#L27

```

The algorithm adds **50 lines of overlap** between consecutive chunks by default. This overlap ensures that functions, classes, or logic blocks that span chunk boundaries maintain sufficient context in at least one of the segments. Without this overlap, findings that reference code near a split point might lack the necessary surrounding context for accurate LLM interpretation.

## The Batch Dataclass Structure

Each unit of work is encapsulated in a `Batch` object defined in [`llm_analyzer_base.py`](https://github.com/NVIDIA/SkillSpector/blob/main/llm_analyzer_base.py). This dataclass contains:

- The **file path** being analyzed
- The **sub-content** (specific line range)
- **Start and end line numbers** for accurate reporting
- Any **pre-existing findings** relevant to that line range

The batch structure ensures that line numbers remain accurate relative to the original file, even when the LLM only sees a subset of the content. This is critical for mapping LLM-generated findings back to the correct locations in the source repository.

## Building Batch Objects with `get_batches`

The `LLMAnalyzerBase.get_batches` method orchestrates the batch creation logic located at line 63 of [`llm_analyzer_base.py`](https://github.com/NVIDIA/SkillSpector/blob/main/llm_analyzer_base.py). It handles three distinct scenarios:

1. **Single batch**: If the entire file fits within the token budget, it creates one `Batch` containing the full content.
2. **Multiple chunks**: If the file is oversized, it invokes `chunk_file_by_lines` and creates a separate `Batch` for each chunk.
3. **Finding filtering**: For each chunk, it filters pre-existing findings to include only those within the chunk's line range using `findings_in_range`.

This approach allows the system to process massive files as a series of independent, parallelizable units while maintaining the relationship between static analysis findings and specific code regions.

## Concurrent Processing with `arun_batches`

Once batches are constructed, `arun_batches` (line 64) executes them concurrently using a semaphore-backed throttling mechanism. By default, SkillSpector limits concurrency to **10 parallel LLM calls** to prevent API rate limit violations and manage costs.

For each batch, the system:

1. Constructs a prompt including line-numbered content
2. Invokes the LLM via the LangChain chat model (configured in [`llm_utils.py`](https://github.com/NVIDIA/SkillSpector/blob/main/llm_utils.py))
3. Parses the response into standardized `Finding` objects

The concurrent execution dramatically reduces analysis time for large repositories, as multiple file chunks (or different files) process simultaneously rather than sequentially.

## Integration in the Meta-Analyzer Pipeline

The batching infrastructure powers the **meta-analyzer node** located in [`src/skillspector/nodes/meta_analyzer.py`](https://github.com/NVIDIA/SkillSpector/blob/main/src/skillspector/nodes/meta_analyzer.py). Here, `LLMMetaAnalyzer` (a subclass of `LLMAnalyzerBase`) applies the batching logic to files that have already passed through static analysis.

The integration at line 85 of [`meta_analyzer.py`](https://github.com/NVIDIA/SkillSpector/blob/main/meta_analyzer.py) follows this flow:

```python
analyzer = LLMMetaAnalyzer(model=state["model_config"]["meta_analyzer"])
batches = analyzer.get_batches(files_with_findings,
                               file_cache,
                               findings)               # Splits large files automatically

batch_results = asyncio.run(
    analyzer.arun_batches(batches, metadata_text=_format_metadata(manifest))
)
filtered_findings = analyzer.apply_filter(findings, batch_results)

```

This node specifically processes files that contain at least one static finding, using the LLM to filter false positives and enrich findings with additional context. The batching ensures that even 10,000-line files receive full LLM attention without hitting token limits.

## Practical Implementation Examples

You can leverage the same batching machinery for custom analysis tasks. Below is a minimal implementation that mirrors the meta-analyzer's approach:

```python
from skillspector.llm_analyzer_base import LLMAnalyzerBase, Batch
from skillspector.models import Finding
from skillspector.logging_config import get_logger

logger = get_logger(__name__)

class SimpleAnalyzer(LLMAnalyzerBase):
    def __init__(self, model: str):
        super().__init__(base_prompt="You are a security analyst.", model=model)

# Load a large file for demonstration

large_file_content = open("very_big.py", "r").read()
file_cache = {"very_big.py": large_file_content}
findings = []  # No prior static findings

analyzer = SimpleAnalyzer(model="gpt-4")
batches = analyzer.get_batches(["very_big.py"], file_cache, findings)

print(f"Created {len(batches)} batch(es) for the file.")
for i, b in enumerate(batches, 1):
    print(f"Batch {i}: lines {b.start_line}–{b.end_line or 'end'} "
          f"({len(b.content)} chars)")

# Execute with reduced concurrency for API safety

results = await analyzer.arun_batches(batches, max_concurrency=5)

for batch, batch_findings in results:
    print(f"LLM returned {len(batch_findings)} finding(s) for {batch.file_path}")

```

For integration within the SkillSpector pipeline, the meta-analyzer demonstrates production usage:

```python
from skillspector.nodes.meta_analyzer import LLMMetaAnalyzer

# Assuming state contains file_cache, manifest, and static findings

analyzer = LLMMetaAnalyzer(model=state["model_config"]["meta_analyzer"])
batches = analyzer.get_batches(
    files_with_findings,
    state["file_cache"],
    state["findings"]
)

# Process all batches concurrently

batch_results = await analyzer.arun_batches(batches)
filtered = analyzer.apply_filter(state["findings"], batch_results)

```

## Summary

- **Token-aware budgeting**: SkillSpector calculates base prompt overhead before processing to determine exact available capacity for code content.
- **Line-based chunking with overlap**: The `chunk_file_by_lines` function splits oversized files using a 50-line overlap buffer to maintain cross-boundary context.
- **Structured batch objects**: Each chunk becomes a `Batch` instance preserving file paths and accurate line numbers for precise finding attribution.
- **Parallel execution**: `arun_batches` processes up to 10 chunks concurrently by default, optimizing throughput while respecting API limits.
- **Pipeline integration**: The `meta_analyzer` node in [`meta_analyzer.py`](https://github.com/NVIDIA/SkillSpector/blob/main/meta_analyzer.py) leverages `LLMAnalyzerBase` to apply LLM filtering to arbitrarily large files without manual segmentation.

## Frequently Asked Questions

### How does SkillSpector handle files that exceed the LLM's token limit?

SkillSpector automatically detects when a file exceeds the remaining token budget after accounting for the base prompt. It then invokes `chunk_file_by_lines` to split the file into smaller segments, each wrapped in a `Batch` object with accurate line numbers. This ensures every line receives analysis without truncation.

### What is the purpose of the 50-line overlap between chunks?

The 50-line overlap prevents context loss when code elements like functions or classes span chunk boundaries. By duplicating these boundary lines in consecutive batches, SkillSpector guarantees that the LLM has sufficient context to understand code that might be split across segments, ensuring accurate findings near chunk edges.

### Can I adjust the concurrency level when running batches?

Yes. While `arun_batches` defaults to 10 parallel calls, you can specify a custom `max_concurrency` parameter. This is useful when working with strict API rate limits or when you need to reduce costs by throttling requests. The semaphore-based implementation in [`llm_analyzer_base.py`](https://github.com/NVIDIA/SkillSpector/blob/main/llm_analyzer_base.py) manages this throttling automatically.

### Where does the batching logic fit into the overall SkillSpector analysis pipeline?

The batching logic resides in `LLMAnalyzerBase` and is consumed by specialized nodes like the meta-analyzer in [`src/skillspector/nodes/meta_analyzer.py`](https://github.com/NVIDIA/SkillSpector/blob/main/src/skillspector/nodes/meta_analyzer.py). This separation allows any analysis node to inherit token-aware chunking and parallel execution capabilities simply by subclassing `LLMAnalyzerBase` and calling `get_batches` followed by `arun_batches`.