How `get_batches` Splits Files Exceeding Token Budgets into Smaller Chunks in NVIDIA SkillSpector

SkillSpector's get_batches method calculates the remaining token budget after accounting for prompt overhead, then uses chunk_file_by_lines to carve oversized files into overlapping line-based chunks that each fit within the model's input limits.

The NVIDIA SkillSpector repository provides LLM-powered code analysis capabilities that must handle arbitrarily large source files. When a file exceeds the model's token limit, the LLMAnalyzerBase.get_batches method in src/skillspector/llm_analyzer_base.py implements a sophisticated budgeting and splitting strategy to ensure no LLM request is truncated while preserving line-accurate context.

Calculate Static Prompt Overhead

Before processing file content, get_batches first determines how many tokens the analyzer's base prompt consumes. This static overhead is calculated using the estimate_tokens utility, which divides character count by CHARS_PER_TOKEN (defined as 4).

base_overhead = estimate_tokens(self.base_prompt)   # lines 265-266

This value represents tokens permanently reserved for instructions, few-shot examples, or system prompts that accompany every request.

Determine the Usable Content Budget

With the overhead established, the method calculates the actual space available for file content. It retrieves the maximum input tokens from get_max_input_tokens, then subtracts both the base overhead and any additional overhead specific to analyzer subclasses. The code guarantees a minimum of 1,024 tokens for content to prevent unusable micro-batches.

content_budget = max(self._input_budget - base_overhead - extra, 1024)   # lines 85-86

This content_budget becomes the hard limit for each individual file or chunk processed by the system.

Whole File vs. Chunked Processing

The method evaluates each file against the calculated budget before deciding whether to batch it whole or split it.

For files that fit: If estimate_tokens(content) ≤ content_budget, get_batches creates a single Batch containing the entire file content, line numbers, and any associated findings.

batches.append(Batch(file_path=path, content=content, findings=file_findings))   # lines 90-95

For oversized files: When content exceeds the budget, the method delegates to chunk_file_by_lines to perform intelligent splitting.

Line-Based Chunking with Context Preservation

The chunk_file_by_lines function (implemented at lines 129-169) processes large files line-by-line, accumulating token counts until the chunk_budget is reached. Critically, it maintains 50 lines of overlap between consecutive chunks using the CHUNK_OVERLAP_LINES constant, ensuring that context around chunk boundaries is not lost during analysis.

The function returns a list of tuples containing the chunk text and its precise line range:

for chunk_text, s_line, e_line in chunk_file_by_lines(content, chunk_budget):   # lines 298-300

This overlap prevents security vulnerabilities or bugs that span chunk boundaries from being missed due to arbitrary file splitting.

Constructing Batches with Attached Findings

Each generated chunk becomes its own Batch object. The Batch dataclass (defined at lines 102-111) tracks whether it represents a partial file via the is_chunk property and stores start_line and end_line for accurate reporting.

Batch(
    file_path=path,
    content=chunk_text,
    start_line=s_line,
    end_line=e_line,
    findings=chunk_findings,
)

The method uses findings_in_range to attach only relevant pre-existing findings to each chunk, ensuring that diagnostic context travels with the specific lines it references.

Practical Implementation Example

The following example demonstrates how get_batches automatically handles mixed file sizes:

from skillspector.llm_analyzer_base import LLMAnalyzerBase

# Initialize analyzer with GPT-4

my_analyzer = LLMAnalyzerBase(
    base_prompt="Find security issues in the following Python file.",
    model="gpt-4"
)

# Cache containing both small and oversized files

file_cache = {
    "big_script.py": open("big_script.py").read(),
    "small.py": "print('hello')",
}

# Generate batches respecting token budgets

batches = my_analyzer.get_batches(
    file_paths=["big_script.py", "small.py"],
    file_cache=file_cache,
)

for b in batches:
    print(f"Batch for {b.file_path} (lines {b.start_line}-{b.end_line or 'end'}), "
          f"{len(b.content)} chars")

The output produces a single batch for small.py and multiple overlapping batches for big_script.py, each conforming to the token budget calculated from the GPT-4 context window.

Summary

• get_batches orchestrates token budgeting by first calculating prompt overhead, then reserving a minimum of 1,024 tokens for actual content.
• chunk_file_by_lines splits oversized files into line-aligned chunks with 50-line overlaps to preserve boundary context.
• Each chunk becomes a Batch object tracking its source file and line range, with relevant findings attached via findings_in_range.
• The implementation in src/skillspector/llm_analyzer_base.py ensures no LLM request exceeds model limits while maintaining line-accurate analysis capabilities.

Frequently Asked Questions

How does SkillSpector calculate token counts for budgeting?

SkillSpector uses a simple heuristic in estimate_tokens that divides character count by CHARS_PER_TOKEN (set to 4). This approximation runs quickly without requiring the full tokenizer of the target LLM, making it suitable for rapid batching decisions in src/skillspector/llm_analyzer_base.py.

Why does the chunking algorithm use 50 lines of overlap?

The CHUNK_OVERLAP_LINES constant (set to 50) ensures that security vulnerabilities or code constructs spanning chunk boundaries are not split into contexts that lose semantic meaning. This overlap provides sufficient surrounding context for the LLM to understand relationships that cross the artificial chunk divider.

What happens if a file is smaller than the content budget?

For files where estimate_tokens(content) is less than or equal to content_budget, get_batches creates a single Batch containing the entire file. The method skips chunking entirely and attaches all file-specific findings to this single batch, optimizing for fewer LLM calls when possible.

Where are the Batch objects defined in the source code?

The Batch dataclass is defined at lines 102-111 in src/skillspector/llm_analyzer_base.py. It includes fields for file_path, content, start_line, end_line, and findings, along with convenience properties like is_chunk and file_label that downstream analyzers use for reporting.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}