# How SkillSpector Improves Precision with LLM Analysis: 7 Techniques from the NVIDIA Codebase

> Discover how SkillSpector enhances LLM analysis precision with 7 techniques from the NVIDIA codebase, including prompt engineering and Pydantic validation. Eliminate low-confidence results.

- Repository: [NVIDIA Corporation/SkillSpector](https://github.com/NVIDIA/SkillSpector)
- Tags: deep-dive
- Published: 2026-06-23

---

**SkillSpector improves precision with LLM analysis by combining precision-first prompt engineering, Pydantic schema validation, confidence score clamping, overlapping file chunking, and multi-layered meta-analysis filtering that explicitly removes low-confidence findings before they reach users.**

NVIDIA's SkillSpector employs a defense-in-depth strategy to maximize precision in LLM-based security analysis. By combining explicit prompt engineering with structured output validation and aggressive post-processing filters, the system ensures that only high-confidence vulnerabilities surface to developers. This approach directly addresses the false positive problem inherent in many AI-powered code analysis tools.

## Precision-First Prompt Engineering

SkillSpector's foundation for high-precision analysis begins with explicit instructions baked into every LLM interaction.

### The Precision-Over-Recall Directive

In [`src/skillspector/llm_analyzer_base.py`](https://github.com/NVIDIA/SkillSpector/blob/main/src/skillspector/llm_analyzer_base.py), the `BASE_ANALYSIS_PROMPT` constant (lines 40-44) appends a strict instruction to every analysis request:

> *"Precision over recall: only report issues you are confident about. It is far better to miss an edge case than to report a false positive."*

This **precision-over-recall prompt** explicitly nudges the model toward conservative answers. By encoding this preference directly into the system prompt, SkillSpector reduces hallucinated vulnerabilities and ensures the LLM prefers empty results over uncertain findings.

## Structured Output Validation

Raw LLM outputs are notoriously unreliable. SkillSpector eliminates malformed responses through rigorous schema enforcement.

### Pydantic Schema Enforcement

The `LLMAnalyzerBase` class declares a strict `response_schema = LLMAnalysisResult` (lines 70-73 in [`src/skillspector/llm_analyzer_base.py`](https://github.com/NVIDIA/SkillSpector/blob/main/src/skillspector/llm_analyzer_base.py)). This Pydantic model forces the LLM to return well-formed `LLMFinding` objects rather than free-form text or hallucinated JSON fields.

### Confidence Normalization and Clamping

Both `LLMFinding` and `MetaAnalyzerFinding` models normalize confidence scores to the 0-1 range and clamp the start line to at least 1 (lines 78-86). This prevents **spurious low-confidence values** or out-of-range line numbers from slipping through downstream processing, ensuring all numerical data adheres to strict boundaries before filtering occurs.

## Intelligent Context Management

Large files often exceed token limits, but naive splitting destroys context. SkillSpector preserves analytical accuracy through strategic chunking.

### Overlapping Chunk Strategy

When files exceed the model's token budget, `LLMAnalyzerBase` splits them into chunks using `CHUNK_OVERLAP_LINES = 50` (lines 47-55). The `get_batches` and `chunk_file_by_lines` methods (lines 90-124) ensure that context at chunk boundaries is preserved. This overlap prevents **false positives caused by missing surrounding code**, allowing the LLM to make accurate decisions even when analyzing code segments near split points.

## Multi-Layered Filtering Architecture

After initial LLM analysis, SkillSpector applies additional validation layers to catch any ambiguous results that escaped initial processing.

### Meta-Analyzer Validation

The `LLMMetaAnalyzer.apply_filter` method (in [`src/skillspector/nodes/meta_analyzer.py`](https://github.com/NVIDIA/SkillSpector/blob/main/src/skillspector/nodes/meta_analyzer.py), lines 60-68) implements strict criteria for findings to proceed:

1. The LLM must explicitly mark the finding as a true vulnerability (`is_vulnerability=True`)
2. The confidence score must be **≥ 0.6**
3. Findings must match exact file-line locations

This ensures duplicate pattern IDs are evaluated independently, and only high-confidence, explicitly validated vulnerabilities pass through.

### Heuristic Fallback for Non-LLM Mode

When no LLM key is configured, SkillSpector falls back to a rule-based filter that drops low-confidence findings unless they are high-severity (**CRITICAL** or **HIGH**) as shown in lines 25-33. This preserves precision even in "no-LLM" mode, ensuring the system remains useful without API access.

### Fail-Closed Error Handling

Should an LLM call raise an exception, the system defaults to passing **all** findings through with default remediations rather than silently discarding them (lines 96-99). This **fail-closed** approach ensures potential security issues are never hidden due to infrastructure failures, maintaining visibility while preserving the integrity of the analysis pipeline.

## Implementation Examples

### Using the Base Analyzer for Discovery

```python
from skillspector.llm_analyzer_base import LLMAnalyzerBase

# Minimal prompt that asks the LLM to find security issues.

BASE_PROMPT = "Find any security problems in the following skill file."

# Instantiate with a model name configured in model_registry.yaml.

analyzer = LLMAnalyzerBase(base_prompt=BASE_PROMPT, model="gpt-4o-mini")

# Prepare file contents cache (normally built by the graph scanner).

file_cache = {"my_skill.py": open("my_skill.py").read()}

# Create batches (automatically chunked if needed).

batches = analyzer.get_batches(["my_skill.py"], file_cache)

# Run batches synchronously.

results = analyzer.run_batches(batches)

# Flatten to findings list.

findings = analyzer.collect_findings(results)

for f in findings:
    print(f"⚠️ {f.rule_id} at line {f.start_line}: {f.message} (confidence={f.confidence})")

```

*Key implementation details:*
- Prompt construction uses `BASE_ANALYSIS_PROMPT` with the precision guideline (lines 40-44)
- Batch handling includes automatic chunking via `get_batches` and `chunk_file_by_lines` (lines 90-124)

### Using the Meta-Analyzer for Filtering and Enrichment

```python
from skillspector.nodes.meta_analyzer import meta_analyzer
from skillspector.state import SkillspectorState

state = SkillspectorState(
    findings=initial_static_findings,   # list[Finding] from static analysis

    file_cache={"skill.py": open("skill.py").read()},
    manifest={"name": "DemoSkill", "description": "Example"},
    model_config={"meta_analyzer": "gpt-4o-mini"},
    use_llm=True,
)

# Run the meta-analyzer node.

response = meta_analyzer(state)

# Contains only LLM-confirmed, enriched findings.

for f in response["filtered_findings"]:
    print(f"[{f.severity}] {f.message}\nExplanation: {f.explanation}")

```

*Key implementation details:*
- `PER_FILE_ANALYSIS_PROMPT` embeds the precision-over-recall mindset (lines 30-44)
- Filtering logic checks `is_vulnerability` and confidence ≥ 0.6 (lines 60-68)

## Summary

- **Precision-over-recall prompting** in `BASE_ANALYSIS_PROMPT` explicitly instructs the LLM to prefer false negatives over false positives.
- **Structured output schemas** via Pydantic models eliminate malformed JSON and hallucinated fields.
- **Confidence normalization** clamps values to valid ranges (0-1) and ensures line numbers start at 1.
- **Overlapping chunking** with `CHUNK_OVERLAP_LINES = 50` preserves context at file boundaries to prevent boundary-related false positives.
- **Meta-analyzer filtering** requires explicit `is_vulnerability=True` flags and confidence scores ≥ 0.6.
- **Heuristic fallback** maintains precision when LLM services are unavailable by filtering low-severity, low-confidence findings.
- **Fail-closed error handling** ensures LLM failures never hide potential security issues.

## Frequently Asked Questions

### How does SkillSpector prevent false positives from LLM hallucinations?

SkillSpector prevents false positives through three complementary mechanisms: the `BASE_ANALYSIS_PROMPT` explicitly instructs the model to prefer empty results over uncertain findings, the `LLMAnalysisResult` Pydantic schema forces structured output that eliminates malformed responses, and the `MetaAnalyzer` filters out any findings with confidence scores below 0.6 or without explicit `is_vulnerability=True` flags.

### What happens to SkillSpector's precision when the LLM service is unavailable?

When no LLM key is configured, SkillSpector falls back to a heuristic filter in [`src/skillspector/nodes/meta_analyzer.py`](https://github.com/NVIDIA/SkillSpector/blob/main/src/skillspector/nodes/meta_analyzer.py) (lines 25-33) that drops low-confidence findings unless they are marked as **CRITICAL** or **HIGH** severity. This ensures the system maintains high precision even without LLM access by applying stricter thresholds to static analysis results.

### Why does SkillSpector use overlapping chunks when analyzing large files?

SkillSpector uses `CHUNK_OVERLAP_LINES = 50` (defined in [`src/skillspector/llm_analyzer_base.py`](https://github.com/NVIDIA/SkillSpector/blob/main/src/skillspector/llm_analyzer_base.py)) to preserve context at chunk boundaries. Without overlap, the LLM might miss vulnerabilities that span split points or generate false positives due to missing surrounding code context. The overlap ensures the model always has sufficient context to make accurate decisions.

### What is the minimum confidence threshold for LLM findings to reach users?

Findings must achieve a confidence score of **at least 0.6** and be explicitly marked with `is_vulnerability=True` by the meta-analyzer to pass through the filter (lines 60-68 in [`src/skillspector/nodes/meta_analyzer.py`](https://github.com/NVIDIA/SkillSpector/blob/main/src/skillspector/nodes/meta_analyzer.py)). Additionally, the base analyzer's confidence normalization ensures all scores are clamped to the 0-1 range before this threshold is applied.