How SkillSpector Improves Precision with LLM Analysis: 7 Techniques from the NVIDIA Codebase

SkillSpector improves precision with LLM analysis by combining precision-first prompt engineering, Pydantic schema validation, confidence score clamping, overlapping file chunking, and multi-layered meta-analysis filtering that explicitly removes low-confidence findings before they reach users.

NVIDIA's SkillSpector employs a defense-in-depth strategy to maximize precision in LLM-based security analysis. By combining explicit prompt engineering with structured output validation and aggressive post-processing filters, the system ensures that only high-confidence vulnerabilities surface to developers. This approach directly addresses the false positive problem inherent in many AI-powered code analysis tools.

Precision-First Prompt Engineering

SkillSpector's foundation for high-precision analysis begins with explicit instructions baked into every LLM interaction.

The Precision-Over-Recall Directive

In src/skillspector/llm_analyzer_base.py, the BASE_ANALYSIS_PROMPT constant (lines 40-44) appends a strict instruction to every analysis request:

"Precision over recall: only report issues you are confident about. It is far better to miss an edge case than to report a false positive."

This precision-over-recall prompt explicitly nudges the model toward conservative answers. By encoding this preference directly into the system prompt, SkillSpector reduces hallucinated vulnerabilities and ensures the LLM prefers empty results over uncertain findings.

Structured Output Validation

Raw LLM outputs are notoriously unreliable. SkillSpector eliminates malformed responses through rigorous schema enforcement.

Pydantic Schema Enforcement

The LLMAnalyzerBase class declares a strict response_schema = LLMAnalysisResult (lines 70-73 in src/skillspector/llm_analyzer_base.py). This Pydantic model forces the LLM to return well-formed LLMFinding objects rather than free-form text or hallucinated JSON fields.

Confidence Normalization and Clamping

Both LLMFinding and MetaAnalyzerFinding models normalize confidence scores to the 0-1 range and clamp the start line to at least 1 (lines 78-86). This prevents spurious low-confidence values or out-of-range line numbers from slipping through downstream processing, ensuring all numerical data adheres to strict boundaries before filtering occurs.

Intelligent Context Management

Large files often exceed token limits, but naive splitting destroys context. SkillSpector preserves analytical accuracy through strategic chunking.

Overlapping Chunk Strategy

When files exceed the model's token budget, LLMAnalyzerBase splits them into chunks using CHUNK_OVERLAP_LINES = 50 (lines 47-55). The get_batches and chunk_file_by_lines methods (lines 90-124) ensure that context at chunk boundaries is preserved. This overlap prevents false positives caused by missing surrounding code, allowing the LLM to make accurate decisions even when analyzing code segments near split points.

Multi-Layered Filtering Architecture

After initial LLM analysis, SkillSpector applies additional validation layers to catch any ambiguous results that escaped initial processing.

Meta-Analyzer Validation

The LLMMetaAnalyzer.apply_filter method (in src/skillspector/nodes/meta_analyzer.py, lines 60-68) implements strict criteria for findings to proceed:

1. The LLM must explicitly mark the finding as a true vulnerability (is_vulnerability=True)
2. The confidence score must be ≥ 0.6
3. Findings must match exact file-line locations

This ensures duplicate pattern IDs are evaluated independently, and only high-confidence, explicitly validated vulnerabilities pass through.

Heuristic Fallback for Non-LLM Mode

When no LLM key is configured, SkillSpector falls back to a rule-based filter that drops low-confidence findings unless they are high-severity (CRITICAL or HIGH) as shown in lines 25-33. This preserves precision even in "no-LLM" mode, ensuring the system remains useful without API access.

Fail-Closed Error Handling

Should an LLM call raise an exception, the system defaults to passing all findings through with default remediations rather than silently discarding them (lines 96-99). This fail-closed approach ensures potential security issues are never hidden due to infrastructure failures, maintaining visibility while preserving the integrity of the analysis pipeline.

Implementation Examples

Using the Base Analyzer for Discovery

from skillspector.llm_analyzer_base import LLMAnalyzerBase

# Minimal prompt that asks the LLM to find security issues.

BASE_PROMPT = "Find any security problems in the following skill file."

# Instantiate with a model name configured in model_registry.yaml.

analyzer = LLMAnalyzerBase(base_prompt=BASE_PROMPT, model="gpt-4o-mini")

# Prepare file contents cache (normally built by the graph scanner).

file_cache = {"my_skill.py": open("my_skill.py").read()}

# Create batches (automatically chunked if needed).

batches = analyzer.get_batches(["my_skill.py"], file_cache)

# Run batches synchronously.

results = analyzer.run_batches(batches)

# Flatten to findings list.

findings = analyzer.collect_findings(results)

for f in findings:
    print(f"⚠️ {f.rule_id} at line {f.start_line}: {f.message} (confidence={f.confidence})")

Key implementation details:

• Prompt construction uses BASE_ANALYSIS_PROMPT with the precision guideline (lines 40-44)
• Batch handling includes automatic chunking via get_batches and chunk_file_by_lines (lines 90-124)

Using the Meta-Analyzer for Filtering and Enrichment

from skillspector.nodes.meta_analyzer import meta_analyzer
from skillspector.state import SkillspectorState

state = SkillspectorState(
    findings=initial_static_findings,   # list[Finding] from static analysis

    file_cache={"skill.py": open("skill.py").read()},
    manifest={"name": "DemoSkill", "description": "Example"},
    model_config={"meta_analyzer": "gpt-4o-mini"},
    use_llm=True,
)

# Run the meta-analyzer node.

response = meta_analyzer(state)

# Contains only LLM-confirmed, enriched findings.

for f in response["filtered_findings"]:
    print(f"[{f.severity}] {f.message}\nExplanation: {f.explanation}")

Key implementation details:

• PER_FILE_ANALYSIS_PROMPT embeds the precision-over-recall mindset (lines 30-44)
• Filtering logic checks is_vulnerability and confidence ≥ 0.6 (lines 60-68)

Summary

• Precision-over-recall prompting in BASE_ANALYSIS_PROMPT explicitly instructs the LLM to prefer false negatives over false positives.
• Structured output schemas via Pydantic models eliminate malformed JSON and hallucinated fields.
• Confidence normalization clamps values to valid ranges (0-1) and ensures line numbers start at 1.
• Overlapping chunking with CHUNK_OVERLAP_LINES = 50 preserves context at file boundaries to prevent boundary-related false positives.
• Meta-analyzer filtering requires explicit is_vulnerability=True flags and confidence scores ≥ 0.6.
• Heuristic fallback maintains precision when LLM services are unavailable by filtering low-severity, low-confidence findings.
• Fail-closed error handling ensures LLM failures never hide potential security issues.

Frequently Asked Questions

How does SkillSpector prevent false positives from LLM hallucinations?

SkillSpector prevents false positives through three complementary mechanisms: the BASE_ANALYSIS_PROMPT explicitly instructs the model to prefer empty results over uncertain findings, the LLMAnalysisResult Pydantic schema forces structured output that eliminates malformed responses, and the MetaAnalyzer filters out any findings with confidence scores below 0.6 or without explicit is_vulnerability=True flags.

What happens to SkillSpector's precision when the LLM service is unavailable?

When no LLM key is configured, SkillSpector falls back to a heuristic filter in src/skillspector/nodes/meta_analyzer.py (lines 25-33) that drops low-confidence findings unless they are marked as CRITICAL or HIGH severity. This ensures the system maintains high precision even without LLM access by applying stricter thresholds to static analysis results.

Why does SkillSpector use overlapping chunks when analyzing large files?

SkillSpector uses CHUNK_OVERLAP_LINES = 50 (defined in src/skillspector/llm_analyzer_base.py) to preserve context at chunk boundaries. Without overlap, the LLM might miss vulnerabilities that span split points or generate false positives due to missing surrounding code context. The overlap ensures the model always has sufficient context to make accurate decisions.

What is the minimum confidence threshold for LLM findings to reach users?

Findings must achieve a confidence score of at least 0.6 and be explicitly marked with is_vulnerability=True by the meta-analyzer to pass through the filter (lines 60-68 in src/skillspector/nodes/meta_analyzer.py). Additionally, the base analyzer's confidence normalization ensures all scores are clamped to the 0-1 range before this threshold is applied.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}