How SkillSpector Uses LLM Semantic Analysis in Its Second Stage

SkillSpector’s second stage employs a meta-analyzer that filters and enriches static findings by calling an LLM with semantic reasoning prompts, validating structured outputs against a Pydantic schema, and retaining only high-confidence vulnerabilities.

The NVIDIA/SkillSpector repository implements a two-stage security analysis pipeline for AI skills. After static-pattern scanners produce raw findings in the first stage, the system applies LLM semantic analysis to evaluate intent, impact, and confidence before finalizing results. This semantic second stage transforms simple pattern matches into contextually aware security assessments.

The Two-Stage Analysis Pipeline

SkillSpector processes skill source code through distinct phases:

1. First stage – static-pattern scanning – Nodes like static_*, behavioral_*, and mcp_* scan code and generate raw findings based on predefined signatures.
2. Second stage – LLM-driven semantic filtering – The meta_analyzer node consumes these findings, invokes an LLM per file, and applies semantic reasoning to determine true vulnerability status, intent, and remediation strategies.

The second stage relies on the LLMAnalyzerBase infrastructure but extends it with custom prompts and structured output schemas in src/skillspector/nodes/meta_analyzer.py.

How the Second Stage Processes Findings

The meta-analyzer orchestrates six distinct steps to transform raw static signatures into semantically validated security findings.

Batch Creation and File Chunking

The process begins in LLMAnalyzerBase.get_batches (defined in src/skillspector/llm_analyzer_base.py). The system groups files containing at least one static finding. For files exceeding the model’s input token budget, chunk_file_by_lines splits the content into overlapping line-chunks to ensure complete coverage without truncation.

Prompt Composition for Semantic Reasoning

The LLMMetaAnalyzer.build_prompt method (in src/skillspector/nodes/meta_analyzer.py) constructs a detailed prompt containing:

• Skill manifest metadata (name, description)
• Line-numbered source text of the target file
• Static findings discovered in that file
• Explicit instructions forcing semantic evaluation (e.g., "Is this a true vulnerability? What is the likely intent? What is the impact?")

This prompt design compels the LLM to perform contextual reasoning rather than pattern matching.

Structured Response Validation

The analyzer declares response_schema = MetaAnalyzerResult, a Pydantic model defined in meta_analyzer.py. This schema enforces per-finding fields including:

• pattern_id
• start_line
• is_vulnerability (boolean)
• confidence (float)
• intent, impact, explanation, and remediation

LangChain’s with_structured_output automatically validates the LLM’s JSON-like response against this schema, ensuring type-safe data extraction.

Parallel Execution and Filtering

LLMMetaAnalyzer.arun_batches executes all per-file (or per-chunk) calls concurrently with a default maximum of 10 simultaneous requests. Each call receives the same prompt template populated with file-specific content.

After receiving LLM responses, LLMMetaAnalyzer.apply_filter matches outputs to original static findings using a granular key comprising (file, rule_id, start_line, end_line). The system retains only findings where is_vulnerability equals true and confidence >= 0.6. Surviving Finding objects receive enrichment via the LLM-generated explanation and remediation fields.

Fallback Path

When use_llm is disabled or LLM calls fail, the system invokes _fallback_filtered (in meta_analyzer.py), returning static-only remediation without semantic enrichment.

Code Implementation Details

The second stage implementation spans several critical files:

File	Role
src/skillspector/nodes/meta_analyzer.py	Implements LLMMetaAnalyzer class with build_prompt, apply_filter, and MetaAnalyzerResult schema
src/skillspector/llm_analyzer_base.py	Provides get_batches, arun_batches, and chunking logic via LLMAnalyzerBase
src/skillspector/models.py	Defines the Finding dataclass that gets enriched with semantic metadata
src/skillspector/llm_utils.py	Handles LLM credential resolution and LangChain model construction

Practical Usage Examples

Programmatic API Integration

Invoke the second stage directly after static analysis:

from skillspector.state import SkillspectorState
from skillspector.nodes.meta_analyzer import meta_analyzer

# State populated from previous static scans

state: SkillspectorState = {
    "findings": [...],  # List of Finding objects from stage 1

    "file_cache": {"my_skill.py": "..."},
    "manifest": {"name": "my_skill", "description": "Demo"},
    "model_config": {"meta_analyzer": "nvidia/openai/gpt-oss-120b"},
    "use_llm": True,
}

# Execute semantic filtering (stage 2)

result = meta_analyzer(state)

# Access filtered and enriched findings

for finding in result["filtered_findings"]:
    print(f"{finding.explanation} | Remediation: {finding.remediation}")

CLI Invocation

Run the complete pipeline including the second stage via command line:

skillspc scan \
    --skill-dir ./my_skill \
    --model meta_analyzer=nvidia/openai/gpt-oss-120b \
    --use-llm true

The CLI automatically triggers meta_analyzer after static scans complete, performing the LLM semantic analysis without additional configuration.

Extending with Custom Semantic Analyzers

To create a semantic discovery analyzer (e.g., semantic_security_discovery), subclass LLMAnalyzerBase and set response_schema = LLMAnalysisResult. Override build_prompt and parse_response when implementing custom semantic logic, following the meta-analyzer’s architecture as a reference implementation.

Summary

• SkillSpector’s second stage uses LLMMetaAnalyzer to apply LLM semantic analysis to static findings.
• Batching and chunking occur via LLMAnalyzerBase.get_batches to handle large files within token limits.
• Semantic prompts force the LLM to evaluate intent, impact, and confidence rather than relying on static patterns.
• Structured validation uses the MetaAnalyzerResult Pydantic schema to ensure consistent, type-safe outputs.
• Filtering logic retains only findings with is_vulnerability=true and confidence >= 0.6, matched by granular file coordinates.
• Fallback mechanisms ensure pipeline continuity when LLM calls are disabled or unavailable.

Frequently Asked Questions

What is the confidence threshold for keeping LLM-filtered findings?

SkillSpector retains only findings where the LLM assigns a confidence score of 0.6 or higher (60%). This threshold is enforced in LLMMetaAnalyzer.apply_filter within meta_analyzer.py.

How does SkillSpector handle files that exceed the LLM token budget?

The system automatically splits oversized files using chunk_file_by_lines in LLMAnalyzerBase.get_batches. These overlapping line-chunks ensure the LLM receives complete context without truncation, processing each chunk separately while maintaining semantic continuity.

Can the second stage operate without LLM calls?

Yes. If use_llm is set to false or the LLM service fails, the pipeline falls back to _fallback_filtered in meta_analyzer.py, returning static findings without semantic enrichment. This ensures analysis completion even during LLM outages.

What structured output format does the LLM return in the second stage?

The LLM returns data validated against the MetaAnalyzerResult Pydantic model, which includes per-finding fields: pattern_id, start_line, is_vulnerability, confidence, intent, impact, explanation, and remediation. LangChain’s with_structured_output validates these fields automatically.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}