How SkillSpector's Two-Stage Static and LLM Analysis Pipeline Works

SkillSpector's two-stage static and LLM analysis pipeline combines fast deterministic static analysis with contextual LLM validation to detect security vulnerabilities in AI agent skills while ensuring critical findings are never silently discarded.

SkillSpector is an open-source security scanner for AI agent skills developed by NVIDIA. Its two-stage static and LLM analysis pipeline processes skill files through a LangGraph workflow that first applies signature-based detection, then uses large language models to validate findings and enrich them with contextual risk assessments and remediation guidance.

Phase 1: Static Analysis Engine

The pipeline begins with deterministic static analysis orchestrated by the build_context node in src/skillspector/nodes/build_context.py. This node materializes skill files into a file cache and prepares code context for downstream analysis.

The static analysis phase operates through several specialized analyzer nodes located in src/skillspector/nodes/analyzers/*.py:

• static_yara.py – Applies YARA signatures to detect known malicious patterns
• static_patterns_privilege_escalation.py – Performs pattern-based checks for privilege escalation vectors
• osv_client.py – Queries the Open Source Vulnerabilities database for dependency risks

Each analyzer independently processes files and returns structured Finding objects defined in src/skillspector/models.py. These findings capture:

• rule_id and message identifying the detection
• severity and confidence levels
• Line numbers and matched text for precise location
• Optional remediation guidance

The findings accumulate in the SkillspectorState typed dictionary (defined in src/skillspector/state.py), which flows through the LangGraph workflow.

Phase 2: LLM-Driven Filtering and Enrichment

After static analysis completes, the meta_analyzer node in src/skillspector/nodes/meta_analyzer.py executes the second stage. This phase leverages the LLMMetaAnalyzer class (a subclass of LLMAnalyzerBase from src/skillspector/llm_analyzer_base.py) to perform contextual validation.

For every file containing static findings, the pipeline constructs a per-file LLM request using the PER_FILE_ANALYSIS_PROMPT template. The prompt includes:

1. Skill metadata and manifest information
2. Complete file contents from the file cache
3. Raw static findings from phase one
4. Security-focused instructions that explicitly forbid the model from trusting self-declared "safe" statements

The LLM response must conform to the MetaAnalyzerResult Pydantic schema, ensuring structured JSON output containing:

• findings – Enriched entries with is_vulnerability, confidence, intent, impact, explanation, and remediation fields
• overall_assessment – A risk-level summary for the entire file

The apply_filter routine merges the LLM response with original static findings using a fail-closed strategy:

• Preserves every HIGH or CRITICAL static finding, adding an "llm-unconfirmed" tag when the LLM does not confirm the vulnerability
• Discards or downgrades lower-severity findings based on the LLM's contextual verdict
• Adds human-readable explanations and targeted remediation guidance to confirmed findings

LangGraph Workflow Architecture

The complete pipeline is assembled in src/skillspector/graph.py as a LangGraph workflow with the following execution flow:

resolve_input → build_context → [static analyzers] → meta_analyzer → report

• resolve_input parses CLI arguments and skill manifests
• build_context creates the file cache and distributes file lists to all analyzers registered in ANALYZER_NODE_IDS
• Static analyzers run in parallel, emitting findings into the shared state
• meta_analyzer performs per-file LLM calls via src/skillspector/llm_utils.py (which handles credential resolution and LangChain ChatModel construction)
• report in src/skillspector/report.py serializes final findings to SARIF or JSON format

This architecture provides defense-in-depth: deterministic static signatures catch obvious problems cheaply, while the LLM adds contextual reasoning and false-positive reduction.

Fail-Closed Safety Mechanisms

SkillSpector guarantees fail-closed semantics to prevent security regressions. If LLM calls are disabled via --no-llm (setting use_llm=False) or if API calls fail, the pipeline activates fallback mechanisms in meta_analyzer.py:

• _fallback_filtered applies confidence-based heuristics to static findings and adds default remediations
• _passthrough_with_defaults ensures critical alerts pass through with default metadata even during total LLM failure

These safeguards ensure the pipeline never silently drops HIGH or CRITICAL severity findings, regardless of LLM availability or configuration.

Running the Pipeline

Execute the full pipeline from the command line with LLM validation enabled:

skillspector scan path/to/skill \
    --model meta_analyzer=gpt-4o-mini \
    --output results.sarif

For programmatic integration in Python applications or test suites:

from skillspector.graph import graph
from skillspector.state import SkillspectorState

# Initialize state with manifest and configuration

state = SkillspectorState(
    manifest={"name": "demo-skill", "description": "example"},
    file_cache={},
    use_llm=True,
    model_config={"meta_analyzer": "gpt-4o-mini"},
)

# Execute the complete workflow

final_state = graph.invoke(state)

# Access LLM-filtered findings

for finding in final_state["filtered_findings"]:
    print(f"{finding.rule_id}: {finding.message} (confidence={finding.confidence:.2f})")

Run without LLM validation for CI environments lacking API credentials:

skillspector scan path/to/skill --no-llm

This command triggers the heuristic fallback mode, applying static analysis with default confidence scoring and remediation templates.

Summary

• SkillSpector's two-stage static and LLM analysis pipeline first applies deterministic static analysis via YARA signatures, pattern matching, and OSV database queries in src/skillspector/nodes/analyzers/.
• The meta_analyzer node in src/skillspector/nodes/meta_analyzer.py validates findings using the LLMMetaAnalyzer class and the MetaAnalyzerResult schema to enrich outputs with contextual risk assessments.
• The pipeline implements fail-closed safety through _fallback_filtered and _passthrough_with_defaults, ensuring HIGH and CRITICAL findings are never dropped.
• The entire workflow is orchestrated through LangGraph in src/skillspector/graph.py, using SkillspectorState to pass data between nodes from resolve_input to final report generation.

Frequently Asked Questions

What happens if the LLM API is unavailable during analysis?

SkillSpector implements fail-closed semantics in src/skillspector/nodes/meta_analyzer.py. If LLM calls fail or are disabled with --no-llm, the pipeline automatically invokes _fallback_filtered or _passthrough_with_defaults, which preserve all HIGH and CRITICAL static findings while applying confidence-based heuristics and default remediations.

How does SkillSpector prevent the LLM from ignoring self-declared "safe" code?

The PER_FILE_ANALYSIS_PROMPT template includes explicit security instructions that forbid the model from trusting self-declared "safe" statements or benign intentions in code comments. This instruction set, processed by the LLMMetaAnalyzer class, ensures the LLM evaluates actual implementation behavior rather than developer assertions.

What static analysis tools does SkillSpector use in the first stage?

The static analysis phase employs multiple specialized analyzers in src/skillspector/nodes/analyzers/, including static_yara.py for signature matching, static_patterns_privilege_escalation.py for privilege escalation detection, and osv_client.py for dependency vulnerability scanning against the Open Source Vulnerabilities database.

Can I customize which LLM model validates the findings?

Yes. The pipeline accepts model configuration through the --model CLI flag or the model_config dictionary in SkillspectorState. The meta_analyzer key specifies the model for the LLM stage (e.g., meta_analyzer=gpt-4o-mini), with credentials resolved via src/skillspector/llm_utils.py using standard environment variables or configuration files.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}