Security Limitations of Static-Only Analysis vs Full LLM Analysis in SkillSpector

SkillSpector's static-only analysis misses hidden intent, dynamic behaviors, and contextual API misuse that the full LLM analysis identifies through semantic reasoning, though the LLM approach introduces potential hallucinations and credential dependencies.

SkillSpector, NVIDIA's open-source security scanning framework, implements a dual-stage pipeline that evaluates AI skills and code repositories. Understanding the security limitations of static-only analysis versus full LLM analysis helps teams balance scanning speed against comprehensive threat detection when assessing potentially malicious code.

How SkillSpector's Two-Stage Analysis Pipeline Works

SkillSpector operates through a configurable graph-based architecture found in src/skillspector/cli.py. The CLI parses the --no-llm flag to determine which analysis path to execute, building state with the use_llm boolean between lines 90 and 100.

When use_llm is set to False, the pipeline executes only static nodes such as static_yara.py, which runs YARA signatures against source files without semantic interpretation. When enabled, the system invokes LLMAnalyzerBase (defined in src/skillspector/llm_analyzer_base.py at lines 16-22) to perform deep semantic analysis of code intent and context.

Critical Gaps in Static-Only Analysis

Running SkillSpector with --no-llm creates significant security blind spots that sophisticated attackers can exploit. The static analysis layer examines concrete code patterns and predefined signatures but cannot infer purpose or interpret natural language.

Hidden Instructions and Intent

Static analyzers search for specific code signatures and patterns, but they cannot comprehend the meaning behind comments or documentation. According to the README at lines 88-96, prompt-injection patterns expressed as hidden directives inside comments are only caught by the LLM's natural-language understanding. A static scanner sees benign text where an LLM recognizes malicious instruction.

Dynamic and Runtime-Dependent Behaviors

Vulnerabilities that manifest only during execution remain invisible to static analysis. Conditional imports, reflection-based code loading, and taint-tracking flows require dynamic interpretation. The behavioral analyzers rely on LLM-driven AST inspection to surface these execution-dependent flows that static rules cannot predict.

Contextual API Misuse

Whether a function call is dangerous often depends on surrounding documentation and usage patterns that static regexes cannot capture. The LLM analysis can flag misuse of tool-calling APIs that appear benign in isolation but become malicious when contextualized within the broader codebase, particularly within the MCP (Model Context Protocol) nodes.

Supply Chain Nuances

While static analysis can match against OSV databases, it misses textual hints from CVE descriptions that require interpretation. The static fallback for supply-chain analysis (SC4) still requires LLM enrichment for nuanced scoring of subtle indicators that predefined patterns do not encode.

False Confidence

Static analysis may report "No findings" even when sophisticated, low-signal malicious intent exists. The meta_analyzer node explicitly notes "No static analysis findings for this file" at README lines 180-184, yet the LLM backend can still surface critical security issues that pattern matching missed.

Advantages of Full LLM Analysis

Activating the full LLM pipeline through LLMAnalyzerBase provides three critical capabilities that static analysis lacks:

Semantic reasoning allows the model to interpret free-form text, comments, and documentation, surfacing hidden directives that static regexes cannot detect.

Contextual risk scoring enables the LLM to aggregate findings across multiple files, assigning higher severity to patterns that appear together and create compound risks.

Language-agnostic insight eliminates the need for language-specific YARA rules, as the LLM can understand any programming language within its training distribution.

Limitations of LLM-Powered Security Analysis

Despite its depth, full LLM analysis introduces distinct trade-offs that security teams must consider:

Model hallucination and false positives — LLMs may invent findings that do not exist in the code, requiring downstream verification to filter erroneous results.

Credential dependencies — The LLM stage requires properly configured API keys. Missing or misconfigured credentials abort the scan entirely, as evidenced by the NO_LLM_API_KEY_MESSAGE handling in the test suite.

Latency and cost implications — Each file or chunk incurs an API request, significantly increasing runtime and potentially incurring usage fees based on token volume.

Privacy considerations — Transmitting skill source code to external model providers may expose proprietary code unless teams deploy self-hosted LLM infrastructure.

Practical Examples: Running Static-Only vs Full Scans

Running Static-Only Analysis

Execute a fast scan without LLM invocation to quickly check for known signatures:

skillspector scan ./my-skill/ --no-llm

The --no-llm flag explicitly disables the LLM stage, forcing the pipeline to rely solely on static YARA rules and pattern matching (see README line 416).

Running Full LLM Analysis

Enable comprehensive semantic analysis by configuring provider credentials:

export SKILLSPECTOR_PROVIDER=anthropic
export ANTHROPIC_API_KEY=sk-ant-...
skillspector scan ./my-skill/

When these environment variables are present, SkillSpector automatically enables the LLMAnalyzerBase integration (see README lines 39-44).

Programmatic Configuration

Control the analysis mode directly when invoking the graph programmatically:

from skillspector.graph import graph

state = {
    "input_path": "./my-skill/",
    "output_format": "terminal",
    "use_llm": True,      # Set to False for static-only

}
result = graph.invoke(state)
print(result["report_body"])

The use_llm key mirrors the CLI flag behavior defined in src/skillspector/cli.py.

Summary

• Static-only analysis (--no-llm) provides fast, credential-free scanning but misses hidden instructions, dynamic behaviors, and contextual API misuse that require semantic interpretation.
• Full LLM analysis leverages LLMAnalyzerBase to detect intent-based threats and language-agnostic vulnerabilities through deep semantic understanding.
• Static analysis limitations include inability to process comments as instructions, blind spots for runtime-dependent code, and false confidence when no patterns match.
• LLM limitations include potential hallucinations, API key dependencies, increased latency/cost, and privacy concerns when transmitting code to external providers.
• Key implementation files include src/skillspector/cli.py for flag parsing, src/skillspector/llm_analyzer_base.py for semantic analysis, and src/skillspector/nodes/meta_analyzer.py for merging findings.

Frequently Asked Questions

What specific vulnerabilities does static-only analysis miss in SkillSpector?

Static-only analysis misses prompt-injection patterns hidden in comments, conditional imports that execute malicious code at runtime, and contextual misuse of tool-calling APIs that appear benign in isolation. The meta_analyzer node specifically documents that static analysis can report zero findings while the LLM still identifies critical security issues (README lines 180-184).

How do I switch between static-only and full LLM analysis in SkillSpector?

Use the --no-llm CLI flag for static-only scans, or set the appropriate environment variables (SKILLSPECTOR_PROVIDER and corresponding API keys) to enable full analysis. The CLI checks these conditions between lines 90-100 in src/skillspector/cli.py to set the use_llm state flag that controls pipeline execution.

What are the main risks of relying on LLM analysis for security scanning?

LLM analysis risks include model hallucinations generating false positives, scan failures due to missing API credentials, increased operational costs from token usage, and potential exposure of proprietary code when using external model providers. Teams should implement verification steps for LLM-generated findings and consider self-hosted models for sensitive codebases.

When should I use static-only analysis instead of full LLM analysis?

Use static-only analysis when scanning large codebases where speed is critical, when API credentials are unavailable, or when performing initial triage to filter known-bad patterns. Reserve full LLM analysis for deep-dive security audits of high-risk skills, code with extensive documentation/comments, or when investigating subtle supply-chain compromises that require contextual interpretation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}