# Comparing Accuracy Between Static‑Only and Full LLM Analysis in SkillSpector

> Compare static-only versus full LLM analysis in NVIDIA SkillSpector. Understand accuracy trade-offs for precise detections versus context-aware reasoning.

- Repository: [NVIDIA Corporation/SkillSpector](https://github.com/NVIDIA/SkillSpector)
- Tags: performance
- Published: 2026-07-09

---

**Static‑only analysis delivers deterministic, high‑precision detections with minimal false positives but limited recall, while full LLM analysis leverages context‑aware reasoning to surface subtle threats at the cost of potential hallucinations and lower precision.**

SkillSpector, NVIDIA’s security scanning framework for AI skills, implements a dual‑mode analysis pipeline that toggles between deterministic pattern matching and generative LLM‑driven inspection. Understanding the accuracy trade‑offs between static‑only and full LLM modes—controlled via the `use_llm` flag in `SkillspectorState`—lets you optimize scan configurations for precision, recall, and operational constraints.

## How Static‑Only Analysis Works

Static‑only mode runs a collection of deterministic pattern‑matchers that scan raw file contents without invoking external language models. In [`src/skillspector/nodes/analyzers/static_patterns_prompt_injection.py`](https://github.com/NVIDIA/SkillSpector/blob/main/src/skillspector/nodes/analyzers/static_patterns_prompt_injection.py) and sibling modules, the analyzer executes regular expressions, Unicode‑tag scans, and other hard‑coded heuristics to identify known vulnerability signatures.

Because every finding maps directly to a concrete regex match or pattern hit, this mode offers **very high precision** with rare false positives. However, it suffers from **limited recall**: it cannot detect novel threat variants, complex data‑flow bugs, or logic flaws that fall outside the encoded pattern set. When `use_llm` is set to `False` (via the `--no-llm` CLI flag), the graph bypasses all LLM calls and routes execution through the static‑runner node ([`static_runner.py`](https://github.com/NVIDIA/SkillSpector/blob/main/static_runner.py)), returning only pattern‑based findings.

## How Full LLM Analysis Works

Full LLM analysis utilizes the `LLMAnalyzerBase` class defined in [`src/skillspector/llm_analyzer_base.py`](https://github.com/NVIDIA/SkillSpector/blob/main/src/skillspector/llm_analyzer_base.py) to process files through chat models. For each file or chunk, the node constructs a prompt using `BASE_ANALYSIS_PROMPT`, sends it to the configured model, and parses the structured response into `Finding` objects.

This mode achieves **higher recall** because the model reasons about code context, comments, and developer intent, surfacing issues that static regexes never anticipate. The trade‑off is **lower precision**: the LLM may hallucinate findings, produce overly‑generic results, or misinterpret ambiguous prompts. The meta‑analyzer node ([`meta_analyzer.py`](https://github.com/NVIDIA/SkillSpector/blob/main/meta_analyzer.py)) orchestrates this process by first running static checks, then optionally invoking LLM passes, and merging the result sets.

## Controlling the Analysis Mode

The switch between modes is governed by the `use_llm` boolean stored in the graph state. In [`src/skillspector/state.py`](https://github.com/NVIDIA/SkillSpector/blob/main/src/skillspector/state.py) (line 80), the `SkillspectorState` dataclass defines this flag, which defaults to `True` but can be overridden via the command line:

```bash

# Static‑only scan (no API key required)

skill-spector scan /path/to/skill --no-llm -o report.json

# Full LLM scan (requires OPENAI_API_KEY or equivalent)

export OPENAI_API_KEY=sk-....
skill-spector scan /path/to/skill -o report.json

```

When `use_llm` is `True` but API calls fail (missing keys, rate limits, or timeouts), the reporting node in [`src/skillspector/nodes/report.py`](https://github.com/NVIDIA/SkillSpector/blob/main/src/skillspector/nodes/report.py) tracks these failures and injects an `_llm_degradation_notice` into the final SARIF/JSON output, alerting you that results may be incomplete.

## Accuracy Characteristics in Practice

The accuracy gap between modes is observable in SkillSpector’s test suite:

- **[`test_semantic_quality_policy.py`](https://github.com/NVIDIA/SkillSpector/blob/main/test_semantic_quality_policy.py)** verifies that with `use_llm=False`, a clean file returns zero findings, while with `use_llm=True`, the LLM may still return findings based on semantic interpretation.
- **[`test_meta_analyzer_use_llm.py`](https://github.com/NVIDIA/SkillSpector/blob/main/test_meta_analyzer_use_llm.py)** demonstrates that missing API credentials raise a `ValueError` when `use_llm=True`, but the scan succeeds (falling back to static‑only results) when `use_llm=False`.

**Static‑only** provides a **baseline guarantee**: deterministic execution, no hallucinations, fast runtime, and zero external dependencies. **Full LLM** adds **context‑aware coverage** capable of detecting complex injection vectors and logic bugs, but requires API credentials and manual triage to filter false positives.

## Running Both Modes Programmatically

You can invoke either mode directly via Python:

```python

# Full LLM analysis (single file)

from skillspector.llm_analyzer_base import LLMAnalyzerBase

prompt = "Detect insecure patterns in the following Python file."
llm = LLMAnalyzerBase(base_prompt=prompt, model="gpt-4")
batch = llm.get_batches(
    file_paths=["example.py"],
    file_cache={"example.py": open("example.py").read()},
)
results = llm.run_batches(batch)
findings = llm.collect_findings(results)  # list[Finding]

print(f"LLM discovered {len(findings)} issues")

```

```python

# Static‑only analysis (no LLM)

from skillspector.nodes.analyzers import static_patterns_prompt_injection as spi
from skillspector.state import SkillspectorState

state = SkillspectorState(
    skill_path="/tmp/skill",
    file_cache={"README.md": "# Example"},

    use_llm=False,
)
static_findings = spi.node(state)["findings"]
print(f"Static scan found {len(static_findings)} issues")

```

## Summary

- **Static‑only analysis** in SkillSpector offers high precision and zero false positives through deterministic regex and pattern matching in modules like [`static_patterns_prompt_injection.py`](https://github.com/NVIDIA/SkillSpector/blob/main/static_patterns_prompt_injection.py), but cannot detect novel or context‑dependent threats.
- **Full LLM analysis** via `LLMAnalyzerBase` increases recall by reasoning about code semantics and developer intent, though it introduces potential hallucinations and requires external API credentials.
- The **`use_llm` flag** in `SkillspectorState` (defined in [`state.py`](https://github.com/NVIDIA/SkillSpector/blob/main/state.py)) and the **`--no-llm` CLI option** provide granular control over which accuracy profile to apply per scan.
- When LLM calls fail during a requested full scan, [`report.py`](https://github.com/NVIDIA/SkillSpector/blob/main/report.py) emits a degradation notice so you know results have fallen back to static‑only coverage.

## Frequently Asked Questions

### When should I choose static‑only analysis over full LLM analysis?

Choose **static‑only** when you need fast, deterministic scans without API dependencies or when operating in air‑gapped environments. Use it for CI pipelines where false positives break builds and you only need to catch known vulnerability signatures. Choose **full LLM** when auditing unfamiliar codebases, reviewing complex data‑flow scenarios, or hunting for novel attack patterns that lack existing regex signatures.

### How does SkillSpector handle LLM failures during a scan?

When `use_llm=True` but API calls fail (network errors, invalid keys, or rate limits), the meta‑analyzer ([`meta_analyzer.py`](https://github.com/NVIDIA/SkillSpector/blob/main/meta_analyzer.py)) continues execution but the report node ([`report.py`](https://github.com/NVIDIA/SkillSpector/blob/main/report.py)) appends an `_llm_degradation_notice` to the output. This notice flags that the scan completed without LLM coverage, ensuring you are not unknowingly viewing incomplete results.

### Can I combine static and LLM analysis in the same scan?

Yes. By default, `use_llm=True` runs both modes sequentially: the meta‑analyzer first executes static pattern checks, then submits files to the LLM, and merges the findings. You receive the union of deterministic detections and model‑generated insights, with the static results serving as a reliable baseline even if the LLM produces noisy output.

### What types of vulnerabilities can only the LLM detect?

The LLM excels at identifying **semantic vulnerabilities** such as subtle prompt injection vectors hidden in natural‑language comments, logic flaws requiring cross‑file context, and insecure defaults that do not match known regex patterns. Static analysis cannot detect these because they require reasoning about intent and context rather than matching predetermined signatures.