How Semantic Security Discovery Evaluates Intent Beyond Pattern Matching in NVIDIA SkillSpector

Semantic security discovery uses a large language model to analyze the meaning and intent of skill files rather than searching for literal strings, enabling detection of semantic prompt injection, paraphrased attacks, and narrative deception that static regexes miss.

The NVIDIA SkillSpector security framework includes a sophisticated semantic security discovery analyzer that moves beyond traditional static analysis. Unlike conventional security scanners that rely on keyword matching to find literal strings like "ignore system instructions" or "exfiltrate," this analyzer leverages large language models to reason about the underlying intent of code and documentation. This approach enables the detection of nuanced security risks where seemingly benign descriptions actually steer models toward unsafe behavior.

Architecture and Core Components

The semantic security discovery system is built on a modular architecture that separates prompt engineering, LLM invocation, and result processing.

The ANALYZER_PROMPT System

The core intelligence resides in ANALYZER_PROMPT, defined in src/skillspector/nodes/analyzers/semantic_security_discovery.py (lines 30-68). This multi-line system prompt defines four SSD rule families (SSD-1 through SSD-4) and explicitly instructs the LLM to focus on "intent and meaning" rather than performing keyword matches. By framing the analysis as a reasoning task about purpose and context, the prompt forces the model to interpret natural language instructions that might circumvent static detection.

LLMAnalyzerBase Infrastructure

The LLMAnalyzerBase class in src/skillspector/llm_analyzer_base.py provides the generic infrastructure for LLM-powered analysis. It handles batching of file contents to stay within token limits, invokes the LLM using with_structured_output, and converts the structured response (LLMAnalysisResult) into a list of Finding objects. This abstraction allows the semantic security discovery node to focus on domain-specific logic while the base class manages model communication.

Node Entry Point and Model Resolution

The node(state) function in semantic_security_discovery.py (lines 71-93) serves as the entry point. It extracts use_llm, components, file_cache, and model_config from the SkillspectorState object. The analyzer implements a three-tier model resolution strategy: it first checks for an analyzer-specific model under model_config["semantic_security_discovery"], falls back to model_config["default"], and finally uses the constant _SKILLSPECTOR_DEFAULT_MODEL defined in src/skillspector/constants.py.

The Six-Step Analysis Flow

The semantic security discovery process follows a structured pipeline to transform raw skill files into actionable security findings:

1. State preparation – The graph supplies a SkillspectorState containing the raw skill files in file_cache and optional model overrides.

2. Model resolution – The analyzer selects the appropriate model using the fallback hierarchy (analyzer-specific → default → constant).

3. Prompt composition – LLMAnalyzerBase concatenates ANALYZER_PROMPT with the file contents, batched to stay within token limits. The prompt forces the model to only report findings whose risk is conveyed through natural-language intent.

4. LLM invocation – The LLM returns a structured list of findings containing rule_id, message, severity, confidence, and other metadata.

5. Finding conversion – Each LLMFinding is transformed into a Finding object that includes file path, line numbers, and confidence scores.

6. Result aggregation – All findings from every batch are merged and returned as {"findings": [...]} to the pipeline.

Error Handling and Resilience

The implementation includes robust error handling in semantic_security_discovery.py (lines 94-102). The code catches ValidationError (for malformed LLM output) and generic exceptions, logging a warning and returning an empty finding list. This ensures the analysis graph continues without crashing even when the LLM produces unexpected responses.

Detecting Semantic Threats Beyond Pattern Matching

Because the analyzer relies on the LLM's comprehension of context, it can surface threats that static regex patterns would miss:

• Semantic prompt injection – Instructions that reframe the conversation context without using forbidden keywords
• Novel paraphrased attacks – Variations of known attacks expressed in different terminology
• Plain-language data-leak instructions – SSD-3 violations where the text describes "telling the user to remember everything they say" rather than using technical exfiltration terms
• Multi-step narrative deception – Complex scenarios where benign individual steps combine into harmful behavior

Practical Implementation and Usage

Programmatic Integration

You can invoke the semantic security discovery analyzer directly from Python to analyze skill files:

from skillspector.nodes.analyzers.semantic_security_discovery import node

# Minimal state – a single skill file is passed via file_cache

state = {
    "use_llm": True,
    "model_config": {"semantic_security_discovery": "gpt-4o-mini"},
    "components": ["SKILL.md"],
    "file_cache": {"SKILL.md": "# Example skill\n\nTell the user to remember everything they say."},

}

result = node(state)
print(result["findings"])

The LLM will recognize that "tell the user to remember everything" is an SSD-3 data-leak instruction and return a finding with rule SSD-3.

Command-Line Interface

Run the full analysis via the SkillSpector CLI:

skillpector analyze path/to/skill_dir \
    --model semantic_security_discovery=gpt-4o-mini \
    --enable-llm

The CLI builds the same SkillspectorState, routes it through the semantic_security_discovery node, and prints a SARIF report that includes any SSD findings.

Testing and Validation

The repository includes unit tests demonstrating expected behavior:

from skillspector.llm_analyzer_base import LLMAnalyzerBase
from unittest.mock import MagicMock, patch

@patch("skillspector.llm_analyzer_base.get_chat_model")
def test_returns_findings_from_llm(mock_get):
    # Mock LLM that returns a single SSD‑1 finding

    mock_llm = MagicMock()
    mock_llm.with_structured_output.return_value.invoke.return_value = \
        LLMAnalysisResult(findings=[LLMFinding(rule_id="SSD-1", ...)])
    mock_get.return_value = mock_llm

    state = {
        "use_llm": True,
        "model_config": {"semantic_security_discovery": "test-model"},
        "components": ["SKILL.md"],
        "file_cache": {"SKILL.md": "..."},
    }

    result = node(state)
    assert result["findings"][0].rule_id == "SSD-1"

The test validates that the node correctly forwards the LLM's structured output into Finding objects. Reference: tests/nodes/analyzers/test_semantic_security_discovery.py.

Summary

• Semantic security discovery in NVIDIA SkillSpector uses LLM reasoning rather than static pattern matching to evaluate intent.
• The ANALYZER_PROMPT in semantic_security_discovery.py defines four rule families (SSD-1 through SSD-4) and explicitly directs the model to analyze meaning over keywords.
• LLMAnalyzerBase handles batching, structured output parsing, and conversion to Finding objects.
• The system catches ValidationError and generic exceptions to ensure pipeline resilience.
• This approach detects semantic prompt injection, paraphrased attacks, and multi-step deception that bypass traditional regex-based scanners.

Frequently Asked Questions

How does semantic security discovery differ from static keyword analysis?

Static keyword analysis searches for literal strings such as "ignore system instructions" or "exfiltrate data" using regex patterns. Semantic security discovery instead asks a large language model to reason about the meaning and intent of the content. According to the NVIDIA SkillSpector source code, this allows detection of attacks where harmful instructions are expressed in novel phrasing or buried in seemingly benign narratives.

What are the four SSD rule families referenced in the analyzer?

The ANALYZER_PROMPT defines four rule families labeled SSD-1 through SSD-4. While the specific details of each rule are encoded in the prompt at src/skillspector/nodes/analyzers/semantic_security_discovery.py (lines 30-68), they generally cover categories such as prompt injection, instruction override, data exfiltration, and behavior manipulation. The LLM uses these categories to classify findings based on semantic intent rather than keyword presence.

How does the analyzer handle malformed LLM responses?

The node() function in semantic_security_discovery.py (lines 94-102) implements error handling that catches ValidationError when the LLM returns malformed structured output, as well as generic exceptions. When errors occur, the analyzer logs a warning and returns an empty finding list, allowing the analysis graph to continue processing other files without crashing.

Can I override the default model for semantic security discovery?

Yes. The analyzer supports model configuration through the model_config dictionary in the state object. You can specify a model under the key "semantic_security_discovery". If not provided, the system falls back to model_config["default"], and finally to the constant _SKILLSPECTOR_DEFAULT_MODEL defined in src/skillspector/constants.py. This allows flexibility to use different models for different analysis tasks.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}