How the semantic_developer_intent Analyzer Evaluates Skill Purpose with LLM in SkillSpector

The semantic_developer_intent analyzer uses a large language model to compare a skill's declared manifest purpose against its actual code behavior, flagging semantic mismatches using four structured SDI rules.

The semantic_developer_intent analyzer is a critical security component in NVIDIA's SkillSpector repository that validates whether AI skills perform only what their developers claim. This LLM-driven node inspects the gap between manifest declarations and implementation by applying structured audit rules to the source code. Understanding how this analyzer works helps developers identify privilege escalations and scope creep before deployment.

How the Semantic Developer Intent Analyzer Works

The analyzer operates as a pipeline node that receives state from the SkillSpector graph and orchestrates an LLM to perform semantic code analysis. In src/skillspector/nodes/analyzers/semantic_developer_intent.py, the node function extracts the skill manifest and source files, then delegates the heavy lifting to LLMAnalyzerBase for batch processing.

Step-by-Step LLM Evaluation Process

1. Context Collection from SkillspectorState

The node first gathers context from the shared state dictionary. According to lines 66-73 in semantic_developer_intent.py, it retrieves:

• manifest: The skill's YAML metadata (name, description, permissions, triggers)
• file_cache: A mapping of filenames to source code strings
• model_config: Optional overrides for the LLM model selection

manifest: dict = state.get("manifest") or {}
file_cache: dict[str, str] = state.get("file_cache") or {}
model_config: dict[str, str] = state.get("model_config") or {}

2. Manifest Formatting for the Prompt

The helper _format_manifest (lines 35-55) converts the dictionary into a human-readable text block interpolated into the system prompt. If the manifest is missing, the function returns a placeholder string, allowing the LLM to flag any detected capabilities as unauthorized.

3. Building the Analyzer Prompt with SDI Rules

The analyzer constructs the ANALYZER_PROMPT defined in lines 35-132, which contains:

• A detailed description of the auditor role
• Four specific rule IDs (SDI-1 through SDI-4)
• Instructions to reference source line numbers using L- prefixes

The manifest block inserts at {manifest_section}, creating a prompt that directs the LLM to compare declared intent against actual implementation.

4. LLM Interaction via LLMAnalyzerBase

Lines 76-81 instantiate the base analyzer and process files in optimized batches:

analyzer = LLMAnalyzerBase(base_prompt=prompt, model=model)
batches = analyzer.get_batches(sorted(file_cache), file_cache)
results = asyncio.run(analyzer.arun_batches(batches))
findings = analyzer.collect_findings(results)

The LLMAnalyzerBase class (in src/skillspector/llm_analyzer_base.py) handles context-window chunking, async API calls to configured providers (OpenAI, Anthropic, etc.), and parsing of JSON-structured responses into finding objects.

5. Result Handling and Error Resilience

Lines 82-88 implement graceful degradation. The node returns {"findings": findings} on success, or logs a warning and returns an empty list if exceptions occur, ensuring pipeline continuity even when LLM calls fail.

The Four SDI Rules for Semantic Validation

The analyzer evaluates skills against four specific criteria defined in the system prompt:

• SDI-1: Detects description-behavior mismatches (e.g., a "summarize" skill performing network calls)
• SDI-2: Flags capabilities unjustified by declared purpose (e.g., a formatter spawning shells)
• SDI-3: Identifies scope creep relative to permissions (e.g., writing files when only read access is declared)
• SDI-4: Finds divergences between intent stated in comments or docstrings and actual code behavior

Each finding includes the rule ID, a brief description, and precise line numbers in L- prefixed format.

Code Examples

Direct Invocation for Unit Testing

You can invoke the analyzer directly in Python to test specific skill configurations:

from skillspector.nodes.analyzers.semantic_developer_intent import node
from skillspector.state import SkillspectorState

state: SkillspectorState = {
    "manifest": {
        "name": "WebSearch",
        "description": "Search the web and return a summary",
        "permissions": ["network"],
    },
    "file_cache": {
        "search.py": """
def run(query):
    import requests
    resp = requests.get(f"https://search.example/{query}")
    return resp.text
""",
    },
    "model_config": {"semantic_developer_intent": "gpt-4o"},
    "use_llm": True,
}

result = node(state)
print(result["findings"])

Command Line Usage

Run the analyzer through the SkillSpector CLI:

skillspector run ./my_skill/ --analyzers semantic_developer_intent

The CLI (src/skillspector/cli.py) automatically loads the manifest from skill.yaml, builds the file cache, and executes the analyzer graph.

Summary

• The semantic_developer_intent analyzer validates that skill implementation matches declared purpose using LLM reasoning in src/skillspector/nodes/analyzers/semantic_developer_intent.py.
• It processes manifest metadata and source code through the SkillspectorState interface, formatting inputs via _format_manifest and analyzing them with LLMAnalyzerBase.
• The four SDI rules (SDI-1 through SDI-4) provide structured criteria for detecting semantic mismatches, privilege escalations, and scope creep between declared and actual behavior.
• LLMAnalyzerBase handles async batch processing, automatic chunking, and provider-agnostic LLM calls via asyncio.run(analyzer.arun_batches(batches)).
• The analyzer returns structured findings with line numbers or fails gracefully with empty results to maintain pipeline stability when LLM services are unavailable.

Frequently Asked Questions

How does the semantic_developer_intent analyzer differ from static code analysis?

Static analyzers check syntax and known vulnerability patterns, while this analyzer uses an LLM to understand semantic intent. It evaluates whether the code's behavior aligns with the developer's stated purpose in the manifest, catching logical inconsistencies and unauthorized capabilities that pattern matching cannot detect.

What happens if the skill manifest is missing or incomplete?

The _format_manifest helper returns a placeholder string when the manifest is absent. The LLM then analyzes the code against an empty declaration, effectively flagging any capabilities as potential mismatches since nothing is explicitly authorized in the metadata.

Can I use a different LLM model for this specific analyzer?

Yes. Pass a model override in state["model_config"]["semantic_developer_intent"] or modify the default in src/skillspector/constants.py. The analyzer instantiates LLMAnalyzerBase with your specified model name, supporting OpenAI, Anthropic, and other configured providers through the unified provider interface.

How does the analyzer handle large codebases?

The LLMAnalyzerBase.get_batches method automatically chunks files into context-window-sized batches. It processes these asynchronously via asyncio.run(analyzer.arun_batches(batches)), allowing efficient analysis of large skills without exceeding token limits or hitting rate limits.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}