Understanding the Limitations of SkillSpector's Static Analysis

SkillSpector's static analysis cannot detect runtime-generated code, encrypted payloads, or non-English text assets, and relies on regex patterns that may produce false positives in documentation.

NVIDIA's SkillSpector uses a fast, regex- and AST-based scanning pipeline to identify risky code patterns without executing the skill. While this approach provides broad coverage and rapid feedback, inherent constraints in the static analysis stage create blind spots for dynamic behaviors and certain file types. Understanding these limitations helps security teams interpret scan results accurately and decide when to augment static analysis with the optional LLM stage.

Language and Script Scope Constraints

The static scanners in src/skillspector/nodes/analyzers/static_runner.py only parse plain-text source files. Non-English text, image-embedded code, or encrypted and binary payloads are ignored entirely during the scan. This means that malicious instructions hidden in image metadata, non-ASCII documentation, or obfuscated binary blobs will bypass detection, as the analyzer lacks optical character recognition (OCR) or decryption capabilities.

Runtime Execution Gap

Static analysis operates entirely without executing the skill code. In src/skillspector/nodes/analyzers/static_runner.py, the tool inspects file contents using regular expressions and AST traversal, but it never invokes the skill. Consequently, it cannot observe dynamic behaviors such as code generated at runtime through exec() or eval(), or logic that depends on external inputs and environment variables. Threats that only manifest during execution remain invisible to this stage.

Pattern-Driven False Positives

The static stage uses a large collection of regular-expression patterns (e.g., PE1-PE4 in static_patterns_privilege_escalation.py) combined with AST-based dangerous-call detectors. This pattern-matching approach yields high recall but can produce false positives, especially when risky patterns appear in documentation strings or example code blocks. While the runner applies heuristics in static_runner.py (lines 30-55) to detect documentation and down-weight code examples, some spurious findings remain inevitable when analyzing complex codebases.

File Type Inference Limitations

SkillSpector infers a file's type from its extension using the FILE_TYPES mapping in static_runner.py (lines 31-39). Files with uncommon extensions, mixed content types, or missing extensions may be misclassified as "other" or skipped entirely. This can cause the scanner to miss vulnerable code in configuration files, template engines, or polyglot scripts that don't match expected patterns.

Binary and Large File Exclusions

To maintain performance and safety, the analyzer excludes any file larger than 1 MiB or detected as binary (by extension or null-byte scan), as implemented in static_runner.py (lines 51-59). This size and format filtering prevents the tool from scanning compiled binaries, machine learning model weights, or large data files that might contain embedded malicious payloads.

Dependency Vulnerability Fallbacks

The SC4 (Supply Chain) detector queries OSV.dev for known CVEs affecting dependencies. However, when network access is unavailable, the static analysis continues operation using a small embedded fallback list. This offline mode means some vulnerable dependencies may be missed if they are not present in the local cache, potentially leaving known-exploitable packages undetected.

Lack of Semantic Context Without LLM

When running with the --no-llm flag, SkillSpector lacks the higher-level reasoning provided by the LLM stage. The optional LLM can disambiguate developer intent—distinguishing a harmful instruction from a benign comment or safe example code. Without this semantic layer, the static analysis relies solely on pattern matches, which can misinterpret nuanced code or miss novel attack vectors that don't match existing regex signatures.

Running Static-Only Analysis

You can invoke the static analyzer directly to observe these limitations in practice.

Run a static-only scan from the command line:

skillspector scan ./my-skill/ --no-llm

Programmatically invoke the static runner while disabling the LLM stage:

from skillspector.graph import invoke

# Static-only analysis (use_llm=False)

result = invoke({
    "input_path": "./my-skill/",
    "output_format": "json",
    "use_llm": False,          # disables the LLM stage

})

print("Risk score:", result["risk_score"])
for f in result["filtered_findings"]:
    print(f"[{f['severity']}] {f['rule_id']}: {f['message']}")

These examples trigger the analysis pipeline defined in src/skillspector/nodes/analyzers/static_runner.py and the pattern modules like static_patterns_privilege_escalation.py, producing findings that reflect the constraints outlined above.

Summary

• Static analysis cannot inspect runtime-generated code or observe dynamic behaviors since it never executes the skill.
• Non-text assets are invisible to the scanner, including images, encrypted payloads, and binary files over 1 MiB.
• Pattern matching produces noise that requires heuristic filtering to reduce false positives in documentation.
• File type detection relies on extensions, potentially skipping files with unusual or missing extensions.
• Offline dependency checks use a limited fallback list, possibly missing recent CVEs when OSV.dev is unreachable.
• Semantic reasoning requires the LLM stage; static-only mode lacks context to distinguish benign patterns from malicious intent.

Frequently Asked Questions

Can SkillSpector detect malware hidden in images or binary files?

No. According to the source code in static_runner.py, SkillSpector's static analysis only processes plain-text source files. It ignores binary files, encrypted payloads, and non-English text assets. Any malicious code embedded in image metadata or compiled binaries will not be detected by the static scanner.

Why does SkillSpector miss vulnerabilities that only appear at runtime?

The static analysis never executes the skill code. As documented in the README and implemented in static_runner.py, the tool relies on regex and AST parsing rather than runtime instrumentation. Code that generates malicious instructions dynamically through exec(), eval(), or external input manipulation will not trigger static detections.

How does SkillSpector handle false positives in documentation?

The static runner applies heuristics to detect documentation files and down-weight code examples, as seen in static_runner.py (lines 30-55). However, pattern-driven detection inherently produces false positives when risky regex patterns appear in comments or README files. Running the full pipeline with the LLM stage enabled helps disambiguate these benign contexts.

What happens when SkillSpector runs without internet access?

When network access is unavailable, the dependency vulnerability checker (SC4) falls back to a small embedded list of known CVEs rather than querying OSV.dev. This means the static analysis may miss recently disclosed vulnerabilities in dependencies, as noted in the repository documentation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}