How to Run Tests and Add New Test Cases for SkillSpector Analyzers

SkillSpector uses pytest orchestrated by a Makefile to execute unit, integration, and provider tests, and new analyzer test cases are added under tests/nodes/analyzers/ using the static_runner.run_static_patterns helper to validate detection logic against synthetic code snippets.

NVIDIA/SkillSpector is an open-source framework for detecting vulnerabilities and anti-patterns in AI-generated code through static analysis. The repository maintains a robust test suite that distinguishes between fast static checks and live LLM invocations, ensuring developers can validate analyzer behavior efficiently without external API dependencies.

Running the SkillSpector Test Suite

The project organizes tests into three categories using pytest markers. Each category maps to a specific Make target in the root Makefile, allowing you to run exactly the validation layer you need.

Unit Tests (Fast Static Analysis)

Unit tests cover pure static-analysis checks that never invoke an LLM or external service. These provide immediate feedback during development and are the default test category.

Run the unit test suite with:

make test-unit

Or use the shorthand that also includes integration tests:

make test

Integration and Provider Tests

Integration tests execute the full analysis graph and may optionally call a live LLM, while provider tests require valid API keys for external services (OpenAI, Anthropic, NVIDIA Build).

To run integration tests:

make test-integration

To execute live provider tests after exporting your credentials:

export OPENAI_API_KEY=sk-...
make test-provider

Test Discovery Layout

All test modules reside under the tests/ directory, with analyzer-specific tests located in tests/nodes/analyzers/. Each test imports the analyzer’s public node() entry point and feeds synthetic source strings to the validation helper.

For example, the Tool-Misuse analyzer implementation in src/skillspector/nodes/analyzers/static_patterns_tool_misuse.py has a corresponding unit test at tests/nodes/analyzers/test_static_patterns_tool_misuse.py. The test invokes static_runner.run_static_patterns and asserts on the resulting AnalyzerFinding objects, checking for expected rule IDs, severity levels, and code locations.

Adding New Test Cases for Analyzers

When extending SkillSpector with new detection rules, follow this workflow to add comprehensive test coverage.

Step 1: Create the Test File

Create a new file under tests/nodes/analyzers/ named test_<analyzer_name>.py to match the existing naming convention. If you are adding tests to an existing analyzer, edit the corresponding test_*.py file already present in that directory.

Step 2: Import Required Modules

Import the analyzer node and the static runner utilities at the top of your test file:

from skillspector.nodes.analyzers.static_patterns_tool_misuse import node as tool_misuse_node
from skillspector.nodes.analyzers import static_runner
from skillspector.models import AnalyzerFinding, Severity
from skillspector.state import SkillspectorState

Step 3: Write a Source Snippet

Define a minimal code string that should trigger (or explicitly not trigger) your detection rule:

MALICIOUS_CODE = """
import subprocess

# classic shell-true abuse

subprocess.run("rm -rf /", shell=True)
"""

Step 4: Invoke and Assert

Feed the snippet through the static runner and assert on the findings. The static_runner.run_static_patterns function handles the boilerplate of executing the analyzer against your synthetic input:

def test_tool_parameter_abuse():
    findings = static_runner.run_static_patterns(
        state=SkillspectorState(input_path="dummy"),
        analyzers=[tool_misuse_node.__module__],
        content=MALICIOUS_CODE,
        file_path="dummy.py",
        file_type="python",
    )
    # Verify at least one HIGH severity finding for rule TM1

    assert any(f.rule_id == "TM1" and f.severity == Severity.HIGH for f in findings)

Step 5: Run Locally

Verify your new test passes before committing:

pytest tests/nodes/analyzers/test_static_patterns_tool_misuse.py

The CI pipeline will automatically run the full suite via make test on push, ensuring your new case does not break existing behavior.

Essential Test Utilities

static_runner.run_static_patterns – A thin wrapper in src/skillspector/nodes/analyzers/static_runner.py that loads content, applies specified analyzer modules, and returns a list of AnalyzerFinding objects.

SkillspectorState – The mutable state container defined in src/skillspector/state.py that flows through the analysis graph. For unit tests, instantiate it with only the input_path argument.

Severity and AnalyzerFinding – Data classes defined in src/skillspector/models.py that encapsulate detection results, including rule IDs, severity levels, confidence scores, and source locations.

Summary

• Execute the full test suite with make test or isolate unit tests with make test-unit.
• Locate analyzer tests under tests/nodes/analyzers/ following the test_<analyzer>.py naming convention.
• Validate new detection logic using static_runner.run_static_patterns with synthetic code snippets.
• Assert on AnalyzerFinding attributes including rule_id, severity, and confidence to ensure precise detection.
• Verify locally with pytest before pushing; the CI pipeline runs make test automatically.

Frequently Asked Questions

How do I run only the tests for a specific analyzer?

Navigate to the specific test file and run pytest directly. For example, to test only the tool-misuse analyzer: pytest tests/nodes/analyzers/test_static_patterns_tool_misuse.py -v. This executes faster than the full suite and provides immediate feedback on your specific changes.

What is the difference between unit and integration tests in SkillSpector?

Unit tests validate static pattern matching against synthetic code without network calls or LLM invocations, making them fast and deterministic. Integration tests execute the full analysis graph and may optionally invoke live LLM services, testing the complete pipeline end-to-end.

Do I need API keys to run the test suite?

No. The default make test and make test-unit commands run entirely offline. You only need API keys (such as OPENAI_API_KEY or ANTHROPIC_API_KEY) when running make test-provider, which validates live integrations with external AI services.

How do I assert that a code snippet produces no findings?

Pass the "clean" code to static_runner.run_static_patterns and assert that the returned list is empty. For example: assert len(findings) == 0 or assert not any(f.rule_id == "TM1" for f in findings). This negative testing ensures your analyzer does not produce false positives on benign code.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}