# How to Run Tests and Add New Test Cases for SkillSpector Analyzers

> Learn how to run tests and add new test cases for SkillSpector analyzers. Discover how pytest and Makefile orchestrate comprehensive testing for your analyzers.

- Repository: [NVIDIA Corporation/SkillSpector](https://github.com/NVIDIA/SkillSpector)
- Tags: how-to-guide
- Published: 2026-06-20

---

**SkillSpector uses pytest orchestrated by a Makefile to execute unit, integration, and provider tests, and new analyzer test cases are added under `tests/nodes/analyzers/` using the `static_runner.run_static_patterns` helper to validate detection logic against synthetic code snippets.**

NVIDIA/SkillSpector is an open-source framework for detecting vulnerabilities and anti-patterns in AI-generated code through static analysis. The repository maintains a robust test suite that distinguishes between fast static checks and live LLM invocations, ensuring developers can validate analyzer behavior efficiently without external API dependencies.

## Running the SkillSpector Test Suite

The project organizes tests into three categories using pytest markers. Each category maps to a specific Make target in the root `Makefile`, allowing you to run exactly the validation layer you need.

### Unit Tests (Fast Static Analysis)

**Unit tests** cover pure static-analysis checks that never invoke an LLM or external service. These provide immediate feedback during development and are the default test category.

Run the unit test suite with:

```bash
make test-unit

```

Or use the shorthand that also includes integration tests:

```bash
make test

```

### Integration and Provider Tests

**Integration tests** execute the full analysis graph and may optionally call a live LLM, while **provider tests** require valid API keys for external services (OpenAI, Anthropic, NVIDIA Build).

To run integration tests:

```bash
make test-integration

```

To execute live provider tests after exporting your credentials:

```bash
export OPENAI_API_KEY=sk-...
make test-provider

```

## Test Discovery Layout

All test modules reside under the `tests/` directory, with analyzer-specific tests located in `tests/nodes/analyzers/`. Each test imports the analyzer’s public `node()` entry point and feeds synthetic source strings to the validation helper.

For example, the **Tool-Misuse** analyzer implementation in [`src/skillspector/nodes/analyzers/static_patterns_tool_misuse.py`](https://github.com/NVIDIA/SkillSpector/blob/main/src/skillspector/nodes/analyzers/static_patterns_tool_misuse.py) has a corresponding unit test at [`tests/nodes/analyzers/test_static_patterns_tool_misuse.py`](https://github.com/NVIDIA/SkillSpector/blob/main/tests/nodes/analyzers/test_static_patterns_tool_misuse.py). The test invokes `static_runner.run_static_patterns` and asserts on the resulting `AnalyzerFinding` objects, checking for expected rule IDs, severity levels, and code locations.

## Adding New Test Cases for Analyzers

When extending SkillSpector with new detection rules, follow this workflow to add comprehensive test coverage.

### Step 1: Create the Test File

Create a new file under `tests/nodes/analyzers/` named `test_<analyzer_name>.py` to match the existing naming convention. If you are adding tests to an existing analyzer, edit the corresponding `test_*.py` file already present in that directory.

### Step 2: Import Required Modules

Import the analyzer node and the static runner utilities at the top of your test file:

```python
from skillspector.nodes.analyzers.static_patterns_tool_misuse import node as tool_misuse_node
from skillspector.nodes.analyzers import static_runner
from skillspector.models import AnalyzerFinding, Severity
from skillspector.state import SkillspectorState

```

### Step 3: Write a Source Snippet

Define a minimal code string that should trigger (or explicitly not trigger) your detection rule:

```python
MALICIOUS_CODE = """
import subprocess

# classic shell-true abuse

subprocess.run("rm -rf /", shell=True)
"""

```

### Step 4: Invoke and Assert

Feed the snippet through the static runner and assert on the findings. The `static_runner.run_static_patterns` function handles the boilerplate of executing the analyzer against your synthetic input:

```python
def test_tool_parameter_abuse():
    findings = static_runner.run_static_patterns(
        state=SkillspectorState(input_path="dummy"),
        analyzers=[tool_misuse_node.__module__],
        content=MALICIOUS_CODE,
        file_path="dummy.py",
        file_type="python",
    )
    # Verify at least one HIGH severity finding for rule TM1

    assert any(f.rule_id == "TM1" and f.severity == Severity.HIGH for f in findings)

```

### Step 5: Run Locally

Verify your new test passes before committing:

```bash
pytest tests/nodes/analyzers/test_static_patterns_tool_misuse.py

```

The CI pipeline will automatically run the full suite via `make test` on push, ensuring your new case does not break existing behavior.

## Essential Test Utilities

**`static_runner.run_static_patterns`** – A thin wrapper in [`src/skillspector/nodes/analyzers/static_runner.py`](https://github.com/NVIDIA/SkillSpector/blob/main/src/skillspector/nodes/analyzers/static_runner.py) that loads content, applies specified analyzer modules, and returns a list of `AnalyzerFinding` objects.

**`SkillspectorState`** – The mutable state container defined in [`src/skillspector/state.py`](https://github.com/NVIDIA/SkillSpector/blob/main/src/skillspector/state.py) that flows through the analysis graph. For unit tests, instantiate it with only the `input_path` argument.

**`Severity` and `AnalyzerFinding`** – Data classes defined in [`src/skillspector/models.py`](https://github.com/NVIDIA/SkillSpector/blob/main/src/skillspector/models.py) that encapsulate detection results, including rule IDs, severity levels, confidence scores, and source locations.

## Summary

- **Execute** the full test suite with `make test` or isolate unit tests with `make test-unit`.
- **Locate** analyzer tests under `tests/nodes/analyzers/` following the `test_<analyzer>.py` naming convention.
- **Validate** new detection logic using `static_runner.run_static_patterns` with synthetic code snippets.
- **Assert** on `AnalyzerFinding` attributes including `rule_id`, `severity`, and `confidence` to ensure precise detection.
- **Verify** locally with pytest before pushing; the CI pipeline runs `make test` automatically.

## Frequently Asked Questions

### How do I run only the tests for a specific analyzer?

Navigate to the specific test file and run pytest directly. For example, to test only the tool-misuse analyzer: `pytest tests/nodes/analyzers/test_static_patterns_tool_misuse.py -v`. This executes faster than the full suite and provides immediate feedback on your specific changes.

### What is the difference between unit and integration tests in SkillSpector?

**Unit tests** validate static pattern matching against synthetic code without network calls or LLM invocations, making them fast and deterministic. **Integration tests** execute the full analysis graph and may optionally invoke live LLM services, testing the complete pipeline end-to-end.

### Do I need API keys to run the test suite?

No. The default `make test` and `make test-unit` commands run entirely offline. You only need API keys (such as `OPENAI_API_KEY` or `ANTHROPIC_API_KEY`) when running `make test-provider`, which validates live integrations with external AI services.

### How do I assert that a code snippet produces no findings?

Pass the "clean" code to `static_runner.run_static_patterns` and assert that the returned list is empty. For example: `assert len(findings) == 0` or `assert not any(f.rule_id == "TM1" for f in findings)`. This negative testing ensures your analyzer does not produce false positives on benign code.