# How SkillSpector Determines Confidence Scores for Security Findings

> Learn how SkillSpector determines confidence scores for security findings. Explore LLM self-assessment and deterministic heuristics for accurate vulnerability identification.

- Repository: [NVIDIA Corporation/SkillSpector](https://github.com/NVIDIA/SkillSpector)
- Tags: internals
- Published: 2026-06-16

---

**SkillSpector assigns confidence scores through either direct LLM self-assessment via the `LLMFinding` schema or deterministic heuristics in static-pattern analyzers, with both pathways clamping values to the [0, 1] range before storage in the `Finding` model.**

SkillSpector is an open-source security analysis framework developed by NVIDIA that evaluates code for potential vulnerabilities and malicious patterns. Every finding generated by the tool carries a **confidence score**—a normalized float between 0 and 1—indicating the certainty of the detection. According to the NVIDIA/SkillSpector source code, the framework employs two distinct methodologies to calculate these scores depending on whether the analysis originates from an LLM-based or static-pattern analyzer.

## LLM-Based Confidence Scoring

For analyzers like `semantic_security_discovery` and `semantic_developer_intent`, confidence scores originate directly from the model itself. The framework defines an `LLMFinding` Pydantic model in [`src/skillspector/llm_analyzer_base.py`](https://github.com/NVIDIA/SkillSpector/blob/main/src/skillspector/llm_analyzer_base.py) that requires the LLM to return a structured output including a confidence field.

The `LLMFinding` class constrains the value using Pydantic validation:

```python
class LLMFinding(BaseModel):
    confidence: float = Field(ge=0.0, le=1.0, default=0.5, description="Confidence score")
    
    def to_finding(self, file: str) -> Finding:
        return Finding(
            rule_id=self.rule_id,
            message=self.message,
            severity=self.severity,
            confidence=self.confidence,  # Directly propagated

            file=file,
            start_line=self.start_line,
            end_line=self.end_line,
            explanation=self.explanation,
            remediation=self.remediation,
        )

```

The system prompts explicitly instruct the model to report only findings with confidence ≥ 0.6, leveraging the LLM's internal probability distributions and token-level logits to self-assess certainty. The framework stores this value without modification.

## Static-Pattern Confidence Calculation

For rule-based analyzers such as `static_patterns_tool_misuse`, `static_patterns_privilege_escalation`, and `static_patterns_prompt_injection`, confidence is computed programmatically within each detection module. These analyzers instantiate a `Finding` object with a base confidence—typically 0.9 for strong literal matches—then apply contextual heuristics to adjust the score.

### Heuristic Adjustment Factors

The static-pattern modules implement several confidence modifiers:

- **File type penalties**: Matches discovered in test helper files receive reduced confidence (below 0.75)
- **Obfuscation detection**: Presence of zero-width characters, HTML comments, or Base64-encoded payloads reduces scores (e.g., 0.85 → 0.75)
- **Payload length**: Excessively long payloads trigger down-weighting to approximately 0.65
- **Trusted context**: Matches appearing within trusted-source blocks may be boosted to 0.95

In [`src/skillspector/nodes/analyzers/static_patterns_tool_poisoning.py`](https://github.com/NVIDIA/SkillSpector/blob/main/src/skillspector/nodes/analyzers/static_patterns_tool_poisoning.py), the `_confidence_for_html_comment` function exemplifies this logic:

```python
def _confidence_for_html_comment(match: str) -> float:
    # Strong literal match → high base confidence

    base = 0.90
    # If the comment contains an instruction keyword, boost a bit

    if "SYSTEM:" in match:
        base = 0.95
    # Apply a generic clamp to keep the score in range

    return _clamp(base)

# Later, when creating the finding:

conf = _confidence_for_html_comment(comment_text)
finding = Finding(
    rule_id="TP1",
    message="HTML comment with system instruction",
    severity="HIGH",
    confidence=conf,
    file=filepath,
    start_line=line_no,
)

```

The `_clamp` helper function ensures all final values remain within the [0, 1] boundary across all static-pattern modules.

## Confidence Aggregation and Filtering

All findings ultimately populate the `Finding` dataclass defined in [`src/skillspector/models.py`](https://github.com/NVIDIA/SkillSpector/blob/main/src/skillspector/models.py), which stores the final confidence attribute. The graph execution engine in [`src/skillspector/graph.py`](https://github.com/NVIDIA/SkillSpector/blob/main/src/skillspector/graph.py) aggregates these findings and enables downstream filtering.

Consumers such as SARIF export modules and report generators typically apply thresholds (commonly ≥ 0.6) to surface only high-confidence results, preventing noise from uncertain pattern matches.

## Summary

- **LLM analyzers** derive confidence directly from model self-assessment via the `LLMFinding` schema, with prompts enforcing minimum thresholds of 0.6
- **Static-pattern analyzers** calculate confidence using base scores (often 0.9) modified by heuristics for file type, obfuscation, payload length, and context
- The `_clamp` function ensures all scores remain normalized between 0 and 1
- The `Finding` model in [`models.py`](https://github.com/NVIDIA/SkillSpector/blob/main/models.py) stores final scores, while [`graph.py`](https://github.com/NVIDIA/SkillSpector/blob/main/graph.py) enables confidence-based filtering for reports

## Frequently Asked Questions

### What is the minimum confidence threshold in SkillSpector?

SkillSpector's LLM-based analyzers use system prompts that instruct models to report only findings with confidence scores of 0.6 or higher. Downstream report generators typically apply similar thresholds to filter noise, though this is configurable in the graph execution layer.

### How does SkillSpector handle ambiguous pattern matches?

Static-pattern analyzers reduce confidence scores for ambiguous matches through specific heuristics: matches in test files drop below 0.75, obfuscated content (such as zero-width characters or Base64) reduces scores by approximately 0.10, and unusually long payloads may decrease confidence to roughly 0.65.

### Can confidence scores be adjusted or customized?

Yes. Since static-pattern analyzers implement confidence calculation locally within each module (such as [`static_patterns_tool_poisoning.py`](https://github.com/NVIDIA/SkillSpector/blob/main/static_patterns_tool_poisoning.py)), developers can modify base values, adjustment factors, and heuristics. The `_clamp` function ensures any modifications remain within valid bounds.

### Where are confidence scores stored in the SkillSpector codebase?

All confidence scores are stored in the `confidence` attribute of the `Finding` dataclass defined in [`src/skillspector/models.py`](https://github.com/NVIDIA/SkillSpector/blob/main/src/skillspector/models.py). This model serves as the universal data structure for both LLM-based findings (converted via `LLMFinding.to_finding()`) and static-pattern findings created directly in analyzer modules.