How SkillSpector Determines Confidence Scores for Security Findings

SkillSpector assigns confidence scores through either direct LLM self-assessment via the LLMFinding schema or deterministic heuristics in static-pattern analyzers, with both pathways clamping values to the [0, 1] range before storage in the Finding model.

SkillSpector is an open-source security analysis framework developed by NVIDIA that evaluates code for potential vulnerabilities and malicious patterns. Every finding generated by the tool carries a confidence score—a normalized float between 0 and 1—indicating the certainty of the detection. According to the NVIDIA/SkillSpector source code, the framework employs two distinct methodologies to calculate these scores depending on whether the analysis originates from an LLM-based or static-pattern analyzer.

LLM-Based Confidence Scoring

For analyzers like semantic_security_discovery and semantic_developer_intent, confidence scores originate directly from the model itself. The framework defines an LLMFinding Pydantic model in src/skillspector/llm_analyzer_base.py that requires the LLM to return a structured output including a confidence field.

The LLMFinding class constrains the value using Pydantic validation:

class LLMFinding(BaseModel):
    confidence: float = Field(ge=0.0, le=1.0, default=0.5, description="Confidence score")
    
    def to_finding(self, file: str) -> Finding:
        return Finding(
            rule_id=self.rule_id,
            message=self.message,
            severity=self.severity,
            confidence=self.confidence,  # Directly propagated

            file=file,
            start_line=self.start_line,
            end_line=self.end_line,
            explanation=self.explanation,
            remediation=self.remediation,
        )

The system prompts explicitly instruct the model to report only findings with confidence ≥ 0.6, leveraging the LLM's internal probability distributions and token-level logits to self-assess certainty. The framework stores this value without modification.

Static-Pattern Confidence Calculation

For rule-based analyzers such as static_patterns_tool_misuse, static_patterns_privilege_escalation, and static_patterns_prompt_injection, confidence is computed programmatically within each detection module. These analyzers instantiate a Finding object with a base confidence—typically 0.9 for strong literal matches—then apply contextual heuristics to adjust the score.

Heuristic Adjustment Factors

The static-pattern modules implement several confidence modifiers:

• File type penalties: Matches discovered in test helper files receive reduced confidence (below 0.75)
• Obfuscation detection: Presence of zero-width characters, HTML comments, or Base64-encoded payloads reduces scores (e.g., 0.85 → 0.75)
• Payload length: Excessively long payloads trigger down-weighting to approximately 0.65
• Trusted context: Matches appearing within trusted-source blocks may be boosted to 0.95

In src/skillspector/nodes/analyzers/static_patterns_tool_poisoning.py, the _confidence_for_html_comment function exemplifies this logic:

def _confidence_for_html_comment(match: str) -> float:
    # Strong literal match → high base confidence

    base = 0.90
    # If the comment contains an instruction keyword, boost a bit

    if "SYSTEM:" in match:
        base = 0.95
    # Apply a generic clamp to keep the score in range

    return _clamp(base)

# Later, when creating the finding:

conf = _confidence_for_html_comment(comment_text)
finding = Finding(
    rule_id="TP1",
    message="HTML comment with system instruction",
    severity="HIGH",
    confidence=conf,
    file=filepath,
    start_line=line_no,
)

The _clamp helper function ensures all final values remain within the [0, 1] boundary across all static-pattern modules.

Confidence Aggregation and Filtering

All findings ultimately populate the Finding dataclass defined in src/skillspector/models.py, which stores the final confidence attribute. The graph execution engine in src/skillspector/graph.py aggregates these findings and enables downstream filtering.

Consumers such as SARIF export modules and report generators typically apply thresholds (commonly ≥ 0.6) to surface only high-confidence results, preventing noise from uncertain pattern matches.

Summary

• LLM analyzers derive confidence directly from model self-assessment via the LLMFinding schema, with prompts enforcing minimum thresholds of 0.6
• Static-pattern analyzers calculate confidence using base scores (often 0.9) modified by heuristics for file type, obfuscation, payload length, and context
• The _clamp function ensures all scores remain normalized between 0 and 1
• The Finding model in models.py stores final scores, while graph.py enables confidence-based filtering for reports

Frequently Asked Questions

What is the minimum confidence threshold in SkillSpector?

SkillSpector's LLM-based analyzers use system prompts that instruct models to report only findings with confidence scores of 0.6 or higher. Downstream report generators typically apply similar thresholds to filter noise, though this is configurable in the graph execution layer.

How does SkillSpector handle ambiguous pattern matches?

Static-pattern analyzers reduce confidence scores for ambiguous matches through specific heuristics: matches in test files drop below 0.75, obfuscated content (such as zero-width characters or Base64) reduces scores by approximately 0.10, and unusually long payloads may decrease confidence to roughly 0.65.

Can confidence scores be adjusted or customized?

Yes. Since static-pattern analyzers implement confidence calculation locally within each module (such as static_patterns_tool_poisoning.py), developers can modify base values, adjustment factors, and heuristics. The _clamp function ensures any modifications remain within valid bounds.

Where are confidence scores stored in the SkillSpector codebase?

All confidence scores are stored in the confidence attribute of the Finding dataclass defined in src/skillspector/models.py. This model serves as the universal data structure for both LLM-based findings (converted via LLMFinding.to_finding()) and static-pattern findings created directly in analyzer modules.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}