Two Signals Used for Computing Confidence Score in Needle: A Complete Guide

Needle computes confidence scores by taking the minimum of two independent signals: a post-hoc calibration head and the decoding probability of call tokens.

The Needle framework by Cactus Compute attaches a calibrated confidence value to every response, enabling developers to filter or threshold model outputs based on reliability. Understanding how this confidence score is derived requires examining the dual-signal architecture implemented in the source code.

The Two Confidence Signals Explained

Signal 1: Post-Hoc Calibration Head

The post-hoc calibration head is a learned component that evaluates the complete prompt together with the model-generated call. This head produces a calibrated probability in the range [0, 1] that reflects how well the model's output matches the expected distribution.

In needle/model/architecture.py, the calibration head is registered as ConfidenceHead with the forward_confidence method handling the computation (lines 488-576). The head operates after the full sequence has been generated, allowing it to assess contextual coherence rather than incremental token probabilities.

Signal 2: Decoding Probability of Call Tokens

The decoding-probability of the call tokens represents the raw probability assigned by the language model to the sequence of tokens constituting the generated call. This signal captures the model's internal certainty during the autoregressive decoding process.

This probability is derived directly from the output logits during the forward pass, measuring the likelihood the model assigns to its own generated sequence.

How the Final Confidence Score Is Calculated

Both signals are computed independently. The final confidence value returned in API responses is the minimum of the two signals. This conservative design ensures that a call receives a high confidence score only when both the calibrated head and the token-level probability agree on the reliability of the output.

As documented in doc/apis.md (line 151): "the confidence field is the minimum of two signals: a calibrated post‑hoc head … and the decoding probability of the call tokens."

Code Examples

Accessing Confidence from API Responses

import needle

# Initialize the Needle agent

agent = needle.Needle()

# Perform a request

response = agent.complete("What is the capital of France?")

# Access the calibrated confidence field

print("Answer:", response["answer"])
print("Confidence:", response["confidence"])

Inspecting Raw Signals (Advanced)

import needle

agent = needle.Needle()
output = agent._model.forward(tokens)                # raw model logits

prob_tokens = output.decode_probability()           # signal 2

prob_head = agent._model.confidence_head(output)    # signal 1 (post-hoc)

# Combined confidence = min of both signals

combined_confidence = min(prob_head, prob_tokens)
print(combined_confidence)

Direct access to internal signals is intended for debugging and research purposes. The public API already returns the combined confidence value.

Key Implementation Files

File	Purpose
doc/apis.md	Defines the public API contract and confidence field specification
needle/model/architecture.py	Implements ConfidenceHead and forward_confidence method
needle/model/export.py	Registers confidence_head for model export and weight handling

Summary

• Two independent signals compute Needle's confidence: a post-hoc calibration head and token decoding probability.
• Conservative aggregation: the final score uses min(signal_1, signal_2), requiring agreement for high confidence.
• Implementation spans: needle/model/architecture.py (lines 488-576) for the core logic, doc/apis.md for API contract.
• Practical access: retrieve via response["confidence"] in public API or inspect raw signals through internal model methods.

Frequently Asked Questions

Why does Needle use the minimum instead of averaging the two signals?

Averaging would allow one strong signal to mask a weak one. Taking the minimum ensures conservative confidence estimation—a call is only trusted when both the learned calibration head and the raw model probability support it. This prevents overconfident predictions when either signal indicates uncertainty.

Can I disable or modify how the confidence score is computed?

The public API does not expose configuration options for confidence computation. Advanced users can access raw signals through agent._model internals as shown above, but this bypasses stability guarantees. For production use, rely on the standard confidence field and apply your own thresholding logic.

What range do the confidence scores return?

Both underlying signals and the final confidence score are calibrated to the range [0, 1]. The calibration head specifically learns to produce well-calibrated probabilities, meaning a score of 0.90 approximately corresponds to 90% empirical accuracy on held-out evaluation data.

Where is the ConfidenceHead defined in the codebase?

The ConfidenceHead is registered and implemented in needle/model/architecture.py between lines 488-576. The same file contains forward_confidence, which orchestrates the dual-signal computation. Head registration for export purposes appears in needle/model/export.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}