# Two Signals Used for Computing Confidence Score in Needle: A Complete Guide

> Discover the two key signals Cactus Compute Needle uses for confidence score calculation: post-hoc calibration and call token decoding probability. Learn how Needle enhances reliability.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: deep-dive
- Published: 2026-08-14

---

**Needle computes confidence scores by taking the minimum of two independent signals: a post-hoc calibration head and the decoding probability of call tokens.**

The Needle framework by Cactus Compute attaches a calibrated confidence value to every response, enabling developers to filter or threshold model outputs based on reliability. Understanding how this confidence score is derived requires examining the dual-signal architecture implemented in the source code.

## The Two Confidence Signals Explained

### Signal 1: Post-Hoc Calibration Head

The **post-hoc calibration head** is a learned component that evaluates the complete prompt together with the model-generated call. This head produces a calibrated probability in the range [0, 1] that reflects how well the model's output matches the expected distribution.

In [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), the calibration head is registered as `ConfidenceHead` with the `forward_confidence` method handling the computation (lines 488-576). The head operates after the full sequence has been generated, allowing it to assess contextual coherence rather than incremental token probabilities.

### Signal 2: Decoding Probability of Call Tokens

The **decoding-probability of the call tokens** represents the raw probability assigned by the language model to the sequence of tokens constituting the generated call. This signal captures the model's internal certainty during the autoregressive decoding process.

This probability is derived directly from the output logits during the forward pass, measuring the likelihood the model assigns to its own generated sequence.

## How the Final Confidence Score Is Calculated

Both signals are computed independently. The final `confidence` value returned in API responses is the **minimum** of the two signals. This conservative design ensures that a call receives a high confidence score only when **both** the calibrated head and the token-level probability agree on the reliability of the output.

As documented in [`doc/apis.md`](https://github.com/cactus-compute/needle/blob/main/doc/apis.md) (line 151): "the confidence field is the minimum of two signals: a calibrated post‑hoc head … and the decoding probability of the call tokens."

## Code Examples

### Accessing Confidence from API Responses

```python
import needle

# Initialize the Needle agent

agent = needle.Needle()

# Perform a request

response = agent.complete("What is the capital of France?")

# Access the calibrated confidence field

print("Answer:", response["answer"])
print("Confidence:", response["confidence"])

```

### Inspecting Raw Signals (Advanced)

```python
import needle

agent = needle.Needle()
output = agent._model.forward(tokens)                # raw model logits

prob_tokens = output.decode_probability()           # signal 2

prob_head = agent._model.confidence_head(output)    # signal 1 (post-hoc)

# Combined confidence = min of both signals

combined_confidence = min(prob_head, prob_tokens)
print(combined_confidence)

```

Direct access to internal signals is intended for debugging and research purposes. The public API already returns the combined confidence value.

## Key Implementation Files

| File | Purpose |
|------|---------|
| [`doc/apis.md`](https://github.com/cactus-compute/needle/blob/main/doc/apis.md) | Defines the public API contract and confidence field specification |
| [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) | Implements `ConfidenceHead` and `forward_confidence` method |
| [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py) | Registers `confidence_head` for model export and weight handling |

## Summary

- **Two independent signals** compute Needle's confidence: a post-hoc calibration head and token decoding probability.
- **Conservative aggregation**: the final score uses `min(signal_1, signal_2)`, requiring agreement for high confidence.
- **Implementation spans**: [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) (lines 488-576) for the core logic, [`doc/apis.md`](https://github.com/cactus-compute/needle/blob/main/doc/apis.md) for API contract.
- **Practical access**: retrieve via `response["confidence"]` in public API or inspect raw signals through internal model methods.

## Frequently Asked Questions

### Why does Needle use the minimum instead of averaging the two signals?

Averaging would allow one strong signal to mask a weak one. Taking the minimum ensures conservative confidence estimation—a call is only trusted when both the learned calibration head and the raw model probability support it. This prevents overconfident predictions when either signal indicates uncertainty.

### Can I disable or modify how the confidence score is computed?

The public API does not expose configuration options for confidence computation. Advanced users can access raw signals through `agent._model` internals as shown above, but this bypasses stability guarantees. For production use, rely on the standard `confidence` field and apply your own thresholding logic.

### What range do the confidence scores return?

Both underlying signals and the final confidence score are calibrated to the range [0, 1]. The calibration head specifically learns to produce well-calibrated probabilities, meaning a score of 0.90 approximately corresponds to 90% empirical accuracy on held-out evaluation data.

### Where is the ConfidenceHead defined in the codebase?

The `ConfidenceHead` is registered and implemented in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) between lines 488-576. The same file contains `forward_confidence`, which orchestrates the dual-signal computation. Head registration for export purposes appears in [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py).