# How to Handle Confidence Scores for Tool Calls in Needle 2

> Learn how to handle confidence scores for tool calls in Needle 2. Gate your model's execution based on response certainty for reliable tool usage.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: how-to-guide
- Published: 2026-09-08

---

**Needle 2 generates a numeric confidence value for every model response containing potential tool calls, allowing you to gate execution based on the model's certainty.**

The **cactus-compute/needle** repository provides a dedicated confidence scoring mechanism that attaches a reliability metric to function call proposals. Understanding how to extract and apply these scores is essential for building robust agentic workflows that avoid executing low-certainty actions.

## Understanding Confidence Score Generation in Needle 2

Every tool call candidate processed by the Needle 2 engine receives a confidence score derived from the model's internal representations. This score indicates the model's certainty that a particular tool invocation is appropriate for the given context.

### The ConfidenceHead Module

The scoring logic resides in the `ConfidenceHead` class located in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) at lines 63–76. This module receives the hidden cell representations from the transformer layers and outputs a single floating-point value per request.

According to the source code, the `forward_confidence` method processes the final hidden states and compresses them into a scalar value through a learned projection layer. The resulting float represents the model's estimated probability that the proposed tool call is correct, typically constrained to the range **[0, 1]**.

## Accessing Confidence Scores in API Responses

When you invoke the high-level Python API, the confidence value travels through the response envelope and surfaces in the returned dictionary.

### Parsing the Engine Envelope

In [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) at lines 150–162, the `Needle` class constructor initializes the wrapper and sets up response parsing. When the underlying engine returns a completion, the wrapper extracts the raw JSON envelope and maps the `"confidence"` field directly into the response dictionary:

```python
import needle

agent = needle.Needle(
    tools=[my_calendar_tool],
    system="You are a helpful assistant."
)

result = agent.complete("Schedule a meeting tomorrow at 10am")
confidence = result["confidence"]  # Float between 0.0 and 1.0, or None

```

###Handling Fine-Tuned Models

If you load fine-tuned weights, the confidence head is typically not trained and therefore unreliable. The wrapper detects this condition and forces the confidence field to `None` to prevent accidental reliance on untrained metrics. When `result["confidence"]` returns `None`, you should either skip confidence gating or implement a fallback strategy.

## Implementing Confidence Thresholds for Tool Calls

Production systems should never execute tool calls blindly. Instead, compare the confidence score against a business-defined threshold before dispatching functions.

### Gating Tool Execution with min_confidence

The built-in test harness in [`needle/environments/_harness.py`](https://github.com/cactus-compute/needle/blob/main/needle/environments/_harness.py) (lines 17–30) demonstrates the canonical pattern for confidence gating. The harness accepts a `min_confidence` parameter and filters the `"function_calls"` list based on the attached score:

```python
from needle.environments.smart_home import run_tests

# Only accept tool calls with confidence >= 0.6

success = run_tests(min_confidence=0.6)

```

If the confidence value falls below the threshold, the harness clears the function calls list, effectively converting the request into a refusal. You can replicate this pattern in your own applications:

```python
THRESHOLD = 0.8

def execute_with_gate(agent, query):
    response = agent.complete(query)
    conf = response.get("confidence")
    
    if conf is None:
        # Handle fine-tuned models: skip gating or use default

        print("Warning: Confidence unavailable for fine-tuned model")
        return execute_tools(response.get("function_calls", []))
    
    if conf >= THRESHOLD:
        return execute_tools(response["function_calls"])
    else:
        print(f"Confidence {conf:.2f} below threshold {THRESHOLD}. Refusing.")
        return {"status": "refused", "reason": "low_confidence"}

```

## Complete Implementation Example

The following workflow demonstrates initializing an agent, inspecting confidence scores, and conditionally executing tools based on a configurable threshold:

```python
import needle

# Initialize with base weights (confidence available)

agent = needle.Needle(
    tools=[search_database, send_email],
    system="You are an administrative assistant."
)

# Generate a response that may contain tool calls

result = agent.complete("Find the Q3 revenue report and email it to the team")

# Extract confidence

conf = result.get("confidence")
print(f"Model confidence: {conf}")  # e.g., 0.87 or None

# Define business logic threshold

MIN_CONFIDENCE = 0.75

if conf is not None and conf >= MIN_CONFIDENCE:
    for call in result.get("function_calls", []):
        tool_name = call["name"]
        arguments = call["arguments"]
        
        # Dispatch to registered function

        if tool_name in agent._functions:
            agent._functions[tool_name](**arguments)
else:
    print("Insufficient confidence to execute tools. Providing text response only.")

```

## Summary

- **ConfidenceHead** in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) generates scores from hidden representations at lines 63–76.
- The Python wrapper in [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) surfaces these scores in the response envelope, but forces `None` for fine-tuned models (lines 150–162).
- **Threshold gating** prevents execution of low-confidence tool calls, as implemented in [`needle/environments/_harness.py`](https://github.com/cactus-compute/needle/blob/main/needle/environments/_harness.py) lines 17–30.
- Fine-tuned models require alternative validation strategies since their confidence heads return `None`.
- Scores typically range from 0.0 to 1.0, where higher values indicate greater certainty.

## Frequently Asked Questions

### What file contains the ConfidenceHead implementation?

The `ConfidenceHead` module is defined in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) at lines 63–76. This class implements the `forward_confidence` method that projects hidden states into a scalar confidence value.

### Why is confidence None for fine-tuned models?

When you load fine-tuned weights via the `Needle` class, the wrapper detects that the confidence head has not been trained and forces the field to `None` to avoid misleading scores. This behavior is handled in [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) between lines 150–162.

### What is the range of confidence scores in Needle 2?

Confidence scores are single-precision floating-point values typically bounded between **0.0 and 1.0**, where 1.0 indicates maximum certainty. However, always check for `None` when working with fine-tuned or specialized model weights.

### How do I filter tool calls based on confidence?

Pass a `min_confidence` parameter to your execution logic or use the pattern from [`needle/environments/_harness.py`](https://github.com/cactus-compute/needle/blob/main/needle/environments/_harness.py) lines 17–30. Compare `response["confidence"]` against your threshold, and only execute functions in `response["function_calls"]` when the score meets or exceeds that value.