# How to Adjust the Confidence Threshold for Accepting Calls in Needle 2

> Learn how to adjust the confidence threshold in Needle 2 by modifying the min_confidence parameter in run_tests(). Filter model-generated calls based on confidence scores effectively.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: how-to-guide
- Published: 2026-09-02

---

**The confidence threshold in Needle 2 is controlled by the `min_confidence` parameter passed to `run_tests()`, which filters out any model-generated call whose confidence score falls below the specified value.**

Needle 2 uses a confidence-based gating mechanism to decide whether to accept or reject calls generated by the underlying language model. According to the cactus-compute/needle source code, this gate is implemented in the test harness and can be adjusted programmatically or via CLI flag.

## Where the Confidence Threshold Is Enforced

The core logic resides in [`needle/environments/_harness.py`](https://github.com/cactus-compute/needle/blob/main/needle/environments/_harness.py). The `run_tests()` function accepts a `min_confidence` parameter and applies it to every model response:

```python

# From needle/environments/_harness.py

def run_tests(module, min_confidence=0.0, verbose=True):
    """
    Run tests for a given environment module.
    
    Parameters:
        module: The environment module containing test cases
        min_confidence: Minimum confidence threshold (0.0-1.0) for accepting calls
        verbose: Whether to print detailed output
    """
    for test_case in module.get_test_cases():
        response = model.generate(test_case.prompt)
        
        # Confidence gate: reject calls below threshold

        confidence = response.get("confidence", 0.0)
        if confidence < min_confidence:
            # Call is rejected/treated as failure

            handle_rejection(test_case, response, confidence)
            continue
            
        # Call accepted: proceed with execution

        execute_call(response)

```

The harness defaults to `min_confidence=0.0`, meaning all calls are accepted unless you explicitly raise the threshold.

## How Confidence Scores Are Generated

The confidence value originates from the model architecture itself. In [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), the `ConfidenceHead` module produces scalar confidence scores during inference:

```python

# From needle/model/architecture.py

class ConfidenceHead(nn.Module):
    """
    Predicts confidence score for model-generated outputs.
    """
    def forward(self, hidden_states):
        # Projects to single confidence value

        return torch.sigmoid(self.projection(hidden_states[:, -1, :]))
    
class NeedleModel(nn.Module):
    # ...

    
    def forward_confidence(self, input_ids, attention_mask=None):
        """Generate confidence score for a given input."""
        hidden = self.encoder(input_ids, attention_mask=attention_mask)
        return self.confidence_head(hidden.last_hidden_state)

```

When the model generates a response, `forward_confidence()` attaches the score to the response dictionary as `response["confidence"]`. If the model lacks a trained confidence head, this field will be `None`; the harness treats `None` as `0.0`, causing any positive threshold to reject those responses automatically.

## Method 1: Programmatic Adjustment via Environment Module

For environment-specific testing, import the environment's `run_tests` function and specify your threshold:

```python
from needle.environments.smart_home import run_tests

# Require 60% confidence for smart home calls

results = run_tests(min_confidence=0.6, verbose=True)

# Results will show which calls passed/failed the confidence gate

print(f"Passed: {results.passed}, Failed: {results.failed}")

```

This approach targets a single environment. The environment module's `run_tests` is typically a thin wrapper that calls the harness with environment-specific test cases.

## Method 2: Global Adjustment Across All Environments

To apply a consistent threshold across multiple environments, use the generic entry point:

```python
from needle.environments import run_tests as run_all_tests

# Apply 45% confidence floor to every environment

all_results = run_all_tests(min_confidence=0.45)

for env_name, result in all_results.items():
    print(f"{env_name}: {result.pass_rate:.1%} pass rate")

```

The generic `run_tests` iterates through registered environments and invokes each with the shared `min_confidence` value.

## Method 3: Command-Line Interface

The CLI in [`needle/cli.py`](https://github.com/cactus-compute/needle/blob/main/needle/cli.py) exposes `--min-confidence` as a direct flag:

```bash

# Run with 70% confidence threshold

needle run --min-confidence 0.7

# Combine with other flags

needle run --min-confidence 0.5 --verbose --output results.json

# Per-environment override

needle run smart_home --min-confidence 0.8

```

The CLI parser forwards the flag value directly to the underlying `run_tests()` call, making it equivalent to the programmatic approaches.

## Choosing an Appropriate Threshold

The optimal `min_confidence` value depends on your risk tolerance and model calibration:

| Threshold | Use Case | Trade-off |
|-----------|----------|-----------|
| 0.0 | Development/debugging | Maximum recall, accepts all calls |
| 0.3-0.5 | Balanced production | Moderate precision/recall balance |
| 0.7-0.8 | High-stakes applications | High precision, may reject valid calls |
| 0.9+ | Critical safety systems | Near-zero false positives, significant false negatives |

Monitor your model's calibration curve to avoid setting thresholds in poorly-calibrated regions. The confidence head's output is passed through a `sigmoid`, so values naturally range 0-1.

## Summary

- The confidence threshold is set via **`min_confidence`** in `run_tests()` from [`needle/environments/_harness.py`](https://github.com/cactus-compute/needle/blob/main/needle/environments/_harness.py)
- Scores originate from **`ConfidenceHead.forward()`** in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py)
- Adjust via **environment module**, **generic entry point**, or **`--min-confidence` CLI flag**
- **`None` confidence values default to 0.0**, so untrained models fail any positive threshold
- Threshold selection should balance **precision requirements** against **call acceptance rates**

## Frequently Asked Questions

### What happens if I set min_confidence above 1.0 or below 0.0?

Values outside [0.0, 1.0] are technically accepted by the function signature but behave predictably: thresholds above 1.0 reject all calls (since sigmoid outputs max at ~1.0), while negative thresholds accept all calls. The CLI validates range in [`needle/cli.py`](https://github.com/cactus-compute/needle/blob/main/needle/cli.py), raising `argparse.ArgumentTypeError` for out-of-bounds values.

### Can I disable the confidence gate entirely?

Pass `min_confidence=0.0` (the default) or any negative value. This accepts all calls regardless of confidence score, effectively disabling the feature without code modification.

### Where can I calibrate or retrain the confidence head?

The `ConfidenceHead` is defined in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py). Fine-tuning requires labeled confidence data; the training loop in [`needle/training/trainer.py`](https://github.com/cactus-compute/needle/blob/main/needle/training/trainer.py) supports auxiliary confidence loss via the `--confidence-weight` hyperparameter. Retrain with your domain-specific data to improve calibration before adjusting thresholds.

### Does the confidence threshold affect test reporting or just execution?

The threshold affects both. In [`needle/environments/_harness.py`](https://github.com/cactus-compute/needle/blob/main/needle/environments/_harness.py), failing the confidence check marks the test case as failed (affecting pass rates) and skips actual call execution. The test output includes both the confidence value and explicit "confidence_rejected" status for transparency.