# Needle 2 Performance Metrics: Decoding the Response Envelope

> Explore Needle 2 performance metrics including prefill and decode TPS, peak RAM, and confidence. Understand your model's response envelope with detailed runtime insights.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: performance
- Published: 2026-08-29

---

**Needle 2 returns four runtime performance metrics in its JSON response envelope: `prefill_tps` and `decode_tps` for token throughput, `peak_ram_mb` for memory usage, and `confidence` for model calibration.**

The cactus-compute/needle library provides a lightweight agent framework that exposes detailed runtime telemetry after every inference call. When you invoke a Needle 2 agent, the response envelope includes specific performance metrics that enable precise monitoring of inference speed and memory consumption directly from the model output.

## Understanding the Four Performance Metrics

The Needle 2 engine measures distinct phases of the inference lifecycle, providing granular visibility into computational performance.

### Token Throughput Metrics (`prefill_tps` and `decode_tps`)

The response envelope separates token generation performance into two distinct phases:

- **`prefill_tps`**: Tokens-per-second for the *prefill* phase, which encompasses embedding the prompt and tool schemas into the model's context window.
- **`decode_tps`**: Tokens-per-second for the *decode* phase, representing the actual generation of the model's output tokens.

These metrics appear as floating-point values in the JSON response, allowing you to distinguish between prompt processing overhead and autoregressive generation speed.

### Memory Utilization (`peak_ram_mb`)

The **`peak_ram_mb`** field reports the peak RAM usage of the engine during the turn, expressed in megabytes. This metric captures the maximum memory footprint reached during inference, enabling developers to track resource consumption across different prompt lengths and tool configurations.

### Model Confidence Scoring (`confidence`)

The **`confidence`** field provides a calibrated confidence score representing the model's certainty in its response generation. For fine-tuned weights, this value returns `null` rather than a numeric score, indicating that calibration metrics are unavailable for custom model weights.

## Response Envelope Structure

According to the API documentation in [`doc/apis.md`](https://github.com/cactus-compute/needle/blob/main/doc/apis.md) (lines 84-96), the performance metrics reside alongside standard response fields in a unified JSON envelope:

```json
{
  "type": "call",
  "success": true,
  "error": null,
  "error_code": null,
  "function_calls": [],
  "reasoning": "…",
  "confidence": 0.94,
  "prefill_tps": 4300.0,
  "decode_tps": 850.0,
  "peak_ram_mb": 28.5
}

```

This structure ensures that performance telemetry travels with the functional response, eliminating the need for separate monitoring hooks or external profiling tools.

## Accessing Metrics in Python

The core `Needle` class in [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) parses the native engine's JSON response and returns it as a Python dictionary. You can access performance metrics directly from the result object:

```python
import needle

# Create an agent with a simple tool

@needle.tool
def echo(message: str):
    """Return the same message."""
    return {"message": message}

agent = needle.Needle(tools=[echo])

# Run a query – the final envelope contains performance metrics

result = agent.run("echo hello world")
print(result["confidence"])      # 0.94 (example)

print(result["prefill_tps"])     # 4300.0 tokens/sec

print(result["decode_tps"])      # 850.0 tokens/sec

print(result["peak_ram_mb"])     # 28.5 MB

```

For low-level completions, the `complete` method returns the identical envelope format:

```python

# Using the low-level `complete` call

response = agent.complete("echo test")
print(response["prefill_tps"])   # Access throughput metrics directly

```

## Implementation Details

The performance metrics flow through several key components in the cactus-compute/needle repository:

- **[`doc/apis.md`](https://github.com/cactus-compute/needle/blob/main/doc/apis.md)**: Defines the envelope contract and documents the four performance fields (`confidence`, `prefill_tps`, `decode_tps`, `peak_ram_mb`).
- **[`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py)**: Implements the core `Needle` class that invokes the native engine and parses the JSON envelope into Python dictionaries.
- **[`tests/test_weights.py`](https://github.com/cactus-compute/needle/blob/main/tests/test_weights.py)**: Validates the envelope structure, ensuring that `type`, `confidence`, and other mandatory fields are present in responses.
- **[`needle/_telemetry.py`](https://github.com/cactus-compute/needle/blob/main/needle/_telemetry.py)**: Handles additional telemetry reporting infrastructure, though the envelope metrics are provided directly by the native engine response rather than through this module.

These metrics allow developers to monitor inference speed and memory usage directly from the model's response, enabling better performance tuning and observability without external profiling overhead.

## Summary

- **`prefill_tps`**: Measures prompt embedding and tool schema processing speed in tokens-per-second.
- **`decode_tps`**: Tracks autoregressive output generation speed in tokens-per-second.
- **`peak_ram_mb`**: Reports maximum memory consumption during the inference turn.
- **`confidence`**: Provides calibrated reliability scores (or `null` for fine-tuned weights).
- All metrics are accessible via the standard response dictionary returned by `agent.run()` or `agent.complete()`.

## Frequently Asked Questions

### What is the difference between `prefill_tps` and `decode_tps`?

The `prefill_tps` metric measures the throughput during the prefill phase, where the system embeds your prompt and tool schemas into the model's context. The `decode_tps` metric measures throughput during the decode phase, where the model generates new tokens autoregressively. Typically, prefill rates are significantly higher than decode rates because the prefill phase processes the entire prompt in parallel.

### Why is the `confidence` field sometimes `null`?

The `confidence` field returns `null` when using fine-tuned weights because calibrated confidence scores are only available for base model weights provided by the Needle engine. Fine-tuned models lack the calibration data required to generate reliable confidence intervals, so the field is intentionally nulled to prevent misleading interpretations.

### How can I monitor memory usage across multiple agent calls?

Track the `peak_ram_mb` value from each response envelope across your conversation turns. For persistent monitoring, you can aggregate these values to identify memory growth patterns or set thresholds for specific tool configurations. The [`needle/_telemetry.py`](https://github.com/cactus-compute/needle/blob/main/needle/_telemetry.py) module provides additional infrastructure for advanced telemetry aggregation if you need historical tracking beyond single-turn metrics.

### Where are these metrics calculated in the source code?

The metrics are calculated by the native engine and returned in the JSON response, as documented in [`doc/apis.md`](https://github.com/cactus-compute/needle/blob/main/doc/apis.md) (lines 84-96). The Python layer in [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) receives this JSON and passes it through to your application code. The envelope structure is validated in [`tests/test_weights.py`](https://github.com/cactus-compute/needle/blob/main/tests/test_weights.py), ensuring that all performance fields are correctly typed and included in production responses.