Needle 2 Performance Metrics: Decoding the Response Envelope

Needle 2 returns four runtime performance metrics in its JSON response envelope: prefill_tps and decode_tps for token throughput, peak_ram_mb for memory usage, and confidence for model calibration.

The cactus-compute/needle library provides a lightweight agent framework that exposes detailed runtime telemetry after every inference call. When you invoke a Needle 2 agent, the response envelope includes specific performance metrics that enable precise monitoring of inference speed and memory consumption directly from the model output.

Understanding the Four Performance Metrics

The Needle 2 engine measures distinct phases of the inference lifecycle, providing granular visibility into computational performance.

Token Throughput Metrics (prefill_tps and decode_tps)

The response envelope separates token generation performance into two distinct phases:

• prefill_tps: Tokens-per-second for the prefill phase, which encompasses embedding the prompt and tool schemas into the model's context window.
• decode_tps: Tokens-per-second for the decode phase, representing the actual generation of the model's output tokens.

These metrics appear as floating-point values in the JSON response, allowing you to distinguish between prompt processing overhead and autoregressive generation speed.

Memory Utilization (peak_ram_mb)

The peak_ram_mb field reports the peak RAM usage of the engine during the turn, expressed in megabytes. This metric captures the maximum memory footprint reached during inference, enabling developers to track resource consumption across different prompt lengths and tool configurations.

Model Confidence Scoring (confidence)

The confidence field provides a calibrated confidence score representing the model's certainty in its response generation. For fine-tuned weights, this value returns null rather than a numeric score, indicating that calibration metrics are unavailable for custom model weights.

Response Envelope Structure

According to the API documentation in doc/apis.md (lines 84-96), the performance metrics reside alongside standard response fields in a unified JSON envelope:

{
  "type": "call",
  "success": true,
  "error": null,
  "error_code": null,
  "function_calls": [],
  "reasoning": "…",
  "confidence": 0.94,
  "prefill_tps": 4300.0,
  "decode_tps": 850.0,
  "peak_ram_mb": 28.5
}

This structure ensures that performance telemetry travels with the functional response, eliminating the need for separate monitoring hooks or external profiling tools.

Accessing Metrics in Python

The core Needle class in needle/__init__.py parses the native engine's JSON response and returns it as a Python dictionary. You can access performance metrics directly from the result object:

import needle

# Create an agent with a simple tool

@needle.tool
def echo(message: str):
    """Return the same message."""
    return {"message": message}

agent = needle.Needle(tools=[echo])

# Run a query – the final envelope contains performance metrics

result = agent.run("echo hello world")
print(result["confidence"])      # 0.94 (example)

print(result["prefill_tps"])     # 4300.0 tokens/sec

print(result["decode_tps"])      # 850.0 tokens/sec

print(result["peak_ram_mb"])     # 28.5 MB

For low-level completions, the complete method returns the identical envelope format:

# Using the low-level `complete` call

response = agent.complete("echo test")
print(response["prefill_tps"])   # Access throughput metrics directly

Implementation Details

The performance metrics flow through several key components in the cactus-compute/needle repository:

• doc/apis.md: Defines the envelope contract and documents the four performance fields (confidence, prefill_tps, decode_tps, peak_ram_mb).
• needle/__init__.py: Implements the core Needle class that invokes the native engine and parses the JSON envelope into Python dictionaries.
• tests/test_weights.py: Validates the envelope structure, ensuring that type, confidence, and other mandatory fields are present in responses.
• needle/_telemetry.py: Handles additional telemetry reporting infrastructure, though the envelope metrics are provided directly by the native engine response rather than through this module.

These metrics allow developers to monitor inference speed and memory usage directly from the model's response, enabling better performance tuning and observability without external profiling overhead.

Summary

• prefill_tps: Measures prompt embedding and tool schema processing speed in tokens-per-second.
• decode_tps: Tracks autoregressive output generation speed in tokens-per-second.
• peak_ram_mb: Reports maximum memory consumption during the inference turn.
• confidence: Provides calibrated reliability scores (or null for fine-tuned weights).
• All metrics are accessible via the standard response dictionary returned by agent.run() or agent.complete().

Frequently Asked Questions

What is the difference between prefill_tps and decode_tps?

The prefill_tps metric measures the throughput during the prefill phase, where the system embeds your prompt and tool schemas into the model's context. The decode_tps metric measures throughput during the decode phase, where the model generates new tokens autoregressively. Typically, prefill rates are significantly higher than decode rates because the prefill phase processes the entire prompt in parallel.

Why is the confidence field sometimes null?

The confidence field returns null when using fine-tuned weights because calibrated confidence scores are only available for base model weights provided by the Needle engine. Fine-tuned models lack the calibration data required to generate reliable confidence intervals, so the field is intentionally nulled to prevent misleading interpretations.

How can I monitor memory usage across multiple agent calls?

Track the peak_ram_mb value from each response envelope across your conversation turns. For persistent monitoring, you can aggregate these values to identify memory growth patterns or set thresholds for specific tool configurations. The needle/_telemetry.py module provides additional infrastructure for advanced telemetry aggregation if you need historical tracking beyond single-turn metrics.

Where are these metrics calculated in the source code?

The metrics are calculated by the native engine and returned in the JSON response, as documented in doc/apis.md (lines 84-96). The Python layer in needle/__init__.py receives this JSON and passes it through to your application code. The envelope structure is validated in tests/test_weights.py, ensuring that all performance fields are correctly typed and included in production responses.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}