# What Is the Precision of Needle 2 Operations? BFloat16 Compute with 2-Bit Quantized Weights

> Explore Needle 2's bfloat16 compute and 2-bit quantized weights. Discover how this hybrid precision system minimizes memory while maintaining accuracy for your models.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: performance
- Published: 2026-09-11

---

**Needle 2 performs arithmetic operations in bfloat16 (16-bit floating-point) precision while compressing model parameters to a 2-bit quantized format (CQ2-bit), creating a hybrid precision system that reduces memory footprint without sacrificing computational accuracy.**

The precision architecture of Needle 2 is defined in the `cactus-compute/needle` repository as a deliberate balance between aggressive quantization for storage and higher-precision floating-point for active computation. This design enables the model to maintain a 14 MB binary size while executing transformer operations with sufficient numerical range for coherent text generation.

## Compute Precision: BFloat16 Arithmetic

The active computational precision is hardcoded in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), where the `TransformerConfig` dataclass establishes the default dtype for all transformer layers. At lines 70-71, the configuration explicitly sets:

```python
dtype: str = "bfloat16"

```

This string value maps through the `DTYPE_MAP` dictionary defined at lines 36-38 of the same file, which exposes `jnp.bfloat16` as the primary JAX data type for matrix multiplications, attention computations, and activation functions. By using bfloat16 rather than float32, Needle 2 halves its memory bandwidth requirements during inference while preserving the 8-bit exponent range necessary to prevent numerical underflow or overflow in deep transformer stacks.

## Storage Precision: CQ2-Bit Weight Quantization

Despite using bfloat16 for calculations, Needle 2 stores its learnable parameters in a drastically different format. According to [`README.md`](https://github.com/cactus-compute/needle/blob/main/README.md) (lines 5-7), the model compresses weights to **CQ2-bit** using Cactus Quants—a custom quantization scheme that represents each parameter with only 2 bits. These values persist as packed integers in memory and expand to full bfloat16 tensors only during the forward pass.

The de-quantization logic resides in [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py), which unpacks the compressed weights on-the-fly as they flow into each layer. This separation allows the model to maintain minimal disk and RAM usage while ensuring that actual mathematical operations occur in a higher-precision format suitable for gradient-free inference.

## Inspecting Precision at Runtime

You can verify these precision characteristics programmatically by loading the model and inspecting its internal configuration. The entry point in [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) exposes the model configuration and parameter tensors for direct examination.

To view the active compute dtype and quantization settings:

```python

# Load the Needle 2 model (default dtype is bfloat16)

import needle
model = needle.load("cactus-needle")          # pulls the 14 MB binary

# Inspect the model's configuration

cfg = model.config
print("Compute dtype :", cfg.dtype)            # → bfloat16

print("Activation bits:", cfg.act_bits)       # → 8 (quantised activations)

print("Weight bits   :", cfg.weight_bits)     # → '' (CQ2-bit internally)

```

During inference, the logits are cast to float32 for API compatibility, though internal calculations remain in bfloat16:

```python

# Run inference and view the numeric type of the logits

tokens = model.tokenize("What is the weather today?")
logits = model(tokens)                        # forward pass

print(logits.dtype)                           # → float32 (output cast for readability)

print(logits.shape)                           # typical (batch, seq_len, vocab_size)

```

To observe the raw storage format of the quantized weights:

```python

# Directly access a weight tensor to see its storage precision

# (weights are stored as 2-bit integers; they are de-quantised to bfloat16 on the fly)

weight = model.params["stack"]["layers"]["self_attn"]["q_proj"]["kernel"]
print(weight.dtype)                           # → uint8 (packed 2-bit values)

```

Four 2-bit values pack into each uint8 element, explaining the compact 14 MB model size.

## Summary

- **Needle 2 uses bfloat16** as its compute precision for all transformer operations, configured in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) at lines 70-71 via `TransformerConfig.dtype`.
- **Model weights utilize CQ2-bit quantization**, storing parameters as 2-bit integers as documented in [`README.md`](https://github.com/cactus-compute/needle/blob/main/README.md) lines 5-7.
- **The DTYPE_MAP** at lines 36-38 of the architecture file maps the `"bfloat16"` string to `jnp.bfloat16` for JAX execution.
- **Runtime weights** appear as `uint8` tensors containing packed 2-bit values that de-quantize to bfloat16 during layer execution.
- **This hybrid architecture** enables efficient inference by minimizing storage size while maintaining computational precision through higher-precision floating-point arithmetic.

## Frequently Asked Questions

### What is the difference between bfloat16 compute and 2-bit weight storage in Needle 2?

Bfloat16 represents the precision of active mathematical operations such as matrix multiplications and softmax calculations, providing 16 bits of floating-point range and precision. The 2-bit weight storage (CQ2-bit) refers to the compressed format of model parameters on disk, where each weight uses only 2 bits; these decompress to bfloat16 during the forward pass. This distinction allows the model to minimize its 14 MB footprint while executing calculations in a numerically stable format.

### Why does Needle 2 use bfloat16 instead of float32 for computations?

Bfloat16 maintains the same 8-bit exponent as float32, preventing the overflow and underflow issues common in float16, while using half the memory bandwidth of full 32-bit operations. This makes it optimal for transformer inference where large matrix operations benefit from reduced data movement without sacrificing the dynamic range needed for attention mechanism stability.

### How are the 2-bit weights de-quantized during inference?

The de-quantization occurs in [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py), where the system unpacks uint8 buffers containing four 2-bit values per byte and scales them to bfloat16 representations on-the-fly. This process happens transparently during layer execution, meaning the `self_attn` and feed-forward layers receive properly scaled floating-point tensors despite the aggressive weight compression.

### Can the compute precision be changed from bfloat16 to float32?

While the `TransformerConfig` class in [`architecture.py`](https://github.com/cactus-compute/needle/blob/main/architecture.py) accepts dtype strings that map through `DTYPE_MAP`, the CQ2-bit quantized weights are specifically calibrated for bfloat16 scale factors. Changing to float32 would require re-quantizing the weight tensors for the new precision, which is not supported by the default loader in [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) and would necessitate regenerating the model binary.