What Is the Precision of Needle 2 Operations? BFloat16 Compute with 2-Bit Quantized Weights

Needle 2 performs arithmetic operations in bfloat16 (16-bit floating-point) precision while compressing model parameters to a 2-bit quantized format (CQ2-bit), creating a hybrid precision system that reduces memory footprint without sacrificing computational accuracy.

The precision architecture of Needle 2 is defined in the cactus-compute/needle repository as a deliberate balance between aggressive quantization for storage and higher-precision floating-point for active computation. This design enables the model to maintain a 14 MB binary size while executing transformer operations with sufficient numerical range for coherent text generation.

Compute Precision: BFloat16 Arithmetic

The active computational precision is hardcoded in needle/model/architecture.py, where the TransformerConfig dataclass establishes the default dtype for all transformer layers. At lines 70-71, the configuration explicitly sets:

dtype: str = "bfloat16"

This string value maps through the DTYPE_MAP dictionary defined at lines 36-38 of the same file, which exposes jnp.bfloat16 as the primary JAX data type for matrix multiplications, attention computations, and activation functions. By using bfloat16 rather than float32, Needle 2 halves its memory bandwidth requirements during inference while preserving the 8-bit exponent range necessary to prevent numerical underflow or overflow in deep transformer stacks.

Storage Precision: CQ2-Bit Weight Quantization

Despite using bfloat16 for calculations, Needle 2 stores its learnable parameters in a drastically different format. According to README.md (lines 5-7), the model compresses weights to CQ2-bit using Cactus Quants—a custom quantization scheme that represents each parameter with only 2 bits. These values persist as packed integers in memory and expand to full bfloat16 tensors only during the forward pass.

The de-quantization logic resides in needle/model/quantize.py, which unpacks the compressed weights on-the-fly as they flow into each layer. This separation allows the model to maintain minimal disk and RAM usage while ensuring that actual mathematical operations occur in a higher-precision format suitable for gradient-free inference.

Inspecting Precision at Runtime

You can verify these precision characteristics programmatically by loading the model and inspecting its internal configuration. The entry point in needle/__init__.py exposes the model configuration and parameter tensors for direct examination.

To view the active compute dtype and quantization settings:

# Load the Needle 2 model (default dtype is bfloat16)

import needle
model = needle.load("cactus-needle")          # pulls the 14 MB binary

# Inspect the model's configuration

cfg = model.config
print("Compute dtype :", cfg.dtype)            # → bfloat16

print("Activation bits:", cfg.act_bits)       # → 8 (quantised activations)

print("Weight bits   :", cfg.weight_bits)     # → '' (CQ2-bit internally)

During inference, the logits are cast to float32 for API compatibility, though internal calculations remain in bfloat16:

# Run inference and view the numeric type of the logits

tokens = model.tokenize("What is the weather today?")
logits = model(tokens)                        # forward pass

print(logits.dtype)                           # → float32 (output cast for readability)

print(logits.shape)                           # typical (batch, seq_len, vocab_size)

To observe the raw storage format of the quantized weights:

# Directly access a weight tensor to see its storage precision

# (weights are stored as 2-bit integers; they are de-quantised to bfloat16 on the fly)

weight = model.params["stack"]["layers"]["self_attn"]["q_proj"]["kernel"]
print(weight.dtype)                           # → uint8 (packed 2-bit values)

Four 2-bit values pack into each uint8 element, explaining the compact 14 MB model size.

Summary

• Needle 2 uses bfloat16 as its compute precision for all transformer operations, configured in needle/model/architecture.py at lines 70-71 via TransformerConfig.dtype.
• Model weights utilize CQ2-bit quantization, storing parameters as 2-bit integers as documented in README.md lines 5-7.
• The DTYPE_MAP at lines 36-38 of the architecture file maps the "bfloat16" string to jnp.bfloat16 for JAX execution.
• Runtime weights appear as uint8 tensors containing packed 2-bit values that de-quantize to bfloat16 during layer execution.
• This hybrid architecture enables efficient inference by minimizing storage size while maintaining computational precision through higher-precision floating-point arithmetic.

Frequently Asked Questions

What is the difference between bfloat16 compute and 2-bit weight storage in Needle 2?

Bfloat16 represents the precision of active mathematical operations such as matrix multiplications and softmax calculations, providing 16 bits of floating-point range and precision. The 2-bit weight storage (CQ2-bit) refers to the compressed format of model parameters on disk, where each weight uses only 2 bits; these decompress to bfloat16 during the forward pass. This distinction allows the model to minimize its 14 MB footprint while executing calculations in a numerically stable format.

Why does Needle 2 use bfloat16 instead of float32 for computations?

Bfloat16 maintains the same 8-bit exponent as float32, preventing the overflow and underflow issues common in float16, while using half the memory bandwidth of full 32-bit operations. This makes it optimal for transformer inference where large matrix operations benefit from reduced data movement without sacrificing the dynamic range needed for attention mechanism stability.

How are the 2-bit weights de-quantized during inference?

The de-quantization occurs in needle/model/quantize.py, where the system unpacks uint8 buffers containing four 2-bit values per byte and scales them to bfloat16 representations on-the-fly. This process happens transparently during layer execution, meaning the self_attn and feed-forward layers receive properly scaled floating-point tensors despite the aggressive weight compression.

Can the compute precision be changed from bfloat16 to float32?

While the TransformerConfig class in architecture.py accepts dtype strings that map through DTYPE_MAP, the CQ2-bit quantized weights are specifically calibrated for bfloat16 scale factors. Changing to float32 would require re-quantizing the weight tensors for the new precision, which is not supported by the default loader in needle/__init__.py and would necessitate regenerating the model binary.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}