# 2‑Bit vs 4‑Bit Quantization Tradeoffs for `.cact` Files: A Complete Technical Guide

> Explore 2-bit vs 4-bit quantization tradeoffs for .cact files in Needle. Understand the balance between model size, distortion, and accuracy for optimal production deployments.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: deep-dive
- Published: 2026-08-16

---

**2‑bit quantization cuts model size in half compared to 4‑bit but increases distortion 2–3×, while 4‑bit offers the best balance of compression and accuracy for most production deployments.** This article examines the engineering tradeoffs between these bit‑widths in Needle's compact binary archive format.

Needle, an open‑source inference engine from [cactus‑compute/needle](https://github.com/cactus-compute/needle), stores quantized model weights in `.cact` files. The **quantization bit‑width** you choose directly impacts archive size, numerical fidelity, and runtime performance. This guide breaks down the implementation details found in the source code to help you make an informed decision.

## How Quantization Bit‑Width Determines Archive Size

The `.cact` format stores **CQ (compressed quantization) tensors** as packed index streams. Archive size scales linearly with bits per weight.

In [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py), the function `cq_model_bytes()` computes total bytes at lines 176–189:

```python

# From quantize.py L176-L189

# Size formula: acc["q"] * bits / 8

```

Key implications:

- **2‑bit (CQ 2)**: Each weight occupies 0.25 bytes. Roughly **half the size** of a 4‑bit model.
- **4‑bit (CQ 4)**: Each weight occupies 0.5 bytes. The **default** for most deployments.

The packed index stream format is defined in [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py) lines 51–58, where the `bits` field in the tensor directory records whether each tensor uses 2 or 4 bits.

## Quantization Distortion and Accuracy Impact

Numerical fidelity degrades predictably with lower bit‑widths.

The `cq_distortion()` function (quantize.py L225–229) measures **relative mean‑squared error** for a given bit‑width. Empirical results show:

| Bit‑Width | Approximate Distortion |
|-----------|------------------------|
| 4‑bit | Baseline (1×) |
| 2‑bit | 2–3× higher than 4‑bit |

This distortion manifests as increased quantization noise during inference. For tasks requiring high precision—mathematical reasoning, code generation, or long‑context coherence—4‑bit quantization preserves more of the original model's capability.

## Runtime Performance Characteristics

### Speed Considerations

The engine's hot‑path uses a **Walsh‑Hadamard rotation** followed by **codebook lookup**. Per‑group work remains identical regardless of bit‑width, so speed differences are modest:

- **CPU**: ~5–10% faster for 2‑bit due to tighter packing
- **GPU**: Negligible difference (memory bandwidth, not compute, is the bottleneck)

The reconstruction pipeline in [`export.py`](https://github.com/cactus-compute/needle/blob/main/export.py) lines 60–66 handles both bit‑widths uniformly: expand packed indices, multiply by per‑group L2 norm, apply Hadamard rotation.

### Memory Bandwidth

2‑bit quantization **halves memory bandwidth** requirements when streaming weights from RAM or SSD. This benefit matters most on:

- Low‑end mobile CPUs
- Edge devices with slow external storage
- Batch‑1 inference where weight loading dominates latency

## Training Implications: Quantization‑Aware Noise

When using quantization‑aware training (QAT), the `noise_scale()` function (quantize.py L231–239) adjusts noise injection based on effective bits:

```python
b = min(max(float(bits), MIN_BITS), MAX_BITS)

```

Lower bit‑widths yield **larger noise scales**, making QAT convergence more difficult. If you plan to fine‑tune with Needle's quantization‑aware helpers, 4‑bit provides a more stable training surface.

## Configuring Bit‑Width in Practice

### Command‑Line Export

Pass `--bits` to `needle build` (quantization configuration in `configure_deploy`, quantize.py L61–66):

```python
import subprocess

# 2-bit model for extreme compression

subprocess.run([
    "needle", "build",
    "model.ckpt",
    "--out", "model_2bit.cact",
    "--bits", "2"
])

# 4-bit default for balanced deployment

subprocess.run([
    "needle", "build",
    "model.ckpt",
    "--out", "model_4bit.cact",
    "--bits", "4"
])

```

### Mixed‑Precision Export

Apply different bit‑widths to different layers using `--bits-map`:

```python

# Hardware layers 2-bit, rest 4-bit

subprocess.run([
    "needle", "build",
    "model.ckpt",
    "--out", "model_mixed.cact",
    "--bits-map", "layer00=2,layer01=2,default=4"
])

```

The parser validates this map in `parse_bits_map()` (quantize.py L301–318).

### Runtime KV Cache Configuration

Control activation and KV cache precision separately via `configure_deploy`:

```python
from needle import Needle, configure_deploy

# 8-bit activations, 2-bit KV cache (saves memory on long contexts)

configure_deploy(act_bits=8, kv_bits=2, kv_group=64)

agent = Needle(weights="model_2bit.cact")

```

## Codebook and Packing Internals

The bit‑width determines codebook structure built in `_cq_codebook_np` (quantize.py L112–115):

- **2‑bit**: 4‑level Lloyd‑Max codebook
- **4‑bit**: 16‑level Lloyd‑Max codebook

Despite different codebook sizes, the runtime decoding path is identical—no functional penalty, only precision difference.

## When to Choose Each Bit‑Width

| Scenario | Recommended Bit‑Width | Rationale |
|----------|----------------------|-----------|
| Edge devices, cheap CPUs, 30–40% size reduction critical | **2‑bit** | Half the bandwidth, acceptable accuracy loss |
| Production services, general deployment | **4‑bit** | Best fidelity/compression balance |
| QAT fine‑tuning | **4‑bit** | Stable convergence, lower noise scale |
| Mixed layer importance | **Mixed** | Critical layers 4‑bit, others 2‑bit |

## Summary

- **Model size**: 2‑bit produces ~50% smaller `.cact` files than 4‑bit, computed via `cq_model_bytes()` in quantize.py.
- **Accuracy**: 4‑bit maintains 2–3× lower distortion per `cq_distortion()` measurements.
- **Speed**: Modest CPU gains (~5–10%) for 2‑bit; GPU performance equivalent.
- **Bandwidth**: 2‑bit halves memory bandwidth, critical for edge deployment.
- **Training**: 4‑bit enables easier QAT convergence with lower `noise_scale()` values.
- **Configuration**: Set via `--bits` or `--bits-map` at export; runtime auto‑detects from tensor directory.

## Frequently Asked Questions

### Is 2‑bit quantization compatible with all Needle features?

Yes. The `.cact` format stores the bit‑width in each tensor's directory (`bits` field, export.py L34–38), and the engine automatically adjusts decoding. Both 2‑bit and 4‑bit support LoRA adapters, KV cache quantization, and the full inference pipeline.

### How do I measure distortion for my specific model?

Call `cq_distortion()` directly from `needle.model.quantize`. Pass your weight tensor and target bit‑width; it returns relative mean‑squared error. Compare 2‑bit vs 4‑bit results on representative layers to estimate downstream accuracy impact.

### Can I mix 2‑bit and 4‑bit within the same `.cact` file?

Yes. Use the `--bits-map` CLI option with layer‑specific assignments. The parser in `parse_bits_map()` (quantize.py L301–318) validates your configuration, and the exporter records per‑tensor bit‑widths in the archive header.

### Does 2‑bit quantization affect the KV cache?

No—KV cache precision is controlled separately via `kv_bits` in `configure_deploy()`. You can run 2‑bit weights with 8‑bit KV cache, or pair 4‑bit weights with 2‑bit KV cache to reduce memory during long‑context generation.