2‑Bit vs 4‑Bit Quantization Tradeoffs for `.cact` Files: A Complete Technical Guide

2‑bit quantization cuts model size in half compared to 4‑bit but increases distortion 2–3×, while 4‑bit offers the best balance of compression and accuracy for most production deployments. This article examines the engineering tradeoffs between these bit‑widths in Needle's compact binary archive format.

Needle, an open‑source inference engine from cactus‑compute/needle, stores quantized model weights in .cact files. The quantization bit‑width you choose directly impacts archive size, numerical fidelity, and runtime performance. This guide breaks down the implementation details found in the source code to help you make an informed decision.

How Quantization Bit‑Width Determines Archive Size

The .cact format stores CQ (compressed quantization) tensors as packed index streams. Archive size scales linearly with bits per weight.

In needle/model/quantize.py, the function cq_model_bytes() computes total bytes at lines 176–189:

# From quantize.py L176-L189

# Size formula: acc["q"] * bits / 8

Key implications:

• 2‑bit (CQ 2): Each weight occupies 0.25 bytes. Roughly half the size of a 4‑bit model.
• 4‑bit (CQ 4): Each weight occupies 0.5 bytes. The default for most deployments.

The packed index stream format is defined in needle/model/export.py lines 51–58, where the bits field in the tensor directory records whether each tensor uses 2 or 4 bits.

Quantization Distortion and Accuracy Impact

Numerical fidelity degrades predictably with lower bit‑widths.

The cq_distortion() function (quantize.py L225–229) measures relative mean‑squared error for a given bit‑width. Empirical results show:

Bit‑Width	Approximate Distortion
4‑bit	Baseline (1×)
2‑bit	2–3× higher than 4‑bit

This distortion manifests as increased quantization noise during inference. For tasks requiring high precision—mathematical reasoning, code generation, or long‑context coherence—4‑bit quantization preserves more of the original model's capability.

Runtime Performance Characteristics

Speed Considerations

The engine's hot‑path uses a Walsh‑Hadamard rotation followed by codebook lookup. Per‑group work remains identical regardless of bit‑width, so speed differences are modest:

• CPU: ~5–10% faster for 2‑bit due to tighter packing
• GPU: Negligible difference (memory bandwidth, not compute, is the bottleneck)

The reconstruction pipeline in export.py lines 60–66 handles both bit‑widths uniformly: expand packed indices, multiply by per‑group L2 norm, apply Hadamard rotation.

Memory Bandwidth

2‑bit quantization halves memory bandwidth requirements when streaming weights from RAM or SSD. This benefit matters most on:

• Low‑end mobile CPUs
• Edge devices with slow external storage
• Batch‑1 inference where weight loading dominates latency

Training Implications: Quantization‑Aware Noise

When using quantization‑aware training (QAT), the noise_scale() function (quantize.py L231–239) adjusts noise injection based on effective bits:

b = min(max(float(bits), MIN_BITS), MAX_BITS)

Lower bit‑widths yield larger noise scales, making QAT convergence more difficult. If you plan to fine‑tune with Needle's quantization‑aware helpers, 4‑bit provides a more stable training surface.

Configuring Bit‑Width in Practice

Command‑Line Export

Pass --bits to needle build (quantization configuration in configure_deploy, quantize.py L61–66):

import subprocess

# 2-bit model for extreme compression

subprocess.run([
    "needle", "build",
    "model.ckpt",
    "--out", "model_2bit.cact",
    "--bits", "2"
])

# 4-bit default for balanced deployment

subprocess.run([
    "needle", "build",
    "model.ckpt",
    "--out", "model_4bit.cact",
    "--bits", "4"
])

Mixed‑Precision Export

Apply different bit‑widths to different layers using --bits-map:

# Hardware layers 2-bit, rest 4-bit

subprocess.run([
    "needle", "build",
    "model.ckpt",
    "--out", "model_mixed.cact",
    "--bits-map", "layer00=2,layer01=2,default=4"
])

The parser validates this map in parse_bits_map() (quantize.py L301–318).

Runtime KV Cache Configuration

Control activation and KV cache precision separately via configure_deploy:

from needle import Needle, configure_deploy

# 8-bit activations, 2-bit KV cache (saves memory on long contexts)

configure_deploy(act_bits=8, kv_bits=2, kv_group=64)

agent = Needle(weights="model_2bit.cact")

Codebook and Packing Internals

The bit‑width determines codebook structure built in _cq_codebook_np (quantize.py L112–115):

• 2‑bit: 4‑level Lloyd‑Max codebook
• 4‑bit: 16‑level Lloyd‑Max codebook

Despite different codebook sizes, the runtime decoding path is identical—no functional penalty, only precision difference.

When to Choose Each Bit‑Width

Scenario	Recommended Bit‑Width	Rationale
Edge devices, cheap CPUs, 30–40% size reduction critical	2‑bit	Half the bandwidth, acceptable accuracy loss
Production services, general deployment	4‑bit	Best fidelity/compression balance
QAT fine‑tuning	4‑bit	Stable convergence, lower noise scale
Mixed layer importance	Mixed	Critical layers 4‑bit, others 2‑bit

Summary

• Model size: 2‑bit produces ~50% smaller .cact files than 4‑bit, computed via cq_model_bytes() in quantize.py.
• Accuracy: 4‑bit maintains 2–3× lower distortion per cq_distortion() measurements.
• Speed: Modest CPU gains (~5–10%) for 2‑bit; GPU performance equivalent.
• Bandwidth: 2‑bit halves memory bandwidth, critical for edge deployment.
• Training: 4‑bit enables easier QAT convergence with lower noise_scale() values.
• Configuration: Set via --bits or --bits-map at export; runtime auto‑detects from tensor directory.

Frequently Asked Questions

Is 2‑bit quantization compatible with all Needle features?

Yes. The .cact format stores the bit‑width in each tensor's directory (bits field, export.py L34–38), and the engine automatically adjusts decoding. Both 2‑bit and 4‑bit support LoRA adapters, KV cache quantization, and the full inference pipeline.

How do I measure distortion for my specific model?

Call cq_distortion() directly from needle.model.quantize. Pass your weight tensor and target bit‑width; it returns relative mean‑squared error. Compare 2‑bit vs 4‑bit results on representative layers to estimate downstream accuracy impact.

Can I mix 2‑bit and 4‑bit within the same .cact file?

Yes. Use the --bits-map CLI option with layer‑specific assignments. The parser in parse_bits_map() (quantize.py L301–318) validates your configuration, and the exporter records per‑tensor bit‑widths in the archive header.

Does 2‑bit quantization affect the KV cache?

No—KV cache precision is controlled separately via kv_bits in configure_deploy(). You can run 2‑bit weights with 8‑bit KV cache, or pair 4‑bit weights with 2‑bit KV cache to reduce memory during long‑context generation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}