How to Configure CQ2-bit Quantization (2-bit vs 4-bit) When Building a .cact File in Needle

Use the --bits CLI flag with the needle build command to select 2-bit compression (--bits 2) or 4-bit compression (--bits 4, default), or supply a --bits-map for per-layer control.

Needle's .cact export format supports CQ (code-book) quantization to reduce model size through low-precision weight compression. When building a .cact file from a trained checkpoint, you configure the quantization bit-width via command-line flags parsed in needle/cli.py and implemented in needle/model/quantize.py.

---

Understanding CQ2-bit Quantization in Needle

The CQ quantizer compresses model weights using learnable codebooks. The bit-width determines the number of centroids in the codebook:

• 2-bit quantization: 4 centroids per codebook → maximum compression, lower precision
• 4-bit quantization: 16 centroids per codebook → balanced compression and accuracy (default)

The default weight-bits value is defined as _WEIGHT_BITS = 4 in needle/model/quantize.py【/cache/repos/github.com/cactus-compute/needle/main/needle/model/quantize.py†L49-L53】.

---

Basic Usage: Setting Bit-Width with --bits

The --bits flag is parsed in needle/cli.py【/cache/repos/github.com/cactus-compute/needle/main/needle/cli.py†L56-L57】 and forwarded to the export routine build_export.

Default 4-bit Quantization

# 4-bit is the default; --bits 4 is optional

needle build checkpoints/needle2.pkl --out my_model.cact

Explicit 2-bit Quantization

# Force 2-bit quantization for maximum compression

needle build checkpoints/needle2.pkl --bits 2 --out my_model_2bit.cact

---

Advanced Usage: Per-Layer Control with --bits-map

For mixed-precision quantization, use --bits-map to specify different bit-widths for different tensors. The parse_bits_map function in needle/model/quantize.py【/cache/repos/github.com/cactus-compute/needle/main/needle/model/quantize.py†L301-L313】 parses this string format.

Syntax and Example

The bits-map uses comma-separated key=value pairs:

# Default 2-bit, but layer0 (first attention) at 4-bit

needle build checkpoints/needle2.pkl \
    --bits-map "default=2,layer0=4" \
    --out my_model_mixed.cact

Common patterns:

• "default=4" — equivalent to omitting the flag
• "default=2,layer0=4,layer1=4" — aggressive compression with critical layers protected
• "weight=2,embedding=4" — tensor-type-based assignment (if supported by model naming)

---

File Reference: Where Quantization Is Implemented

File	Function	Role
needle/cli.py	CLI argument parsing【/cache/repos/github.com/cactus-compute/needle/main/needle/cli.py†L56-L57】	Parses --bits and --bits-map for build subcommand
needle/model/quantize.py	_WEIGHT_BITS = 4, parse_bits_map()【/cache/repos/github.com/cactus-compute/needle/main/needle/model/quantize.py†L49-L53】【/cache/repos/github.com/cactus-compute/needle/main/needle/model/quantize.py†L301-L313】	Defines defaults and parses per-tensor specifications
needle/model/export.py	build_export()	Applies quantization settings when packing to .cact
needle/model/finetune.py	build_main()	Entry point that forwards CLI options to export pipeline

---

Summary

• Specify bit-width with --bits 2 or --bits 4 (default) when running needle build
• Use --bits-map for fine-grained, per-layer quantization control via parse_bits_map in needle/model/quantize.py
• 4-bit is the default as defined by _WEIGHT_BITS = 4 in the quantizer source
• 2-bit reduces size further but may degrade accuracy; use --bits-map to protect sensitive layers

---

Frequently Asked Questions

What is the difference between 2-bit and 4-bit CQ quantization in Needle?

2-bit quantization uses 4 codebook centroids to represent weights, while 4-bit uses 16 centroids. More bits mean higher precision and typically better model accuracy, but larger file sizes. The 4-bit default in needle/model/quantize.py【/cache/repos/github.com/cactus-compute/needle/main/needle/model/quantize.py†L49-L53】 balances compression and quality.

Can I mix 2-bit and 4-bit quantization in the same .cact file?

Yes. The --bits-map flag accepts a comma-separated string (e.g., "default=2,layer0=4") that parse_bits_map processes【/cache/repos/github.com/cactus-compute/needle/main/needle/model/quantize.py†L301-L313】. This lets you compress most layers aggressively while keeping critical structures at higher precision.

Where does Needle parse the --bits CLI argument?

The needle build command parses --bits and --bits-map in needle/cli.py【/cache/repos/github.com/cactus-compute/needle/main/needle/cli.py†L56-L57】. These values propagate through build_main in needle/model/finetune.py to the export pipeline that generates the .cact archive.

How do I verify which bit-width my .cact file uses?

The quantization configuration is embedded in the .cact archive metadata. While Needle does not expose a direct inspection command, you can infer the setting from the file size—2-bit models are roughly half the size of 4-bit equivalents for the same architecture—or by checking the build command history.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}