Cactus Quants Supported Bit Widths and Ternary Quantization Explained

Cactus Quants supports 2-bit, 3-bit, and 4-bit standard weight quantization, plus specialized ternary quantization at approximately 1.58 bits per weight, alongside 8-bit activations and optional KV-cache compression up to 7 bits.

Cactus Quants is the quantization engine powering the cactus-compute/needle open-source inference framework, designed to compress large language models for efficient deployment. The library implements flexible quantization schemes that balance model size reduction with computational accuracy. This guide examines the specific bit widths available in Cactus Quants and explains the mechanics of its unique ternary quantization approach.

Supported Bit Widths in Cactus Quants

The quantization configuration in needle/model/quantize.py defines distinct precision levels for weights, activations, and KV-cache tensors. The implementation separates standard integer bit widths from specialized ternary compression modes.

Standard CQ Weight Quantization (2, 3, and 4 Bits)

Standard Cactus Quant (CQ) weights support three discrete bit widths defined in the CQ_BITS constant. According to lines 198-200 of needle/model/quantize.py, the supported values are 2, 3, and 4 bits:

# From needle/model/quantize.py lines 198-200

CQ_BITS = (2, 3, 4)

The default quantization level for weights is set by _WEIGHT_BITS = 4 at lines 49-51. When configuring quantization-aware training (QAT) or deployment, you select from these three integer options to compress model parameters while maintaining inference quality.

Ternary Quantization (~1.58 Bits per Weight)

Ternary quantization represents a special case in the Cactus Quants framework, utilizing a three-level codebook [-1, 0, +1]. The effective bit width is defined by the constant TERNARY_BITS = 1.58 (lines 108-110), which corresponds to the theoretical entropy of three distinct values (log₂3 ≈ 1.585).

For on-disk storage, the implementation uses TERNARY_RECORD_BITS = 5 (lines 92-94), providing byte-aligned records that accommodate the ternary indices efficiently despite the non-integer bit requirement.

Activation and KV-Cache Quantization

Activations are quantized using ACT_BITS = 8 (line 51) by default, maintaining higher precision for intermediate feature maps during inference. The KV-cache follows a separate configuration pattern:

• Default: KV_BITS = 0 (disabled, full precision)
• Optional range: Any value ≤ 7 bits when enabled via configure_deploy (lines 61-64)

If kv_bits is set to 8 or higher, the KV-cache remains in full floating-point precision; otherwise, it compresses to the specified bit width.

How Ternary Quantization Works in Needle

Ternary quantization in Cactus Quants employs a custom packing strategy to handle the three-level codebook efficiently. The implementation spans across needle/model/quantize.py and needle/model/export.py.

Three-Level Codebook Implementation

The ternary codebook is constructed once as _TERNARY_CB at lines 109-110 of quantize.py, containing the values [-1, 0, +1] scaled by the group size. During quantization, each weight is mapped to the nearest of these three values, effectively reducing the precision to trits (ternary digits) rather than bits.

Packing Strategy and Storage Format

Because three levels cannot be addressed by a whole number of bits, the library packs ternary indices into 2-bit "crumbs" using _pack_ternary_crumbs, then expands them to 5-bit records for storage. The unpacking logic resides in needle/model/export.py (lines 74-81):

# Conceptual unpacking flow in export.py

def _unpack_ternary_crumbs(packed_buffer):
    # Extracts 2-bit crumbs from 5-bit records

    # Reconstructs ternary indices for dequantization

    pass

This approach balances storage efficiency with deserialization speed, using byte-aligned 5-bit containers rather than bit-packed streams that would require expensive shift operations at runtime.

Effective Bit Width Calculation

While the storage format uses 5-bit records, the effective compression achieved is approximately 1.58 bits per weight. This represents the information-theoretic minimum for three distinct symbols, making ternary quantization more efficient than 2-bit binary quantization while preserving more expressiveness than 1-bit binary quantization.

Implementation Details and Source Code

The Cactus Quants implementation is distributed across several core files in the cactus-compute/needle repository:

• needle/model/quantize.py: Core quantization utilities, constants (CQ_BITS, TERNARY_BITS, ACT_BITS), and configuration functions (configure_qat, configure_deploy)
• needle/model/export.py: Packing and unpacking logic for ternary weights (_pack_ternary_crumbs, _unpack_ternary_crumbs) and standard CQ weight serialization
• needle/model/run.py: Runtime quantization application to model parameters during inference
• needle/model/architecture.py: Layer definitions where quantization hooks (kernel, embedding) are applied to model weights
• needle/cli.py: Command-line interface exposing quantization configuration options

Configuring Cactus Quants in Practice

You can configure quantization settings through the Python API or CLI. The following examples demonstrate how to apply the supported bit widths in training and deployment scenarios.

Quantization-Aware Training Configuration

Configure 4-bit weight quantization with 128-element grouping during training:

from needle.model.quantize import configure_qat

# Apply QAT every 1000 steps with 4-bit weights (from CQ_BITS)

configure_qat(every=1000, weight_group=128, weight_bits=4)

Deployment with Ternary Weights

Enable ternary quantization for inference, alongside 8-bit activations:

from needle.model.quantize import configure_deploy, TERNARY_BITS

# Use 8-bit activations and ternary weights (~1.58 bits)

# KV-cache remains in full precision (kv_bits=0)

configure_deploy(act_bits=8, kv_bits=0, kv_group=64)

# Internally, setting bits=TERNARY_BITS selects the ternary codebook

Manual Parameter Quantization

Apply quantization directly to a parameter tree for export:

from needle.model.quantize import cq_quantize_params, TERNARY_BITS

# Quantize all layers to 3-bit standard CQ

params_q3 = cq_quantize_params(params, bits=3, group_size=128)

# Quantize specific layers to ternary using the special constant

params_ternary = cq_quantize_params(params, bits=TERNARY_BITS, group_size=128)

Summary

• Standard CQ weights support 2-bit, 3-bit, and 4-bit quantization as defined by the CQ_BITS constant in needle/model/quantize.py.
• Ternary quantization achieves approximately 1.58 bits per weight using a three-level codebook [-1, 0, +1], stored in 5-bit records for byte alignment.
• Activations default to 8-bit precision, while the KV-cache supports compression up to 7 bits or remains in full precision when disabled.
• Configuration is handled through configure_qat() for training and configure_deploy() for inference, with manual quantization available via cq_quantize_params().

Frequently Asked Questions

What bit widths does Cactus Quants support for model weights?

Cactus Quants supports 2-bit, 3-bit, and 4-bit quantization for standard CQ weights, as specified by the CQ_BITS = (2, 3, 4) constant in needle/model/quantize.py (lines 198-200). Additionally, the framework supports ternary quantization at approximately 1.58 bits per weight using a specialized three-level codebook.

How does ternary quantization achieve 1.58 bits per weight?

Ternary quantization uses a codebook with three values ([-1, 0, +1]), requiring log₂3 ≈ 1.58 bits of information per weight theoretically. The implementation stores these values in 5-bit records (TERNARY_RECORD_BITS = 5) to maintain byte alignment, packing the ternary indices as 2-bit "crumbs" using the _pack_ternary_crumbs function in needle/model/export.py.

Can I use different bit widths for activations and weights?

Yes, Cactus Quants allows independent configuration of activation and weight bit widths. Weights can be set to 2, 3, 4 bits, or ternary (~1.58 bits), while activations default to 8 bits (ACT_BITS = 8). The KV-cache can be configured separately with any bit width ≤ 7 or disabled (0) to remain in full precision.

Where is the quantization logic implemented in the Needle repository?

The core quantization logic resides in needle/model/quantize.py, which defines constants like CQ_BITS and TERNARY_BITS, and provides the cq_quantize_params() function. Ternary packing utilities are implemented in needle/model/export.py, while the runtime application of quantization hooks occurs in needle/model/run.py and needle/model/architecture.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}