# Cactus Quants Supported Bit Widths and Ternary Quantization Explained

> Explore Cactus Quants supported bit widths: 2-bit, 3-bit, 4-bit, and ternary quantization. Learn about 8-bit activations and KV-cache compression for efficient model performance.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: deep-dive
- Published: 2026-09-09

---

**Cactus Quants supports 2-bit, 3-bit, and 4-bit standard weight quantization, plus specialized ternary quantization at approximately 1.58 bits per weight, alongside 8-bit activations and optional KV-cache compression up to 7 bits.**

Cactus Quants is the quantization engine powering the `cactus-compute/needle` open-source inference framework, designed to compress large language models for efficient deployment. The library implements flexible quantization schemes that balance model size reduction with computational accuracy. This guide examines the specific bit widths available in Cactus Quants and explains the mechanics of its unique ternary quantization approach.

## Supported Bit Widths in Cactus Quants

The quantization configuration in [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py) defines distinct precision levels for weights, activations, and KV-cache tensors. The implementation separates standard integer bit widths from specialized ternary compression modes.

### Standard CQ Weight Quantization (2, 3, and 4 Bits)

Standard Cactus Quant (CQ) weights support three discrete bit widths defined in the `CQ_BITS` constant. According to lines 198-200 of [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py), the supported values are **2, 3, and 4 bits**:

```python

# From needle/model/quantize.py lines 198-200

CQ_BITS = (2, 3, 4)

```

The default quantization level for weights is set by `_WEIGHT_BITS = 4` at lines 49-51. When configuring quantization-aware training (QAT) or deployment, you select from these three integer options to compress model parameters while maintaining inference quality.

### Ternary Quantization (~1.58 Bits per Weight)

Ternary quantization represents a special case in the Cactus Quants framework, utilizing a three-level codebook `[-1, 0, +1]`. The effective bit width is defined by the constant `TERNARY_BITS = 1.58` (lines 108-110), which corresponds to the theoretical entropy of three distinct values (log₂3 ≈ 1.585).

For on-disk storage, the implementation uses `TERNARY_RECORD_BITS = 5` (lines 92-94), providing byte-aligned records that accommodate the ternary indices efficiently despite the non-integer bit requirement.

### Activation and KV-Cache Quantization

Activations are quantized using `ACT_BITS = 8` (line 51) by default, maintaining higher precision for intermediate feature maps during inference. The KV-cache follows a separate configuration pattern:

- **Default**: `KV_BITS = 0` (disabled, full precision)
- **Optional range**: Any value ≤ 7 bits when enabled via `configure_deploy` (lines 61-64)

If `kv_bits` is set to 8 or higher, the KV-cache remains in full floating-point precision; otherwise, it compresses to the specified bit width.

## How Ternary Quantization Works in Needle

Ternary quantization in Cactus Quants employs a custom packing strategy to handle the three-level codebook efficiently. The implementation spans across [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py) and [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py).

### Three-Level Codebook Implementation

The ternary codebook is constructed once as `_TERNARY_CB` at lines 109-110 of [`quantize.py`](https://github.com/cactus-compute/needle/blob/main/quantize.py), containing the values `[-1, 0, +1]` scaled by the group size. During quantization, each weight is mapped to the nearest of these three values, effectively reducing the precision to trits (ternary digits) rather than bits.

### Packing Strategy and Storage Format

Because three levels cannot be addressed by a whole number of bits, the library packs ternary indices into 2-bit "crumbs" using `_pack_ternary_crumbs`, then expands them to 5-bit records for storage. The unpacking logic resides in [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py) (lines 74-81):

```python

# Conceptual unpacking flow in export.py

def _unpack_ternary_crumbs(packed_buffer):
    # Extracts 2-bit crumbs from 5-bit records

    # Reconstructs ternary indices for dequantization

    pass

```

This approach balances storage efficiency with deserialization speed, using byte-aligned 5-bit containers rather than bit-packed streams that would require expensive shift operations at runtime.

### Effective Bit Width Calculation

While the storage format uses 5-bit records, the **effective compression** achieved is approximately **1.58 bits per weight**. This represents the information-theoretic minimum for three distinct symbols, making ternary quantization more efficient than 2-bit binary quantization while preserving more expressiveness than 1-bit binary quantization.

## Implementation Details and Source Code

The Cactus Quants implementation is distributed across several core files in the `cactus-compute/needle` repository:

- **[`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py)**: Core quantization utilities, constants (`CQ_BITS`, `TERNARY_BITS`, `ACT_BITS`), and configuration functions (`configure_qat`, `configure_deploy`)
- **[`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py)**: Packing and unpacking logic for ternary weights (`_pack_ternary_crumbs`, `_unpack_ternary_crumbs`) and standard CQ weight serialization
- **[`needle/model/run.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py)**: Runtime quantization application to model parameters during inference
- **[`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py)**: Layer definitions where quantization hooks (`kernel`, `embedding`) are applied to model weights
- **[`needle/cli.py`](https://github.com/cactus-compute/needle/blob/main/needle/cli.py)**: Command-line interface exposing quantization configuration options

## Configuring Cactus Quants in Practice

You can configure quantization settings through the Python API or CLI. The following examples demonstrate how to apply the supported bit widths in training and deployment scenarios.

### Quantization-Aware Training Configuration

Configure 4-bit weight quantization with 128-element grouping during training:

```python
from needle.model.quantize import configure_qat

# Apply QAT every 1000 steps with 4-bit weights (from CQ_BITS)

configure_qat(every=1000, weight_group=128, weight_bits=4)

```

### Deployment with Ternary Weights

Enable ternary quantization for inference, alongside 8-bit activations:

```python
from needle.model.quantize import configure_deploy, TERNARY_BITS

# Use 8-bit activations and ternary weights (~1.58 bits)

# KV-cache remains in full precision (kv_bits=0)

configure_deploy(act_bits=8, kv_bits=0, kv_group=64)

# Internally, setting bits=TERNARY_BITS selects the ternary codebook

```

### Manual Parameter Quantization

Apply quantization directly to a parameter tree for export:

```python
from needle.model.quantize import cq_quantize_params, TERNARY_BITS

# Quantize all layers to 3-bit standard CQ

params_q3 = cq_quantize_params(params, bits=3, group_size=128)

# Quantize specific layers to ternary using the special constant

params_ternary = cq_quantize_params(params, bits=TERNARY_BITS, group_size=128)

```

## Summary

- **Standard CQ weights** support **2-bit, 3-bit, and 4-bit** quantization as defined by the `CQ_BITS` constant in [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py).
- **Ternary quantization** achieves approximately **1.58 bits per weight** using a three-level codebook `[-1, 0, +1]`, stored in 5-bit records for byte alignment.
- **Activations** default to **8-bit** precision, while the **KV-cache** supports compression up to **7 bits** or remains in full precision when disabled.
- Configuration is handled through `configure_qat()` for training and `configure_deploy()` for inference, with manual quantization available via `cq_quantize_params()`.

## Frequently Asked Questions

### What bit widths does Cactus Quants support for model weights?

Cactus Quants supports **2-bit, 3-bit, and 4-bit** quantization for standard CQ weights, as specified by the `CQ_BITS = (2, 3, 4)` constant in [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py) (lines 198-200). Additionally, the framework supports **ternary quantization** at approximately **1.58 bits per weight** using a specialized three-level codebook.

### How does ternary quantization achieve 1.58 bits per weight?

Ternary quantization uses a codebook with three values (`[-1, 0, +1]`), requiring log₂3 ≈ **1.58 bits** of information per weight theoretically. The implementation stores these values in **5-bit records** (`TERNARY_RECORD_BITS = 5`) to maintain byte alignment, packing the ternary indices as 2-bit "crumbs" using the `_pack_ternary_crumbs` function in [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py).

### Can I use different bit widths for activations and weights?

Yes, Cactus Quants allows independent configuration of activation and weight bit widths. Weights can be set to 2, 3, 4 bits, or ternary (~1.58 bits), while activations default to **8 bits** (`ACT_BITS = 8`). The KV-cache can be configured separately with any bit width ≤ 7 or disabled (0) to remain in full precision.

### Where is the quantization logic implemented in the Needle repository?

The core quantization logic resides in **[`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py)**, which defines constants like `CQ_BITS` and `TERNARY_BITS`, and provides the `cq_quantize_params()` function. Ternary packing utilities are implemented in **[`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py)**, while the runtime application of quantization hooks occurs in **[`needle/model/run.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py)** and **[`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py)**.