How to Export a Needle 2 Model with 2‑Bit Quantization: A Complete Guide

You can export a Needle 2 model with 2‑bit quantization by passing --bits 2 to the needle build CLI or setting bits=2 in the write_export Python API, which triggers the Cactus‑Quants (CQ) packing pipeline in needle/model/export.py.

Needle 2 stores model weights in a custom binary format called cact, utilizing the Cactus‑Quants (CQ) system to compress matrices into 2‑, 3‑, or 4‑bit representations. When you export a Needle 2 model with 2‑bit quantization, you significantly reduce memory footprint while maintaining full inference compatibility, as the runtime engine automatically detects the bit‑width from the file header. This guide covers both command‑line and programmatic methods to generate 2‑bit quantized .cact files.

Understanding Needle 2’s Cactus‑Quants (CQ) System

Needle 2 implements quantization through the CB_BITS constant in needle/model/export.py (lines 90‑92), which defines the supported bit widths for the CQ format. The quantizer supports 2‑, 3‑, and 4‑bit representations, with 2‑bit offering the highest compression ratio for deployment scenarios where model size is critical.

The CB_BITS Constant and Supported Bit Widths

In the source code, the CB_BITS tuple explicitly lists the valid quantization levels. When you request 2‑bit export, the system validates your selection against this constant before proceeding to the packing phase.

CLI Method: Using needle build with --bits 2

The fastest way to export a Needle 2 model with 2‑bit quantization is through the command‑line interface. The main() function in needle/model/export.py (lines 41‑58) parses the --bits argument and forwards it to write_export.

Basic 2‑Bit Export

To convert a base checkpoint to 2‑bit format:

needle build checkpoints/needle2.pkl \
    --out my_needle_2bit.cact \
    --bits 2

The --bits 2 flag forces the export pipeline to use 2‑bit codebooks during quantization. As documented in the README (lines 44‑46), this produces a .cact file containing LSB‑first packed indices and the corresponding 2‑bit codebook metadata.

Exporting with LoRA Fine‑Tuning

If you have applied LoRA (Low‑Rank Adaptation) fine‑tuning, merge the adapters during export:

needle build checkpoints/needle2.pkl \
    --lora checkpoints/needle_lora.pkl \
    --out my_needle_2bit.cact \
    --bits 2

This combines the base model parameters with LoRA weights before applying 2‑bit quantization, ensuring the adapted model is compressed and exported as a single .cact artifact.

Programmatic Method: The write_export Python API

For custom workflows, import write_export from needle.model.export and invoke it with bits=2. This method gives you explicit control over the quantization group size and tokenizer configuration.

import needle
from needle.model.export import write_export

# Load checkpoint (params, config) – see needle.model.run.load_checkpoint

params, config, _ = needle.model.run.load_checkpoint("checkpoints/needle2.pkl")

# Export with 2‑bit CQ quantization

info = write_export(
    params,
    config,
    path="my_needle_2bit.cact",
    bits=2,                     # Forces 2‑bit quantization

    group=128,                  # Default group size for CQ packing

    tokenizer=needle.model.run.get_tokenizer(config.vocab_size),
)
print(f"Exported {info['path']} ({info['bytes']/1e6:.2f} MB, {info['tensors']} tensors)")

The bits parameter reaches write_export (lines 87‑90) and propagates through to _pack_cact (lines 81‑85), initiating the low‑bit packing pipeline.

How 2‑Bit Quantization Works Under the Hood

When bits=2 is specified, the export pipeline follows a specific call chain in needle/model/export.py: write_export → _pack_cact → _q → _cq_pack.

The _cq_pack and _cq_codebook_np Functions

The _cq_pack helper selects the appropriate codebook using _cq_codebook_np(bits, group) (lines 84‑89). For 2‑bit quantization, this generates a 2‑bit codebook and packs the weight indices using an LSB‑first bit scheme. The packed blobs are then prepared for file serialization.

Header Structure and Runtime Detection

During export, the write_export function constructs a header that records the quantization parameters. Specifically, the tensor directory stores the bits field (lines 33‑38), which the Needle runtime reads to automatically configure the dequantizer. Because the header explicitly declares the 2‑bit format, no additional runtime configuration is required to execute the model.

Key Source Files for 2‑Bit Export

Understanding these files helps when debugging or extending the quantization pipeline:

• needle/model/export.py: Contains the core export logic, CQ packing implementation (_cq_pack), header construction, and CLI entry point (main()). This is where the --bits flag is parsed and the 2‑bit codebook is selected.
• needle/model/quantize.py: Provides helper functions for CQ codebook generation, Hadamard transforms, and bit‑width handling used during the packing process.
• needle/model/architecture.py: Defines the model configuration (attention heads, engram settings, KV window size) embedded in the .cact header.
• README.md: User‑level documentation covering the --bits 2 flag for smaller model exports.
• doc/finetuning.md: Describes the complete workflow from data synthesis through LoRA fine‑tuning to final quantized export.

Summary

• Needle 2 uses Cactus‑Quants (CQ) to support 2‑, 3‑, and 4‑bit weight quantization via the CB_BITS constant in export.py.
• Use the CLI flag --bits 2 with needle build to export 2‑bit models from the command line, compatible with both base and LoRA‑merged checkpoints.
• Call write_export(bits=2) in Python for programmatic export, allowing custom group sizes and tokenizer integration.
• The quantization pipeline invokes _cq_codebook_np(bits, group) and packs indices LSB‑first when 2‑bit mode is active.
• The exported .cact file header stores the bit‑width in the tensor directory, enabling automatic runtime detection without manual configuration.

Frequently Asked Questions

What bit widths does Needle 2 support for quantization?

Needle 2 supports 2‑bit, 3‑bit, and 4‑bit quantization through the Cactus‑Quants system. The CB_BITS constant in needle/model/export.py (lines 90‑92) explicitly defines these supported values, with 2‑bit providing the maximum compression ratio.

Can I export a LoRA‑fine‑tuned model with 2‑bit quantization?

Yes. Pass both the base checkpoint and LoRA weights to needle build, then specify --bits 2. The CLI merges the adapters before quantization, outputting a single compressed .cact file that contains the fine‑tuned parameters in 2‑bit format.

Does the Needle runtime require special configuration to load 2‑bit models?

No. The export process stores the bit‑width in the .cact file header (specifically in the tensor directory’s bits field). When you load the model, the engine reads this metadata and automatically initializes the appropriate dequantization kernels for 2‑bit inference.

How do I verify that my export actually used 2‑bit quantization?

The write_export function returns a dictionary containing export statistics. Check the file size or inspect the info return value; programmatically, you can also examine the .cact header structure, which explicitly records bits: 2 in the tensor metadata stored by the export logic in needle/model/export.py (lines 33‑38).

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}