How to Configure Mixed-Precision Quantization During Needle 2 Export

Needle 2 supports mixed-precision quantization during export through a --bits-map CLI flag that maps tensor name prefixes to specific bit-widths, requiring a mandatory default=<bits> fallback for unspecified tensors.

Needle 2 enables efficient model deployment through configurable mixed-precision quantization, allowing different architectural components to use varying bit-widths based on their sensitivity. When exporting a checkpoint to the Needle 2 binary format, you can specify per-tensor precision levels using either command-line arguments or embedded configuration metadata. This guide explains how to configure these quantization schemes using the actual implementation in the Cactus Compute Needle repository.

Using the --bits-map CLI Flag

The primary method for configuring mixed-precision quantization is the --bits-map argument passed to the needle export command. This flag accepts a comma-separated specification where each entry follows the format <tensor-prefix>=<bits>.

Syntax and Required Parameters

The bits-map string must include a default=<bits> entry that serves as the fallback quantization width for any tensor not explicitly matched by other prefixes. Additional entries override this default for specific tensor groups based on canonical name matching.

needle export \
    --checkpoint my_model.ckpt \
    --out my_model.n2.bin \
    --bits-map default=4,attention=8,mlp=4

In this example:

• default=4: Assigns 4-bit quantization to all tensors without a specific override.
• attention=8: Uses 8-bit precision for any tensor whose canonical name starts with "attention".
• mlp=4: Explicitly sets 4-bit for MLP layers (redundant here but demonstrates override capability).

Prefix Matching Rules

When applying quantization, Needle 2 uses longest-prefix matching against canonical tensor names. The _bits_for function iterates through the bits map to find the most specific prefix match, falling back to the default width only when no prefixes match.

Embedding Quantization Schemes in Checkpoints

Alternatively, you can store the quantization configuration directly within the checkpoint metadata. If the checkpoint's configuration contains a weight_bits field, Needle 2 automatically reads this value when no --bits-map flag is provided.

In needle/model/export.py (lines 540-546), the export logic checks for an embedded scheme:

bits_map = getattr(args, "bits_map", None)
if bits is None and not bits_map:
    bits_map = getattr(config, "weight_bits", "") or ""
    if bits_map:
        print(f"[export] scheme from the checkpoint: {bits_map}")

This allows you to omit the --bits-map flag during export if the checkpoint was saved with a predefined quantization strategy.

How Mixed-Precision Quantization Works Internally

Understanding the internal pipeline helps debug precision configurations. The export process involves two main phases: parsing the specification and applying per-tensor quantization.

Parsing the Bits Map

The parse_bits_map function in needle/model/quantize.py (lines 301-318) validates and converts the specification string into a structured dictionary. It handles special cases such as ternary weights (represented by the TERNARY_BITS constant) and ensures the mandatory default entry exists.

def parse_bits_map(spec):
    m, default = {}, None
    for part in str(spec or "").split(","):
        if not part.strip():
            continue
        k, v = part.split("=", 1)
        b = TERNARY_BITS if float(v) == TERNARY_BITS else int(v)
        if k.strip() == "default":
            default = b
        else:
            m[canonical_tensor_name(k.strip())] = b
    if default is None:
        raise ValueError(f"bits map {spec!r} needs a default=<b> entry")
    return m, default

The function returns two objects: a dictionary mapping canonical tensor prefixes to bit-widths, and the default fallback width.

Applying Per-Tensor Quantization

After parsing, the cq_mixed_params function (lines 321-328 in needle/model/quantize.py) iterates through all leaf tensors in the parameter tree. For each tensor, it determines the appropriate bit-width using the longest-matching prefix from the bits map, then applies uniform quantization via cq_quantize.

def cq_mixed_params(params, bits_map, default_bits, group_size=CQ_GROUP_SIZE):
    names = [n for n, _ in quant_leaf_names(params)]

    def fn(w, i):
        b = _bits_for(names[i], bits_map, default_bits)
        return cq_quantize(w, b, group_size)

    return _map_quant_leaves(params, fn)

Each leaf tensor receives the bit-width returned by _bits_for, which implements the longest-matching prefix logic against the canonical tensor names.

Programmatic Export API

For Python-based workflows, you can invoke the export function directly from needle.model.export. The bits_map parameter accepts the same string format as the CLI flag.

from needle.model.export import export

export(
    checkpoint="gpt2.ckpt",
    out="gpt2_n2.bin",
    bits_map="default=4,attention=8,mlp=4"
)

Verifying the Export Output

Upon successful export, Needle 2 prints a summary showing the applied quantization scheme. According to lines 564-566 in needle/model/export.py, the output indicates whether mixed-precision was used and displays the specific bits map configuration.

wrote model.n2.bin: 128 tensors (+tokenizer), 512.00 MB (mixed[default=4,attention=8]A8)

The mixed[...] notation confirms that mixed-precision quantization was applied, while A8 indicates that activations remain at 8-bit precision.

Summary

• Use the --bits-map CLI flag with format default=<bits>,<prefix>=<bits> to configure mixed-precision quantization during Needle 2 export.
• The default=<bits> entry is mandatory and serves as the fallback for unmatched tensors.
• Store persistent quantization schemes in checkpoint configuration under the weight_bits field to omit CLI flags.
• The parse_bits_map and cq_mixed_params functions in needle/model/quantize.py handle parsing and application of per-tensor bit-widths.
• Longest-prefix matching determines which bit-width applies to each tensor based on its canonical name.

Frequently Asked Questions

What is the required format for the --bits-map string?

The string must be comma-separated with key=value pairs. You must include default=<integer> to set the fallback bit-width. Other entries like attention=8 or mlp=4 override the default for tensors whose canonical names start with those prefixes. The parser in needle/model/quantize.py splits on commas and equals signs, converting values to integers (or TERNARY_BITS for special cases).

Can I mix different bit-widths within the same model?

Yes, this is the core capability of Needle 2's mixed-precision quantization. You can assign different bit-widths to different architectural components (e.g., 8-bit for attention layers, 4-bit for MLP layers) using a single bits-map specification. The cq_mixed_params function handles the per-tensor application automatically during export.

What happens if I don't specify a --bits-map flag?

If omitted, Needle 2 checks the checkpoint configuration for a weight_bits attribute. If found, it uses that embedded scheme automatically. If neither the CLI flag nor the checkpoint field exists, the export process may fail or require additional parameters depending on the specific model architecture and export version.

How does Needle 2 determine which bit-width to use for each tensor?

The system uses longest-prefix matching against canonical tensor names. The _bits_for helper function iterates through the parsed bits map and selects the most specific matching prefix. If no prefix matches, it falls back to the default bit-width specified in the map. This allows fine-grained control where transformer.attention can match more specifically than transformer alone.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}