# How to Configure Mixed-Precision Quantization During Needle 2 Export

> Learn how to configure mixed-precision quantization in Needle 2 export using the bits-map CLI flag. Optimize your models for performance and reduced memory footprint.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: how-to-guide
- Published: 2026-08-30

---

**Needle 2 supports mixed-precision quantization during export through a `--bits-map` CLI flag that maps tensor name prefixes to specific bit-widths, requiring a mandatory `default=<bits>` fallback for unspecified tensors.**

Needle 2 enables efficient model deployment through configurable mixed-precision quantization, allowing different architectural components to use varying bit-widths based on their sensitivity. When exporting a checkpoint to the Needle 2 binary format, you can specify per-tensor precision levels using either command-line arguments or embedded configuration metadata. This guide explains how to configure these quantization schemes using the actual implementation in the Cactus Compute Needle repository.

## Using the --bits-map CLI Flag

The primary method for configuring mixed-precision quantization is the `--bits-map` argument passed to the `needle export` command. This flag accepts a comma-separated specification where each entry follows the format `<tensor-prefix>=<bits>`.

### Syntax and Required Parameters

The bits-map string must include a `default=<bits>` entry that serves as the fallback quantization width for any tensor not explicitly matched by other prefixes. Additional entries override this default for specific tensor groups based on canonical name matching.

```bash
needle export \
    --checkpoint my_model.ckpt \
    --out my_model.n2.bin \
    --bits-map default=4,attention=8,mlp=4

```

In this example:
- **default=4**: Assigns 4-bit quantization to all tensors without a specific override.
- **attention=8**: Uses 8-bit precision for any tensor whose canonical name starts with "attention".
- **mlp=4**: Explicitly sets 4-bit for MLP layers (redundant here but demonstrates override capability).

### Prefix Matching Rules

When applying quantization, Needle 2 uses longest-prefix matching against canonical tensor names. The `_bits_for` function iterates through the bits map to find the most specific prefix match, falling back to the default width only when no prefixes match.

## Embedding Quantization Schemes in Checkpoints

Alternatively, you can store the quantization configuration directly within the checkpoint metadata. If the checkpoint's configuration contains a `weight_bits` field, Needle 2 automatically reads this value when no `--bits-map` flag is provided.

In [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py) (lines 540-546), the export logic checks for an embedded scheme:

```python
bits_map = getattr(args, "bits_map", None)
if bits is None and not bits_map:
    bits_map = getattr(config, "weight_bits", "") or ""
    if bits_map:
        print(f"[export] scheme from the checkpoint: {bits_map}")

```

This allows you to omit the `--bits-map` flag during export if the checkpoint was saved with a predefined quantization strategy.

## How Mixed-Precision Quantization Works Internally

Understanding the internal pipeline helps debug precision configurations. The export process involves two main phases: parsing the specification and applying per-tensor quantization.

### Parsing the Bits Map

The `parse_bits_map` function in [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py) (lines 301-318) validates and converts the specification string into a structured dictionary. It handles special cases such as ternary weights (represented by the `TERNARY_BITS` constant) and ensures the mandatory default entry exists.

```python
def parse_bits_map(spec):
    m, default = {}, None
    for part in str(spec or "").split(","):
        if not part.strip():
            continue
        k, v = part.split("=", 1)
        b = TERNARY_BITS if float(v) == TERNARY_BITS else int(v)
        if k.strip() == "default":
            default = b
        else:
            m[canonical_tensor_name(k.strip())] = b
    if default is None:
        raise ValueError(f"bits map {spec!r} needs a default=<b> entry")
    return m, default

```

The function returns two objects: a dictionary mapping canonical tensor prefixes to bit-widths, and the default fallback width.

### Applying Per-Tensor Quantization

After parsing, the `cq_mixed_params` function (lines 321-328 in [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py)) iterates through all leaf tensors in the parameter tree. For each tensor, it determines the appropriate bit-width using the longest-matching prefix from the bits map, then applies uniform quantization via `cq_quantize`.

```python
def cq_mixed_params(params, bits_map, default_bits, group_size=CQ_GROUP_SIZE):
    names = [n for n, _ in quant_leaf_names(params)]

    def fn(w, i):
        b = _bits_for(names[i], bits_map, default_bits)
        return cq_quantize(w, b, group_size)

    return _map_quant_leaves(params, fn)

```

Each leaf tensor receives the bit-width returned by `_bits_for`, which implements the longest-matching prefix logic against the canonical tensor names.

## Programmatic Export API

For Python-based workflows, you can invoke the export function directly from `needle.model.export`. The `bits_map` parameter accepts the same string format as the CLI flag.

```python
from needle.model.export import export

export(
    checkpoint="gpt2.ckpt",
    out="gpt2_n2.bin",
    bits_map="default=4,attention=8,mlp=4"
)

```

## Verifying the Export Output

Upon successful export, Needle 2 prints a summary showing the applied quantization scheme. According to lines 564-566 in [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py), the output indicates whether mixed-precision was used and displays the specific bits map configuration.

```

wrote model.n2.bin: 128 tensors (+tokenizer), 512.00 MB (mixed[default=4,attention=8]A8)

```

The `mixed[...]` notation confirms that mixed-precision quantization was applied, while `A8` indicates that activations remain at 8-bit precision.

## Summary

- Use the `--bits-map` CLI flag with format `default=<bits>,<prefix>=<bits>` to configure mixed-precision quantization during Needle 2 export.
- The `default=<bits>` entry is mandatory and serves as the fallback for unmatched tensors.
- Store persistent quantization schemes in checkpoint configuration under the `weight_bits` field to omit CLI flags.
- The `parse_bits_map` and `cq_mixed_params` functions in [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py) handle parsing and application of per-tensor bit-widths.
- Longest-prefix matching determines which bit-width applies to each tensor based on its canonical name.

## Frequently Asked Questions

### What is the required format for the --bits-map string?

The string must be comma-separated with `key=value` pairs. You must include `default=<integer>` to set the fallback bit-width. Other entries like `attention=8` or `mlp=4` override the default for tensors whose canonical names start with those prefixes. The parser in [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py) splits on commas and equals signs, converting values to integers (or `TERNARY_BITS` for special cases).

### Can I mix different bit-widths within the same model?

Yes, this is the core capability of Needle 2's mixed-precision quantization. You can assign different bit-widths to different architectural components (e.g., 8-bit for attention layers, 4-bit for MLP layers) using a single bits-map specification. The `cq_mixed_params` function handles the per-tensor application automatically during export.

### What happens if I don't specify a --bits-map flag?

If omitted, Needle 2 checks the checkpoint configuration for a `weight_bits` attribute. If found, it uses that embedded scheme automatically. If neither the CLI flag nor the checkpoint field exists, the export process may fail or require additional parameters depending on the specific model architecture and export version.

### How does Needle 2 determine which bit-width to use for each tensor?

The system uses longest-prefix matching against canonical tensor names. The `_bits_for` helper function iterates through the parsed bits map and selects the most specific matching prefix. If no prefix matches, it falls back to the `default` bit-width specified in the map. This allows fine-grained control where `transformer.attention` can match more specifically than `transformer` alone.