# How Needle Compiles the Byte-Level Grammar for Constrained Decoding

> Learn how Needle compiles the byte-level grammar for constrained decoding by serializing model architecture, tensors, and weights into a compact binary format for efficient token generation.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: internals
- Published: 2026-09-12

---

**Needle compiles the byte-level grammar for constrained decoding by serializing the model architecture, tensor directory, and quantized weights into a compact binary format that the generator can scan directly during token generation.**

Needle compiles the byte-level grammar for constrained decoding by serializing the model architecture, tensor directory, and quantized weights into a compact binary format that the generator can scan directly during token generation. This process, implemented in [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py), transforms high-level model parameters into a self-contained byte stream that eliminates Python-level overhead during inference. The resulting binary layout enables the decoder to enforce grammar constraints while maintaining JIT compatibility.

## The Binary Compilation Pipeline

The compilation process converts PyTree model parameters into a byte-level grammar through three distinct serialization stages. Each stage produces a specific binary segment that collectively forms the complete model representation ready for constrained generation.

### Step 1: 120‑Byte Header Construction

The compilation begins with a fixed **120‑byte header** that encodes the complete model architecture in little-endian order. This header contains critical metadata including dimensions, token IDs, and configuration flags that provide the decoder with necessary context for interpreting subsequent data.

The header is generated using Python's `struct` module according to lines 12‑14:

```python
bytearray(struct.pack(_TK_HDR, …))

```

This header precedes all other data in the compiled binary, establishing the foundational layout that the constrained decoding engine relies upon to locate tensors and respect grammar boundaries.

### Step 2: Nameless Tensor Directory

Following the header, the compiler generates a **nameless tensor directory** where each entry occupies a fixed `REC_SIZE` bytes. This directory eliminates textual naming overhead by using a strict binary record format containing:

- `dtype` (u8): Data type identifier
- `ndim` (u8): Number of dimensions
- `padding` (u16): Alignment padding
- `shape` (4 × u32): Tensor dimensions
- `offset` (u64): Byte position in the file
- `byte length` (u64): Size of tensor data

As implemented at lines 33‑35 in [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py), these records enable the decoder to locate every tensor—including weights and embeddings—without string comparisons. The `_Tensor` objects construct these entries during export, creating a compact index that the inference engine consults during constrained generation.

### Step 3: 64‑Byte Aligned Tensor Packing

The final segment stores actual tensor data with **64‑byte alignment** to optimize memory access patterns. For quantized tensors, the compiler applies specialized packing strategies:

1. **Packed indices** stored as `out * in_pad * bits/8` bytes
2. **Per‑group L2 norms** appended for applicable quantization schemes
3. **Ternary tensor optimization** where four signed 2‑bit "crumbs" pack into a single byte (5‑bit representation)

The packing logic in `_packed_row_bytes` (lines 52‑55) and the ternary handling code (lines 62‑71) ensure the byte-level grammar minimizes storage while remaining fast to scan. This alignment and packing prove essential when the decoder must rapidly fetch specific weight tensors during grammar-constrained token generation.

## Runtime Constrained Decoding Execution

During **constrained decoding**, the generator reads the compiled byte array and consults the tensor directory to fetch appropriate weight tensors. The binary layout stores **grammar‑driven token masks** as part of the byte stream, allowing the decoder to enforce constraints without invoking Python-level logic.

According to [`needle/model/decode.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/decode.py), the `generate_cached` and `forward_cached` functions process this self-contained format directly, scanning the byte-level grammar to determine valid next tokens. Because the entire constraint system operates on the pre-compiled binary representation, the inference loop remains tight and compatible with JIT compilation.

## Exporting and Loading Compiled Models

The `export_model` function in [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py) orchestrates the compilation:

```python
from needle.model.export import export_model

# `params` is the PyTree of model weights, `config` holds architecture details

blob, tensor_count = export_model(params, config, path="my_model.bin")
print(f"Saved {tensor_count} tensors, total size {len(blob)} bytes")

```

To utilize the compiled grammar for constrained generation, load the binary using:

```python
from needle.model.export import load_model

model = load_model("my_model.bin")

# `model` now contains the header, directory, and packed tensors ready for

# `generate_cached` which respects any grammar constraints.

```

The `load_model` function reconstructs the tensor directory in memory, enabling fast lookups during generation. For quantized deployments, [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py) provides helper functions like `cq_model_bytes` and `model_bytes_fp16` that produce the packed byte streams consumed by the exporter.

## Summary

- **Header construction**: A fixed 120‑byte little-endian header encodes model architecture metadata at the start of the binary file in [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py).
- **Tensor directory**: Nameless records of `REC_SIZE` bytes map tensor locations without string overhead, using u8/u16/u32/u64 fields for dtype, shape, offset, and length.
- **Packed alignment**: Tensor data follows 64‑byte alignment with specialized packing for quantized weights, including ternary compression where four 2‑bit crumbs fit into one byte.
- **Runtime execution**: The `generate_cached` function in [`needle/model/decode.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/decode.py) reads the compiled byte stream directly to enforce grammar constraints during token generation without Python-level interpretation.

## Frequently Asked Questions

### What is the size of the header in Needle's compiled binary format?

The compiled binary format begins with a **fixed 120‑byte header** that encodes the complete model architecture. This header is created using `bytearray(struct.pack(_TK_HDR, …))` at lines 12‑14 of [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py) and stores dimensions, token IDs, and other critical metadata in little‑endian format.

### How does the tensor directory locate weights without names?

The tensor directory uses **nameless fixed‑size records** of `REC_SIZE` bytes that contain binary fields for `dtype` (u8), `ndim` (u8), `shape` (4 × u32), `offset` (u64), and `byte length` (u64). These numeric identifiers allow the decoder to jump directly to tensor data without parsing string names, as implemented in the `_Tensor` object construction at lines 33‑35.

### Why does Needle use 64‑byte alignment for tensor data?

Needle aligns tensor data to **64‑byte boundaries** to optimize memory access patterns and cache utilization during constrained decoding. This alignment ensures that weight tensors—even when quantized with complex packing schemes like ternary (four 2‑bit crumbs per byte)—remain efficiently accessible to the `generate_cached` function during high‑speed token generation.

### How are quantized tensors packed in the byte‑level grammar?

Quantized tensors follow a **packed layout** where indices are stored first as `out * in_pad * bits/8` bytes, followed by per‑group L2 norms when applicable. For ternary tensors, the compiler packs four signed 2‑bit values into each byte using bit‑manipulation logic in `_packed_row_bytes` (lines 52‑55 and 62‑71), significantly reducing the binary size while maintaining fast scan performance.