# How the TQ+ Calibration Trailer Enables Online Calibration in Turbovec

> Discover how the TQ+ calibration trailer enables online calibration in Turbovec by storing parameters for immediate reuse, eliminating offline retraining and optimizing workflow.

- Repository: [Ryan Codrai/turbovec](https://github.com/RyanCodrai/turbovec)
- Tags: internals
- Published: 2026-06-10

---

**The TQ+ calibration trailer stores per-coordinate shift and scale parameters at the end of the index file, allowing Turbovec to fit calibration on the first batch of vectors and automatically reuse it across process restarts without offline retraining.**

TurboQuant+ (TQ+) is a quantization method implemented in the `RyanCodrai/turbovec` repository that learns per-coordinate shift and scale parameters to map rotated coordinates onto a canonical Beta distribution. Unlike traditional quantization approaches requiring offline calibration datasets, Turbovec's TQ+ calibration trailer enables online calibration by persisting these parameters directly in the index file format. This design allows the system to fit calibration on the first batch of added vectors and seamlessly reuse those parameters for all future additions and queries.

## What Is TQ+ Calibration?

TQ+ improves upon standard quantization by learning **per-coordinate shift and scale** parameters that map the empirical distribution of each rotated coordinate onto the canonical Beta-marginal used to train the Lloyd-Max codebook. This coordinate-wise transformation ensures that the quantized values better match the expected distribution, reducing quantization error. In [`turbovec/src/encode.rs`](https://github.com/RyanCodrai/turbovec/blob/main/turbovec/src/encode.rs), the `compute_tqplus_calibration` function (lines 36-45) fits these parameters by analyzing the first batch of vectors added to the index.

## How the TQ+ Calibration Trailer Works

The online calibration capability relies on three complementary mechanisms baked into the indexing pipeline and file format:

### Fitting Calibration on First Add

When a batch of vectors is added for the first time, `TurboQuantIndex::add` calls `encode::encode` with `existing_calibration = None`. Inside [`encode.rs`](https://github.com/RyanCodrai/turbovec/blob/main/encode.rs) (lines 36-45), the `compute_tqplus_calibration` function fits a fresh `(shift, scale)` pair per dimension. The returned vectors are stored in the index fields `tqplus_shift` and `tqplus_scale` (defined in [`turbovec/src/lib.rs`](https://github.com/RyanCodrai/turbovec/blob/main/turbovec/src/lib.rs), lines 107-112).

### Persisting Parameters in the Trailer

The fitted calibration survives process restarts through a dedicated trailer appended to the index file. In [`turbovec/src/io.rs`](https://github.com/RyanCodrai/turbovec/blob/main/turbovec/src/io.rs), the `write_core` function (lines 97-112) appends the **TQ+ trailer** after the per-vector scales, writing the length (`n_calib`) followed by the two float arrays containing the shift and scale values. This binary format ensures the calibration travels with the index data on disk.

### Reusing Calibration on Subsequent Adds

When a file is later loaded, `io::read_core_v3` (lines 44-58) reads the trailer and restores the same calibration vectors into memory. On any subsequent call to `add`, the index detects that `tqplus_shift` is non-empty and constructs `existing = Some((shift, scale))` (as seen in [`lib.rs`](https://github.com/RyanCodrai/turbovec/blob/main/lib.rs), lines 66-73). The encoder then **locks** the calibration (lines 89-95 in [`encode.rs`](https://github.com/RyanCodrai/turbovec/blob/main/encode.rs)), ensuring that all vectors—both old and new—share the same coordinate system without refitting.

## The Binary Trailer Format

The TQ+ calibration trailer occupies the end of the Turbovec index file. According to the source code in [`io.rs`](https://github.com/RyanCodrai/turbovec/blob/main/io.rs), the trailer consists of:

- A length field (`n_calib`) indicating the number of calibrated dimensions
- A float array for the shift parameters
- A float array for the scale parameters

This append-only design allows the calibration metadata to be written atomically with the index data and read back efficiently when the index is loaded via `TurboQuantIndex::load`.

## Practical Implementation

The following Rust code demonstrates the complete workflow, from initial creation through persistence and reuse:

```rust
// 1️⃣ Create a new index (lazy dim) and add the first batch
let mut idx = TurboQuantIndex::new_lazy(4).unwrap(); // 4‑bit quantization
idx.add_2d(&first_batch, dim).unwrap(); // fits TQ+ calibration internally

// 2️⃣ Save the index – the calibration trailer is written automatically
idx.write("my_index.tv").unwrap();

// 3️⃣ Load the index later – calibration is restored from the trailer
let mut loaded = TurboQuantIndex::load("my_index.tv").unwrap();

// 4️⃣ Add more vectors – the same calibration is reused (no re‑fit)
loaded.add_2d(&second_batch, dim).unwrap();

// 5️⃣ Search – query vectors are un‑calibrated on the fly, using the stored shift/scale
let results = loaded.search(&queries, 10);

```

Because the calibration vectors travel with the index on disk, as implemented in `RyanCodrai/turbovec`, a freshly loaded index automatically knows the exact `(shift, scale)` that was used when the vectors were originally encoded. This eliminates any need for an offline "re-train" step and lets the system **calibrate online**: the moment the first vectors are added, the calibration is created, saved, and later reused without recomputation.

## Summary

- **Online fitting**: The `compute_tqplus_calibration` function in [`encode.rs`](https://github.com/RyanCodrai/turbovec/blob/main/encode.rs) fits shift and scale parameters during the first call to `add` when no existing calibration is present.
- **Persistent storage**: The TQ+ calibration trailer written by `io::write_core` (lines 97-112) stores these parameters at the end of the index file.
- **Automatic restoration**: `io::read_core_v3` (lines 44-58) reads the trailer when loading an index, restoring calibration into `tqplus_shift` and `tqplus_scale`.
- **Locked reuse**: Subsequent calls to `add` detect existing calibration in [`lib.rs`](https://github.com/RyanCodrai/turbovec/blob/main/lib.rs) (lines 66-73) and lock it during encoding (lines 89-95), ensuring all vectors share the same coordinate system.
- **Process independence**: The trailer enables online calibration across process restarts without requiring access to the original calibration dataset.

## Frequently Asked Questions

### Where is the TQ+ calibration trailer stored in the index file?

The trailer is appended at the very end of the file after the per-vector scales. In [`turbovec/src/io.rs`](https://github.com/RyanCodrai/turbovec/blob/main/turbovec/src/io.rs), the `write_core` function writes the length field followed by the shift and scale float arrays (lines 97-112), making it the final component of the binary format.

### What happens if I add vectors to an existing index that already has calibration?

The index checks if `tqplus_shift` is non-empty in [`lib.rs`](https://github.com/RyanCodrai/turbovec/blob/main/lib.rs) (lines 66-73). If calibration exists, it constructs `existing = Some((shift, scale))` and passes it to the encoder. The encoder then locks these parameters (lines 89-95 in [`encode.rs`](https://github.com/RyanCodrai/turbovec/blob/main/encode.rs)) and applies them to new vectors without refitting, ensuring consistency with previously stored vectors.

### Does Turbovec require an offline calibration dataset?

No. Turbovec performs **online calibration** by fitting the shift and scale parameters on the first batch of vectors added to the index. The `compute_tqplus_calibration` function in [`encode.rs`](https://github.com/RyanCodrai/turbovec/blob/main/encode.rs) analyzes the empirical distribution of the first `add` operation and stores the results in the TQ+ trailer for all future use.

### How does the trailer ensure calibration survives process restarts?

When `TurboQuantIndex::load` is called, `io::read_core_v3` (lines 44-58) reads the TQ+ trailer from the end of the file and restores the shift and scale vectors into the index instance. This allows the loaded index to immediately use the same calibration parameters that were fitted during the initial creation, without requiring access to the original training data.