How Turbovec's Rotation Fingerprint Prevents Recall Loss When Loading Indices

Turbovec's rotation fingerprint detects bit-level mismatches in rebuilt orthogonal rotation matrices at load time, forcing an explicit load failure instead of silently corrupting search results and crashing recall.

The RyanCodrai/turbovec vector search engine uses deterministic orthogonal rotation matrices to transform vector indices in its v4-format. Because these matrices are rebuilt from a fixed ROTATION_SEED when an index is loaded, subtle environmental differences can introduce bit-level variations that devastate search accuracy. The rotation fingerprint is the safeguard that validates every rebuilt matrix before it is used, ensuring that Turbovec either recalls exactly what was indexed or fails fast with no results.

The Problem: Why Rotation Matrices Drift Across Environments

Turbovec stores a deterministic orthogonal rotation matrix with every vector index. At load time, the matrix is regenerated from a fixed seed using QR decomposition. However, the exact output of a QR decomposition is not guaranteed to be bit-identical across different CPUs, thread counts, or underlying library versions. If the loader naïvely assumed the rebuilt matrix matched the original exactly, a mismatched rotation would corrupt inner-product calculations and drive recall to near zero.

What the Rotation Fingerprint Stores

Instead of trusting the rebuilt matrix, Turbovec writes a rotation fingerprint into the index file header. This fingerprint captures two complementary views of the matrix that together detect both exact mismatches and harmless floating-point drift.

The Hash Component

The fingerprint contains a 64-bit FNV-1a hash computed over the exact bit patterns of the entire rotation matrix in row-major, little-endian order. In turbovec/src/rotation.rs (lines 81–84), this hash is generated when the index is saved. If the hash of the rebuilt matrix equals the stored hash, Turbovec can instantly confirm bit-identical reconstruction.

The Probes Component

When the hash differs, the fingerprint falls back to 64 sampled f32 values taken from deterministic positions in the matrix. These positions are generated by probe_positions. As shown in lines 85–88 of rotation.rs, these probes allow a tolerant comparison that can accept the harmless floating-point noise introduced by cross-environment QR variations.

How Fingerprint Verification Works at Load Time

When an index file is opened, Turbovec runs a strict four-step verification pipeline before any vectors are searched. This ensures the rebuilt rotation matches the original matrix that was used to construct the index.

1. Rebuild the rotation matrix from the stored dimension using make_rotation_matrix(dim).
2. Compute a fresh fingerprint via RotationFingerprint::compute(&rot, dim).
3. Load the stored fingerprint from the file header.
4. Compare them with stored_fp.matches(&rebuilt_fp).

The matches logic in turbovec/src/rotation.rs (lines 31–38) first checks the 64-bit FNV-1a hash. If the hashes match, verification passes immediately. Otherwise, every stored probe must be within PROBE_TOLERANCE = 1e-4 of its rebuilt counterpart. This tolerance is intentionally far larger than the typical one-ULP differences that arise from cross-environment QR variations, as discussed in lines 59–68 of rotation.rs.

Preventing Recall Loss Through Explicit Failure

If stored_fp.matches(&rebuilt_fp) returns false, Turbovec treats the rotation as invalid and fails the load early. Rather than silently using a mismatched matrix and returning corrupted search results packed with false positives and negatives, the loader returns an empty result set or an error. This creates a semantic guarantee that loading an index either works exactly as originally written or yields no results.

This explicit failure mode is what prevents recall loss. A wrong rotation would cause a dramatic drop in recall, often to near zero. By catching matrix drift before any search occurs, the rotation fingerprint protects Turbovec's recall guarantees across library updates, hardware changes, and differing thread counts.

Loading and Verifying an Index in Practice

The following Rust pattern shows how the fingerprint check fits into the load path. The verification happens after the header is read but before the searcher is instantiated.

let idx = turbovec::Index::open("my_index.tvim")?;   // reads header
let dim = idx.dim();                                 // matrix dimension
let stored_fp = idx.rotation_fingerprint();          // fingerprint from file

// Rebuild the rotation matrix from the seed
let rebuilt_rot = turbovec::rotation::make_rotation_matrix(dim);
let rebuilt_fp = turbovec::rotation::RotationFingerprint::compute(&rebuilt_rot, dim);

// Verify fingerprint
if !stored_fp.matches(&rebuilt_fp) {
    // Rotation drift detected – abort or return empty search results
    return Err(turbovec::Error::RotationMismatch);
}

// Safe to proceed: the rebuilt rotation is verified
let searcher = idx.searcher();  // uses the verified rotation

Core Files in the Fingerprint Pipeline

Three source files implement this safety net:

• turbovec/src/rotation.rs — Implements deterministic rotation generation (make_rotation_matrix), fingerprint calculation (RotationFingerprint::compute), and the tolerant matching logic (matches).
• turbovec/src/io.rs — Handles reading and writing the index header, including serializing and deserializing the stored fingerprint.
• turbovec/src/lib.rs — Exposes rotation_fingerprint() on the index and integrates fingerprint verification into the standard load path.

Summary

• Turbovec rebuilds an orthogonal rotation matrix from a fixed seed every time an index is loaded.
• QR decomposition can produce non-bit-identical results across environments, which would silently destroy recall if left unchecked.
• The rotation fingerprint stores a 64-bit FNV-1a hash and 64 deterministic probes to validate the rebuilt matrix.
• Verification logic in rotation.rs uses exact hashing first, then falls back to a PROBE_TOLERANCE of 1e-4 for tolerant comparison.
• If validation fails, Turbovec aborts the load rather than returning corrupted results, preserving the guarantee that recall is never silently degraded.

Frequently Asked Questions

What causes rotation matrix drift in Turbovec indices?

The drift stems from the QR decomposition used to generate the orthogonal rotation matrix in make_rotation_matrix. Minor differences in CPU architectures, thread counts, or underlying linear-algebra library versions can produce mathematically equivalent but bitwise different results. Without the fingerprint, Turbovec would not detect this divergence.

What is PROBE_TOLERANCE and why is it set to 1e-4?

PROBE_TOLERANCE is a constant defined in turbovec/src/rotation.rs that sets the maximum acceptable absolute difference between stored and rebuilt probe values. It is set to 1e-4 because that threshold is far larger than typical one-ULP floating-point noise from cross-environment QR variations, yet small enough to catch genuinely corrupted matrices.

What happens if the rotation fingerprint fails verification during load?

If stored_fp.matches(&rebuilt_fp) returns false, Turbovec flags the rotation as invalid and fails the load early. According to the source implementation, this prevents the engine from silently using a mismatched matrix, ensuring it returns zero results or an error instead of corrupted vectors that would crash recall.

Where is the rotation fingerprint stored in a Turbovec index file?

The fingerprint is stored in the index file header. The io.rs module handles persisting the fingerprint when the index is saved and retrieving it when Index::open is called, making it available to the verification routine before any search operations begin.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}