How to Use Turbovec for In-Memory Serialization and Deserialization

Turbovec provides symmetric to_bytes() and from_bytes() methods that serialize quantized vector indexes to an in-memory v7 binary format (.tv for TurboQuantIndex and .tvim for IdMapIndex), enabling deterministic, validated round-trip serialization without touching disk.

The RyanCodrai/turbovec library implements a high-performance vector quantization system that supports zero-copy in-memory serialization. This capability allows you to cache indexes in databases, transmit them over networks, or store them in memory-mapped buffers while maintaining full compatibility with the on-disk format.

Understanding the v7 Binary Format

Turbovec stores its index payload in a version-7 binary image format. When you serialize an index using to_bytes(), the library builds the complete v7 image using the same builder that powers file-based write() operations. Consequently, the byte stream produced in memory is byte-identical to what would be written to a .tv or .tvim file on disk.

This architecture guarantees three critical properties:

• Deterministic serialization: Repeated calls to to_bytes() on the same index produce identical byte sequences
• Full validation: The from_bytes() method applies the same rigorous validation as file-based load(), rejecting malformed or out-of-range values
• Format compatibility: In-memory payloads can be persisted to databases, Redis caches, or message queues without conversion overhead

Serialization with to_bytes()

The to_bytes() method constructs the complete binary representation of the index, including calibration state, codebook, and scale information.

Rust Implementation

In turbovec/src/lib.rs (lines 2167-2180), the Rust implementation forwards to_bytes to the internal v7_image builder:

use turbovec::TurboQuantIndex;

// Build an index
let idx = TurboQuantIndex::new_lazy(4).unwrap();

// Serialize to in-memory bytes
let payload: Vec<u8> = idx.to_bytes();

The method returns a Vec<u8> containing the complete binary image ready for transmission or storage.

Python Implementation

The Python API exposes the same functionality, returning a standard bytes object:

from turbovec import TurboQuantIndex

# Create and populate an index

idx = TurboQuantIndex(dim=1536, bit_width=4)
idx.add(vectors)  # NumPy array of shape (n, dim), dtype float32

# Serialize to bytes

payload = idx.to_bytes()  # Returns Python bytes object

Deserialization with from_bytes()

The from_bytes(data) method consumes a byte slice and reconstructs the index. According to the source code in turbovec/src/lib.rs, this delegates to load_from_reader, ensuring parity with file-based loading.

Validation and Safety

The core parsing logic resides in turbovec/src/io_v7.rs within the load_image function. This low-level loader performs strict validation on:

• Version compatibility markers
• Calibration state integrity
• Codebook dimension alignment
• Scale factor bounds

Any deviation from the expected format triggers an error rather than silent corruption.

// Rust: Deserialize with full validation
let restored = TurboQuantIndex::from_bytes(&payload).unwrap();

// Verify deterministic round-trip
assert_eq!(idx.to_bytes(), restored.to_bytes());

# Python: Reconstruct from bytes

restored = TurboQuantIndex.from_bytes(payload)

# Verify byte-identical reconstruction

assert idx.to_bytes() == restored.to_bytes()

Complete Round-Trip Examples

Here are production-ready patterns for both languages demonstrating the full lifecycle: build → serialize → deserialize → verify.

Rust Example

use turbovec::TurboQuantIndex;

fn main() {
    // Initialize index with 4-bit quantization
    let idx = TurboQuantIndex::new_lazy(4).unwrap();
    
    // Add vectors (omitted for brevity)...
    
    // Serialize to in-memory buffer
    let payload = idx.to_bytes();
    
    // Store in database or send over network...
    
    // Deserialize and reconstruct
    let restored = TurboQuantIndex::from_bytes(&payload).unwrap();
    
    // Verify integrity: restored index matches original exactly
    assert_eq!(idx.to_bytes(), restored.to_bytes());
}

Python Example

from turbovec import TurboQuantIndex
import numpy as np

# Create index

idx = TurboQuantIndex(dim=768, bit_width=4)

# Generate sample vectors

vectors = np.random.randn(1000, 768).astype('float32')
idx.add(vectors)

# In-memory serialization

payload = idx.to_bytes()

# Simulate network transmission or cache storage

# ...

# Reconstruct index from bytes

restored = TurboQuantIndex.from_bytes(payload)

# Validate round-trip: byte sequences match exactly

assert idx.to_bytes() == restored.to_bytes()

IdMapIndex Support

For indexes requiring stable ID mapping, the IdMapIndex variant provides identical serialization methods. According to turbovec/src/id_map.rs, the to_bytes and from_bytes methods follow the same implementation pattern as TurboQuantIndex, producing .tvim format payloads.

Replace the class name in the examples above:

// Rust
let idx = IdMapIndex::new_lazy(4).unwrap();
let payload = idx.to_bytes();
let restored = IdMapIndex::from_bytes(&payload).unwrap();

# Python

from turbovec import IdMapIndex

idx = IdMapIndex(dim=1536, bit_width=4)
idx.add(ids, vectors)  # ids: list of int, vectors: NumPy array

payload = idx.to_bytes()
restored = IdMapIndex.from_bytes(payload)

Summary

• Turbovec implements symmetric to_bytes() and from_bytes() methods in both Rust and Python APIs for zero-copy in-memory serialization.
• The v7 binary format (.tv for TurboQuantIndex, .tvim for IdMapIndex) produces byte-identical output whether targeting memory or disk.
• Validation occurs during deserialization through load_from_reader in turbovec/src/lib.rs and the load_image parser in turbovec/src/io_v7.rs, ensuring corrupted payloads cannot instantiate invalid indexes.
• Determinism is guaranteed: repeated serialization of the same index yields identical byte sequences, enabling reliable checksums and caching strategies.
• IdMapIndex supports the same serialization interface as defined in turbovec/src/id_map.rs, maintaining API consistency across index types.

Frequently Asked Questions

What is the difference between to_bytes() and the file-based write() method?

Both methods use the same underlying v7_image builder in turbovec/src/lib.rs, producing byte-identical output. The write() method streams this data to disk, while to_bytes() returns the buffer directly as a Vec<u8> (Rust) or bytes object (Python). This guarantees that an index saved via write() can be loaded via from_bytes() and vice versa.

Does the from_bytes() method validate the payload before reconstruction?

Yes. According to the implementation in turbovec/src/io_v7.rs, the from_bytes method delegates to load_image, which performs comprehensive validation of version headers, calibration state, codebook dimensions, and scale factors. Malformed or truncated payloads result in immediate errors rather than undefined behavior.

Can I serialize an IdMapIndex using the same approach as TurboQuantIndex?

Yes. The IdMapIndex class defined in turbovec/src/id_map.rs exposes identical to_bytes() and from_bytes() methods. The resulting payload uses the .tvim format variant, which includes the ID-to-vector mapping alongside the quantized data.

Is the serialized format stable across different Turbovec versions?

The current implementation writes version-7 binary images. While the library maintains backward compatibility for reading older formats, to_bytes() always produces the latest v7 specification. For long-term storage, pinning to a specific Turbovec version ensures format stability, as future versions may introduce new binary revisions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}