Security Considerations for Loading Untrusted .tv and .tvim Files in TurboVec

TurboVec implements defense-in-depth validation—including magic-byte checks, version enforcement, and overflow-protected allocation—to safely load binary index files from untrusted sources without risking memory exhaustion or code execution.

The RyanCodrai/turbovec repository stores persistent vector indexes in two binary containers: .tv files containing the core TurboQuantIndex and .tvim files containing the IdMapIndex with associated payload data. When loading untrusted .tv and .tvim files from third-party sources, the library enforces strict validation chains to prevent magic-byte spoofing, version confusion, integer overflow, and uncontrolled memory allocation attacks.

Magic-Byte Validation and Format Fingerprinting

TurboVec begins every load operation by verifying the file header against expected magic strings. This prevents the loader from interpreting arbitrary binary data as a valid index.

For .tv files, the loader reads the first four bytes and expects the ASCII string TVPI. For .tvim files, the expected magic is TVIM. If the header does not match, the function immediately returns an InvalidData error.

In turbovec/src/io.rs, the .tv loader performs this check at lines 70–72:

// io.rs lines 70-72
if magic != TV_MAGIC {
    return Err(Error::new(ErrorKind::InvalidData, "Invalid magic number"));
}

Similarly, the .tvim loader validates the header at lines 39–42 in the same file. This magic-byte validation ensures that only genuine TurboVec indexes proceed to parsing, blocking attempts to load executable code or corrupted data disguised as index files.

Version Control and Legacy Compatibility

After magic verification, TurboVec enforces strict version checking to prevent version-confusion attacks and ensure backward compatibility without exposing unsafe code paths.

Legacy v1 File Rejection

Files created before version 0.4.4 lack magic headers and use a different scalar encoding. If the first byte after the magic check falls in the range 2–4, the loader identifies the file as a legacy v1 format and aborts with a clear "rebuild required" message rather than silently misinterpreting the data. This logic appears at lines 73–86 in turbovec/src/io.rs, where the code constructs an error containing the REBUILD_HINT constant:

// io.rs lines 81-86 (within legacy branch)
return Err(Error::new(
    ErrorKind::InvalidData,
    format!("Version 1 files are not supported. {}", REBUILD_HINT)
));

v2 and v3 Forward Compatibility

For current formats, the loader reads the version byte at lines 93–96 and forwards it to read_core_versioned, which discriminates between v2 (without TQ+ calibration) and v3 (with TQ+ calibration) payloads. This guarantees that newer loaders can safely process older files without executing incompatible parsing logic.

Integer Overflow Protection and Memory Safety

TurboVec implements controlled allocation strategies to prevent malicious files from triggering excessive memory consumption through attacker-controlled length fields.

When loading the slot_to_id table from a .tvim file, the code first computes the required byte count using checked_mul at lines 164–170 in turbovec/src/io.rs:

// io.rs lines 164-170
let id_bytes = n_vectors
    .checked_mul(8)
    .ok_or_else(|| Error::new(ErrorKind::InvalidData, "Integer overflow"))?;

This integer overflow check ensures that a malformed file declaring an astronomically large n_vectors cannot force a massive allocation. The actual buffer allocation via read_exact_vec uses the exact computed byte count rather than a pre-reserved capacity, preventing denial-of-service attacks via memory exhaustion.

Data Integrity and Side-Table Validation

To prevent corrupted writes from compromising future reads, TurboVec validates the relationship between the vector count and the ID mapping table before any I/O occurs.

When writing a .tvim file via write_id_map, the code asserts that the length of the slot_to_id slice equals the declared n_vectors value. This check at lines 110–115 in turbovec/src/io.rs enforces the invariant at serialization time:

// io.rs lines 110-115
assert_eq!(
    slot_to_id.len(),
    n_vectors as usize,
    "slot_to_id length must match n_vectors"
);

By verifying this constraint during the write operation, the library ensures that any subsequently loaded file maintains internal consistency between its header declarations and actual payload dimensions.

Security Test Coverage

The repository includes comprehensive fuzz-style testing to verify resilience against malicious inputs. The Python test suite in turbovec-python/tests/test_security.py feeds deliberately corrupted .tv and .tvim payloads to the Rust library, asserting that the defensive checks raise the expected errors.

This test coverage validates that the defense-in-depth strategy—spanning magic checks, version validation, and arithmetic verification—remains functional across library updates.

Safe Loading Patterns in Rust and Python

When integrating TurboVec into applications that process untrusted indexes, handle load operations through explicit error checking to capture validation failures.

Loading in Rust

The following pattern safely handles a potentially malicious .tv file, catching InvalidData errors from any layer of the validation chain:

use turbovec::io::{self, load};

fn main() -> std::io::Result<()> {
    // Attempt to load an index; any malformed or malicious file will
    // return an `Err` with a clear message.
    match load("path/to/untrusted/index.tv") {
        Ok((bit_width, dim, n_vectors, codes, scales, shift, scale)) => {
            println!("Loaded {} vectors (dim={})", n_vectors, dim);
            // … safe to use the payload …
            Ok(())
        }
        Err(e) => {
            eprintln!("Failed to load index: {}", e);
            Err(e)
        }
    }
}

Loading in Python

The Python bindings expose the same safety checks through the load_id_map function:

import turbovec

def safe_load_tvim(path: str):
    try:
        idx = turbovec.load_id_map(path)   # Calls the Rust loader under the hood

        print(f"Loaded {idx.n_vectors} vectors with {len(idx.slot_to_id)} id mappings")
        return idx
    except Exception as exc:
        # The Rust side will raise a clear IOError if the file is malformed

        print(f"Error loading .tvim file: {exc}")
        raise

# Example usage

safe_load_tvim("downloads/model.tvim")

Validating Write Operations

When generating indexes for distribution, enforce the side-table invariant to protect downstream consumers:

let slot_to_id: Vec<u64> = /* … generate mapping … */;
turbovec::io::write_id_map(
    "my_index.tvim",
    bit_width,
    dim,
    n_vectors,
    &packed_codes,
    &scales,
    &tqplus_shift,
    &tqplus_scale,
    &slot_to_id,
)?;

Unit Testing Overflow Protection

To verify that your application correctly handles corrupted files, you can simulate an overflow attack in your test suite:

#[test]
fn reject_overflow_id_table() {
    // Simulate a corrupted `.tvim` with a huge `n_vectors` that would overflow usize
    let mut data = vec![];
    data.extend(b"TVIM");                // magic
    data.push(3u8);                      // version
    // Core header with n_vectors = u64::MAX / 8 + 1 (intentional overflow)
    data.extend(&[2u8]);                 // bit_width
    data.extend(&(128u32).to_le_bytes()); // dim
    data.extend(&(u64::MAX / 8 + 1u64).to_le_bytes()[..4]); // truncated n_vectors
    // ... rest omitted
    let tmp = std::env::temp_dir().join("overflow.tvim");
    std::fs::write(&tmp, data).unwrap();
    assert!(turbovec::io::load_id_map(&tmp).is_err());
}

Summary

• Magic-byte validation (TVPI/TVIM) at file headers prevents spoofing attacks by rejecting non-TurboVec binaries before parsing begins.
• Version enforcement blocks legacy v1 files and safely discriminates between v2 and v3 formats via read_core_versioned, preventing version-confusion exploits.
• Integer overflow protection via checked_mul guards against memory exhaustion attacks when processing attacker-controlled vector counts.
• Side-table validation ensures slot_to_id length matches n_vectors during serialization, maintaining data integrity for subsequent loads.
• Explicit error messages containing REBUILD_HINT guide users toward secure remediation rather than dangerous workarounds.
• Comprehensive test coverage in test_security.py verifies that the Rust validation layers correctly reject malformed inputs from Python.

Frequently Asked Questions

What are the main security risks when loading untrusted .tv or .tvim files?

Loading binary files from untrusted sources exposes applications to magic-byte spoofing (where executable code masquerades as an index), version confusion (triggering undefined behavior in outdated parsers), and denial-of-service via integer overflow or massive memory allocation. TurboVec mitigates these through strict header validation, version checking, and checked_mul arithmetic on all length fields.

How does TurboVec prevent memory exhaustion from malicious files?

The library uses controlled allocation patterns in turbovec/src/io.rs. When loading the slot_to_id table, it computes n_vectors.checked_mul(8) at lines 164–170, returning an error if the multiplication overflows usize. This prevents a tiny file declaring billions of vectors from forcing a huge memory allocation.

What should I do if TurboVec rejects a file with a version error?

If the loader returns an InvalidData error mentioning "Version 1 files are not supported" or an incompatible version number, the file must be regenerated from source vectors. The error message includes the REBUILD_HINT constant, directing you to recreate the index using the current TurboVec version rather than attempting unsafe migration of the binary format.

Are there tests that verify these security checks work correctly?

Yes. The repository includes turbovec-python/tests/test_security.py, which contains fuzz-style tests that deliberately corrupt magic bytes, version fields, and length parameters. These tests assert that the Rust library raises the expected errors, ensuring the defensive code remains functional across releases.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}