# Security Considerations for Loading Untrusted .tv and .tvim Files in TurboVec

> Safely load untrusted .tv and .tvim files in TurboVec with defense-in-depth validation including magic-byte checks version enforcement and overflow-protected allocation

- Repository: [Ryan Codrai/turbovec](https://github.com/RyanCodrai/turbovec)
- Tags: security
- Published: 2026-07-18

---

**TurboVec implements defense-in-depth validation—including magic-byte checks, version enforcement, and overflow-protected allocation—to safely load binary index files from untrusted sources without risking memory exhaustion or code execution.**

The RyanCodrai/turbovec repository stores persistent vector indexes in two binary containers: `.tv` files containing the core **TurboQuantIndex** and `.tvim` files containing the **IdMapIndex** with associated payload data. When loading untrusted `.tv` and `.tvim` files from third-party sources, the library enforces strict validation chains to prevent **magic-byte spoofing**, **version confusion**, **integer overflow**, and **uncontrolled memory allocation** attacks.

## Magic-Byte Validation and Format Fingerprinting

TurboVec begins every load operation by verifying the file header against expected magic strings. This prevents the loader from interpreting arbitrary binary data as a valid index.

For `.tv` files, the loader reads the first four bytes and expects the ASCII string `TVPI`. For `.tvim` files, the expected magic is `TVIM`. If the header does not match, the function immediately returns an `InvalidData` error.

In [`turbovec/src/io.rs`](https://github.com/RyanCodrai/turbovec/blob/main/turbovec/src/io.rs), the `.tv` loader performs this check at lines 70–72:

```rust
// io.rs lines 70-72
if magic != TV_MAGIC {
    return Err(Error::new(ErrorKind::InvalidData, "Invalid magic number"));
}

```

Similarly, the `.tvim` loader validates the header at lines 39–42 in the same file. This **magic-byte validation** ensures that only genuine TurboVec indexes proceed to parsing, blocking attempts to load executable code or corrupted data disguised as index files.

## Version Control and Legacy Compatibility

After magic verification, TurboVec enforces strict version checking to prevent **version-confusion attacks** and ensure backward compatibility without exposing unsafe code paths.

### Legacy v1 File Rejection

Files created before version 0.4.4 lack magic headers and use a different scalar encoding. If the first byte after the magic check falls in the range 2–4, the loader identifies the file as a legacy v1 format and aborts with a clear "rebuild required" message rather than silently misinterpreting the data. This logic appears at lines 73–86 in [`turbovec/src/io.rs`](https://github.com/RyanCodrai/turbovec/blob/main/turbovec/src/io.rs), where the code constructs an error containing the `REBUILD_HINT` constant:

```rust
// io.rs lines 81-86 (within legacy branch)
return Err(Error::new(
    ErrorKind::InvalidData,
    format!("Version 1 files are not supported. {}", REBUILD_HINT)
));

```

### v2 and v3 Forward Compatibility

For current formats, the loader reads the version byte at lines 93–96 and forwards it to `read_core_versioned`, which discriminates between v2 (without TQ+ calibration) and v3 (with TQ+ calibration) payloads. This guarantees that newer loaders can safely process older files without executing incompatible parsing logic.

## Integer Overflow Protection and Memory Safety

TurboVec implements **controlled allocation** strategies to prevent malicious files from triggering excessive memory consumption through attacker-controlled length fields.

When loading the `slot_to_id` table from a `.tvim` file, the code first computes the required byte count using `checked_mul` at lines 164–170 in [`turbovec/src/io.rs`](https://github.com/RyanCodrai/turbovec/blob/main/turbovec/src/io.rs):

```rust
// io.rs lines 164-170
let id_bytes = n_vectors
    .checked_mul(8)
    .ok_or_else(|| Error::new(ErrorKind::InvalidData, "Integer overflow"))?;

```

This **integer overflow** check ensures that a malformed file declaring an astronomically large `n_vectors` cannot force a massive allocation. The actual buffer allocation via `read_exact_vec` uses the exact computed byte count rather than a pre-reserved capacity, preventing denial-of-service attacks via memory exhaustion.

## Data Integrity and Side-Table Validation

To prevent corrupted writes from compromising future reads, TurboVec validates the relationship between the vector count and the ID mapping table before any I/O occurs.

When writing a `.tvim` file via `write_id_map`, the code asserts that the length of the `slot_to_id` slice equals the declared `n_vectors` value. This check at lines 110–115 in [`turbovec/src/io.rs`](https://github.com/RyanCodrai/turbovec/blob/main/turbovec/src/io.rs) enforces the invariant at serialization time:

```rust
// io.rs lines 110-115
assert_eq!(
    slot_to_id.len(),
    n_vectors as usize,
    "slot_to_id length must match n_vectors"
);

```

By verifying this constraint during the write operation, the library ensures that any subsequently loaded file maintains internal consistency between its header declarations and actual payload dimensions.

## Security Test Coverage

The repository includes comprehensive fuzz-style testing to verify resilience against malicious inputs. The Python test suite in [`turbovec-python/tests/test_security.py`](https://github.com/RyanCodrai/turbovec/blob/main/turbovec-python/tests/test_security.py) feeds deliberately corrupted `.tv` and `.tvim` payloads to the Rust library, asserting that the defensive checks raise the expected errors.

This test coverage validates that the **defense-in-depth** strategy—spanning magic checks, version validation, and arithmetic verification—remains functional across library updates.

## Safe Loading Patterns in Rust and Python

When integrating TurboVec into applications that process untrusted indexes, handle load operations through explicit error checking to capture validation failures.

### Loading in Rust

The following pattern safely handles a potentially malicious `.tv` file, catching `InvalidData` errors from any layer of the validation chain:

```rust
use turbovec::io::{self, load};

fn main() -> std::io::Result<()> {
    // Attempt to load an index; any malformed or malicious file will
    // return an `Err` with a clear message.
    match load("path/to/untrusted/index.tv") {
        Ok((bit_width, dim, n_vectors, codes, scales, shift, scale)) => {
            println!("Loaded {} vectors (dim={})", n_vectors, dim);
            // … safe to use the payload …
            Ok(())
        }
        Err(e) => {
            eprintln!("Failed to load index: {}", e);
            Err(e)
        }
    }
}

```

### Loading in Python

The Python bindings expose the same safety checks through the `load_id_map` function:

```python
import turbovec

def safe_load_tvim(path: str):
    try:
        idx = turbovec.load_id_map(path)   # Calls the Rust loader under the hood

        print(f"Loaded {idx.n_vectors} vectors with {len(idx.slot_to_id)} id mappings")
        return idx
    except Exception as exc:
        # The Rust side will raise a clear IOError if the file is malformed

        print(f"Error loading .tvim file: {exc}")
        raise

# Example usage

safe_load_tvim("downloads/model.tvim")

```

### Validating Write Operations

When generating indexes for distribution, enforce the side-table invariant to protect downstream consumers:

```rust
let slot_to_id: Vec<u64> = /* … generate mapping … */;
turbovec::io::write_id_map(
    "my_index.tvim",
    bit_width,
    dim,
    n_vectors,
    &packed_codes,
    &scales,
    &tqplus_shift,
    &tqplus_scale,
    &slot_to_id,
)?;

```

### Unit Testing Overflow Protection

To verify that your application correctly handles corrupted files, you can simulate an overflow attack in your test suite:

```rust
#[test]
fn reject_overflow_id_table() {
    // Simulate a corrupted `.tvim` with a huge `n_vectors` that would overflow usize
    let mut data = vec![];
    data.extend(b"TVIM");                // magic
    data.push(3u8);                      // version
    // Core header with n_vectors = u64::MAX / 8 + 1 (intentional overflow)
    data.extend(&[2u8]);                 // bit_width
    data.extend(&(128u32).to_le_bytes()); // dim
    data.extend(&(u64::MAX / 8 + 1u64).to_le_bytes()[..4]); // truncated n_vectors
    // ... rest omitted
    let tmp = std::env::temp_dir().join("overflow.tvim");
    std::fs::write(&tmp, data).unwrap();
    assert!(turbovec::io::load_id_map(&tmp).is_err());
}

```

## Summary

- **Magic-byte validation** (`TVPI`/`TVIM`) at file headers prevents spoofing attacks by rejecting non-TurboVec binaries before parsing begins.
- **Version enforcement** blocks legacy v1 files and safely discriminates between v2 and v3 formats via `read_core_versioned`, preventing version-confusion exploits.
- **Integer overflow protection** via `checked_mul` guards against memory exhaustion attacks when processing attacker-controlled vector counts.
- **Side-table validation** ensures `slot_to_id` length matches `n_vectors` during serialization, maintaining data integrity for subsequent loads.
- **Explicit error messages** containing `REBUILD_HINT` guide users toward secure remediation rather than dangerous workarounds.
- **Comprehensive test coverage** in [`test_security.py`](https://github.com/RyanCodrai/turbovec/blob/main/test_security.py) verifies that the Rust validation layers correctly reject malformed inputs from Python.

## Frequently Asked Questions

### What are the main security risks when loading untrusted .tv or .tvim files?

Loading binary files from untrusted sources exposes applications to **magic-byte spoofing** (where executable code masquerades as an index), **version confusion** (triggering undefined behavior in outdated parsers), and **denial-of-service** via integer overflow or massive memory allocation. TurboVec mitigates these through strict header validation, version checking, and `checked_mul` arithmetic on all length fields.

### How does TurboVec prevent memory exhaustion from malicious files?

The library uses **controlled allocation** patterns in [`turbovec/src/io.rs`](https://github.com/RyanCodrai/turbovec/blob/main/turbovec/src/io.rs). When loading the `slot_to_id` table, it computes `n_vectors.checked_mul(8)` at lines 164–170, returning an error if the multiplication overflows `usize`. This prevents a tiny file declaring billions of vectors from forcing a huge memory allocation.

### What should I do if TurboVec rejects a file with a version error?

If the loader returns an `InvalidData` error mentioning "Version 1 files are not supported" or an incompatible version number, the file must be regenerated from source vectors. The error message includes the `REBUILD_HINT` constant, directing you to recreate the index using the current TurboVec version rather than attempting unsafe migration of the binary format.

### Are there tests that verify these security checks work correctly?

Yes. The repository includes [`turbovec-python/tests/test_security.py`](https://github.com/RyanCodrai/turbovec/blob/main/turbovec-python/tests/test_security.py), which contains fuzz-style tests that deliberately corrupt magic bytes, version fields, and length parameters. These tests assert that the Rust library raises the expected errors, ensuring the defensive code remains functional across releases.