turbovec MAX_DIM Limits and Dimension Requirements: The 16384 Cap and Multiple-of-8 Rule

TurboVec hard-codes MAX_DIM at 16384 and requires every vector dimension to be a positive multiple of 8, enforcing both rules during construction, first insertion, and file loading to prevent memory overruns and storage corruption.

The RyanCodrai/turbovec repository imposes strict bounds on vector dimensionality to protect against excessive memory allocation and to preserve its packed bit-plane layout. Understanding these turbovec MAX_DIM limits and dimension requirements is critical before you create or load an index, because any violation immediately returns a descriptive, hard error.

What Is the MAX_DIM Limit in turbovec?

TurboVec sets the constant MAX_DIM to 16384 in its source. This ceiling exists because a rotation matrix for higher dimensions would consume approximately 2 GiB just for f64 values, a footprint the library is explicitly designed to avoid. Any attempt to instantiate or grow an index beyond this boundary returns ConstructError::DimTooLarge or AddError::DimTooLarge, depending on the entry point.

Why Dimensions Must Be a Multiple of 8

TurboVec stores coordinates in a packed bit-plane layout that packs data into blocks of 8 bits. As implemented in turbovec/src/pack.rs, this format assumes whole-byte alignment for every dimension. If a dimension were not divisible by 8, the layout would shift and corrupt the index. Therefore, the library treats any concrete dimensionality that fails a modulus check as a fatal construction error.

How turbovec Enforces MAX_DIM Limits and Dimension Requirements

Construction in TurboQuantIndex::new

When an index is instantiated, TurboQuantIndex::new in turbovec/src/lib.rs (lines 82–84) verifies that the supplied dim is a positive multiple of 8. If dim % 8 != 0, the function returns ConstructError::DimNotPositiveMultipleOf8, defined in turbovec/src/error.rs at lines 15–17.

At lines 229–230 of lib.rs, the constructor also compares dim against the MAX_DIM constant. Any request exceeding 16384 dimensions returns ConstructError::DimTooLarge, whose definition resides in error.rs at lines 18–20.

Lazy-First-Add in TurboQuantIndex::add_2d

The add_2d method in turbovec/src/lib.rs (lines 1000–1004) repeats the multiple-of-8 validation when the first vectors are actually committed to a lazy index. If the committed dimension is not divisible by 8, the library raises AddError::DimNotMultipleOf8 before any data is written.

File Loading in validate_header_fields

For persisted indexes, the validate_header_fields helper in turbovec/src/io.rs (lines 762–764) inspects the file header. It rejects any dimension that is not a multiple of 8, ensuring that on-disk layouts remain compatible with the packed bit-plane representation.

Practical Examples: Valid and Invalid Dimensions

use turbovec::{TurboQuantIndex, MAX_DIM};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // ✅ Valid construction – dim 4096 (multiple of 8, below MAX_DIM)
    let mut idx = TurboQuantIndex::new(4096, 4)?;
    // Add some dummy vectors (here 2 vectors)
    idx.add_2d(&vec![0.0_f32; 2 * 4096], 4096)?;

    // ❌ Dimension not a multiple of 8 → error
    let err = TurboQuantIndex::new(4100, 4).unwrap_err();
    println!("{}", err); // prints "dim must be a positive multiple of 8, got 4100"

    // ❌ Dimension exceeds MAX_DIM → error
    let err = TurboQuantIndex::new(MAX_DIM + 8, 4).unwrap_err();
    println!("{}", err); // prints "dim 16392 exceeds maximum 16384"
    Ok(())
}

Running this example produces three clear outcomes:

• dim = 4096: Index created and vectors added successfully.
• dim = 4100: ConstructError::DimNotPositiveMultipleOf8 is returned.
• dim = 16392: ConstructError::DimTooLarge is returned.

Summary

• MAX_DIM is 16384: TurboVec refuses to create or load an index whose dimension exceeds this hard memory-safety limit.
• Multiple-of-8 rule: Every dimension must be a positive multiple of 8 to maintain the packed bit-plane layout used in pack.rs.
• Three enforcement layers: The rules are checked in TurboQuantIndex::new, TurboQuantIndex::add_2d, and validate_header_fields in io.rs.
• Clear error types: Violations surface as ConstructError::DimNotPositiveMultipleOf8, AddError::DimNotMultipleOf8, or ConstructError::DimTooLarge.

Frequently Asked Questions

What happens if I pass a dimension that is not a multiple of 8?

TurboVec immediately returns ConstructError::DimNotPositiveMultipleOf8 from TurboQuantIndex::new or AddError::DimNotMultipleOf8 from add_2d. The error message includes the offending dimension, and the index is neither created nor modified.

Can I increase the MAX_DIM constant beyond 16384?

No. The MAX_DIM value is hard-coded in the RyanCodrai/turbovec source to prevent accidental allocation of rotation matrices larger than approximately 2 GiB. Changing it would require forking the repository and recompiling, which is discouraged unless you fully understand the memory implications.

Why does the multiple-of-8 requirement exist?

The requirement exists because TurboVec uses a packed bit-plane storage format where each coordinate is stored in blocks of 8 bits. Dimensions that are not divisible by 8 would break this alignment and corrupt the index layout. This design choice optimizes storage density at the cost of enforcing strict dimension alignment.

Are dimension checks performed when loading a saved index?

Yes. The validate_header_fields function in turbovec/src/io.rs (lines 762–764) inspects the header of every .tv or .tvim file. If the declared dimension is not a multiple of 8 or exceeds MAX_DIM, the loader rejects the file before reading any vector data.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}