TurboQuantIndex vs IdMapIndex: What’s the Difference in Turbovec?

Turbovec’s TurboQuantIndex stores and searches quantized vectors by internal slot numbers, while IdMapIndex wraps that index to add stable external ID mapping, letting you work with user-defined identifiers that persist across mutating operations.

Both index types are part of the turbovec repository and share the same underlying quantization engine. The key difference lies in how you reference vectors: TurboQuantIndex exposes raw slot numbers, whereas IdMapIndex maintains an ID‑to‑slot map that keeps your external IDs stable when the index is updated or persisted.

Overview of Turbovec’s Two Index Types

Turbovec is a Rust‐based vector search library that stores vectors in a compact, block‑wise quantized format. As shown in turbovec/src/lib.rs, the two primary index types are:

• TurboQuantIndex — the core approximate‑nearest‑neighbor (ANN) index. It manages vectors directly by their internal slot numbers and supports add, search, delete, sync, and serialization.
• IdMapIndex — a wrapper around TurboQuantIndex that adds a mapping from external IDs (e.g., u64) to internal slots. This layer is what makes user‑facing identifiers stable over time.

Both types are defined in the same source file—turbovec/src/lib.rs around line 295—and share methods like add, search, write, and load. The major difference is the API surface and the persisted data.

TurboQuantIndex: Direct Slot‑Based Vector Search

TurboQuantIndex is the foundational structure. It handles the low‑level quantized storage and calculates distances using SIMD instructions when possible. When you add a vector, you receive an integer slot that identifies that vector inside the index. This slot is only meaningful while the index is unchanged; operations like deletion or re‑ordering can invalidate slot numbers.

Here’s how it appears in practice:

use turbovec::TurboQuantIndex;

let mut qidx = TurboQuantIndex::new(128, 4).unwrap();
let slot = qidx.add(&vec![0.1_f32; 128]).unwrap();
let results = qidx.search(&vec![0.1_f32; 128], 5).unwrap();
println!("Nearest slots: {:?}", results.indices);

This direct interface is ideal for low‑overhead experimentation or when you, the caller, already manage your own vector IDs externally.

IdMapIndex: Stable External ID Mapping

IdMapIndex is designed for production scenarios where you need to refer to vectors by a stable identifier—one that remains unchanged even if the index internally rearranges its vector blocks. When you add a vector, you supply an ID (for example, a u64), and the index stores the mapping between that ID and the actual slot. Later searches return those IDs instead of raw slots.

use turbovec::IdMapIndex;

let mut iidx = IdMapIndex::new(128, 4).unwrap();
iidx.add(42u64, &vec![0.1_f32; 128]).unwrap();
let results = iidx.search(&vec![0.1_f32; 128], 5).unwrap();
println!("Nearest IDs: {:?}", results.ids);

The ID map is kept in memory and is serialized to disk when the index is written, guaranteeing that your external identifiers stay in sync with the vector data across write/load cycles.

Key Differences at a Glance

Feature	TurboQuantIndex	IdMapIndex
Core reference	Internal slot numbers returned by add	User‑supplied stable IDs (e.g., u64)
Data layout	Magic header TV7\0 followed by quantized blocks	Same block data plus a serialized ID‑to‑slot map
Public API	add(vec) returns slot; search returns slot indices	add(id, vec); search returns user IDs
Use case	Prototyping, or when you control ID assignment externally	Persisting a retrieval system that needs stable keys
Implementation	Core lib.rs struct (~line 295)	Wrapper struct in the same lib.rs module

Both types share the same block‑wise quantization scheme, SIMD distance calculations, and optional calibration. The only real functional delta is the ID mapping layer.

Code Examples: Using Both Index Types

The following examples show the full lifecycle for each index, including persisting to disk:

use turbovec::{TurboQuantIndex, IdMapIndex};

// ---------- TurboQuantIndex ----------
let mut qidx = TurboQuantIndex::new(128, 4).unwrap();
let slot = qidx.add(&vec![0.1_f32; 128]).unwrap();
let results = qidx.search(&vec![0.1_f32; 128], 5).unwrap();
println!("Nearest slots: {:?}", results.indices);

qidx.write("my_quant.idx").unwrap();
let loaded_q = TurboQuantIndex::load("my_quant.idx").unwrap();

// ---------- IdMapIndex ----------
let mut iidx = IdMapIndex::new(128, 4).unwrap();
iidx.add(42u64, &vec![0.1_f32; 128]).unwrap();
let results = iidx.search(&vec![0.1_f32; 128], 5).unwrap();
println!("Nearest IDs: {:?}", results.ids);

iidx.write("my_idmap.idx").unwrap();
let loaded_i = IdMapIndex::load("my_idmap.idx").unwrap();

Note that both write and load methods return Result<(), Box<dyn Error>>; the .unwrap() here is for brevity—production code should handle the errors explicitly.

Where Each Index Shines

• TurboQuantIndex is the better choice when you are building a quick prototype or when you already hold your own identifier scheme separate from the index. Slot numbers are efficient and avoid the overhead of a mapping table.
• IdMapIndex is recommended for long‑running services or databases that need to add and remove vectors while keeping a stable external ID. For example, an e‑commerce search backend can store product IDs as keys, and the index stays consistent across inserting new products or removing old ones.

Because IdMapIndex wraps the same TurboQuantIndex engine inside, you get all the performance benefits of the quantized storage with the extra lifecycle safety of stable IDs.

Summary

• TurboQuantIndex gives you raw slot‑based access to quantized vectors—fast and low‑overhead.
• IdMapIndex adds a persistent mapping from external IDs to slots, making it ideal for production use cases requiring stable references.
• Both types share the same quantized block format and remain defined in turbovec/src/lib.rs (structs begin around line 295).
• The persistence files (write/load) behave identically except that IdMapIndex serializes the ID map alongside the vector blocks.

Frequently Asked Questions

Can a TurboQuantIndex be converted to an IdMapIndex?

There is no direct in‑memory conversion method in the public API. You would need to add all vectors into a new IdMapIndex using your own IDs, or load the TurboQuantIndex and re‑insert them with a chosen ID scheme.

Do both index types use the same quantized storage format?

Yes. Both TurboQuantIndex and IdMapIndex use the same block‑wise, quantized vector storage layout in their underlying engine, including the same magic header and SIMD‑accelerated distance calculations.

Are add and search methods safe to call concurrently?

The public API as of the current source is not thread‑safe. You must guard the index with a mutex or another synchronization primitive if multiple threads need to call add, remove, or search on the same instance.

Which index type is better for a large‑scale deployment?

IdMapIndex is generally recommended for production because it decouples the internal slot layout from the external identifiers. That way, even if the index reorders slots internally (e.g., after deletions), your application’s references remain valid. TurboQuantIndex remains ideal for lightweight, single‑threaded experiments where slot numbers are already under the caller’s control.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}