# How to Migrate from FAISS IndexPQ to turbovec: A Complete Guide

> Easily migrate from FAISS IndexPQ to turbovec. Learn how to replace FAISS with TurboQuantIndex or IdMapIndex, skip training, and leverage built-in filtering for higher compression and performance.

- Repository: [Ryan Codrai/turbovec](https://github.com/RyanCodrai/turbovec)
- Tags: migration-guide
- Published: 2026-06-08

---

**You can migrate from FAISS IndexPQ to turbovec by replacing `faiss.IndexPQ` with `TurboQuantIndex` or `IdMapIndex`, eliminating the training step, and using the same `add()` and `search()` patterns while gaining built-in filtering and higher compression.**

Migrating from FAISS IndexPQ to turbovec simplifies vector search pipelines by removing the k-means training requirement and adding SIMD-optimized kernels. According to the RyanCodrai/turbovec source code, the library implements a training-free quantization pipeline in [`turbovec/src/encode.rs`](https://github.com/RyanCodrai/turbovec/blob/main/turbovec/src/encode.rs) and accelerated search in [`turbovec/src/search.rs`](https://github.com/RyanCodrai/turbovec/blob/main/turbovec/src/search.rs), making it a drop-in alternative for CPU-based approximate nearest neighbor search.

## Why Switch from FAISS IndexPQ to turbovec?

turbovec's `TurboQuant` index replaces FAISS product quantization with a data-oblivious random rotation and a pre-computed Lloyd-Max codebook. This architectural shift eliminates the separate training phase required by FAISS IndexPQ while delivering higher compression ratios and native hybrid filtering.

### Core Architectural Differences

| Feature | FAISS IndexPQ | turbovec (TurboQuant) |
|---------|---------------|----------------------|
| **Training** | Requires k-means training on learn vectors. | No training needed. |
| **Memory** | ~4× size with 8-bit PQ. | 16× compression at 2-bit; 8× at 4-bit. |
| **Speed** | FastScan AVX-512 kernels decode codebooks. | NEON and AVX-512BW kernels match or beat FastScan by 12-20% on ARM. |
| **Online Ingest** | May require rebuild after training. | Vectors index on the fly via `add()`. |
| **External IDs** | Returns internal IDs only. | `IdMapIndex` stores stable `uint64` IDs with O(1) `remove()`. |
| **Filtering** | Manual post-filtering required. | Kernel respects `allowlist` mask during search. |

### Encoding and Search Implementation

The encoding pipeline lives in [`turbovec/src/encode.rs`](https://github.com/RyanCodrai/turbovec/blob/main/turbovec/src/encode.rs) and executes **normalize → rotate → TQ+ calibration → Lloyd-Max quantization → bit-pack**. The per-vector **scale** (`norm / ⟨u, x̂⟩`) is stored as an `f32` for unbiased inner-product scoring, as implemented in lines 1-53 of that file.

Search kernels are defined in [`turbovec/src/search.rs`](https://github.com/RyanCodrai/turbovec/blob/main/turbovec/src/search.rs). The library provides hand-written **NEON** (`score_4bit_block_neon`) and **x86** (`search_multi_query_avx2`, `search_multi_query_avx512bw`) routines that operate directly on packed codes and apply per-vector scales at query time. The `block_has_allowed` logic in lines 13-21 enables early-exit masking, which lets the kernel guarantee that only allowed IDs are scored without extra post-processing.

## Step-by-Step Migration from FAISS IndexPQ to turbovec

### Install turbovec

Add the library to your project using `pip` or Cargo.

```bash
pip install turbovec

```

```toml
[dependencies]
turbovec = "0.1"

```

```rust
use turbovec::{TurboQuantIndex, IdMapIndex};

```

### Replace Index Creation and Remove Training

In FAISS, you must instantiate `IndexPQ`, define the number of sub-quantizers `M`, call `train()`, and then `add()`. In turbovec, you instantiate `TurboQuantIndex` with a **bit width** and call `add()` immediately.

```python

# FAISS IndexPQ

import faiss
d = 1536
M = d // 8
index = faiss.IndexPQ(d, M, 8)
index.train(train_vectors)  # Required

index.add(db_vectors)

```

```python

# turbovec

from turbovec import TurboQuantIndex
index = TurboQuantIndex(dim=d, bit_width=4)  # 4-bit = 16 centroids

index.add(db_vectors)                        # No training step

```

### Add Vectors and Search

The `add()` and `search()` call signatures mirror FAISS. Pass a NumPy array of `float32` vectors and a top-k value.

```python
import numpy as np

# Add vectors

db_vectors = np.random.randn(100_000, 1536).astype(np.float32)
index.add(db_vectors)

# Search

query = np.random.randn(1, 1536).astype(np.float32)
scores, ids = index.search(query, k=10)

```

### Use IdMapIndex for Stable External IDs

If your FAISS workflow uses `IndexIDMap` to preserve external identifiers, switch to `IdMapIndex`. It maps each vector to a stable `uint64` ID and supports `remove(id)` in constant time.

```python
from turbovec import IdMapIndex

idx = IdMapIndex(dim=1536, bit_width=4)

# external_ids must be uint64

external_ids = np.arange(100_000, dtype=np.uint64) + 1_000_000
idx.add_with_ids(db_vectors, external_ids)

# Deletion

idx.remove(1_000_042)

# Search returns external IDs directly

scores, ext_ids = idx.search(query, k=10)

```

### Enable Hybrid Filtering with Allowlists

Instead of retrieving top-k results and intersecting them with a candidate set manually, pass an **allowlist** array directly to `search()`. The SIMD kernel filters inside the scoring loop.

```python
candidates = np.array([1_000_123, 1_000_456, 1_000_789], dtype=np.uint64)
scores, ids = idx.search(query, k=10, allowlist=candidates)

```

Under the hood, [`search.rs`](https://github.com/RyanCodrai/turbovec/blob/main/search.rs) checks `block_has_allowed` to skip blocks that contain no allowed IDs, which avoids wasted decode cycles.

### Persist and Load Indexes

Serialization works similarly to FAISS `write_index` and `read_index`.

```python

# Save

index.write("my_index.tq")

# Load

loaded = TurboQuantIndex.load("my_index.tq")

```

In Rust, the same methods are available on both `TurboQuantIndex` and `IdMapIndex`.

```rust
index.write("my_index.tv")?;
let loaded = TurboQuantIndex::load("my_index.tv")?;

```

## turbovec vs FAISS IndexPQ: Key Differences and Gotchas

Understanding the behavioral gaps between the two libraries prevents silent accuracy or performance regressions.

### No Training Data Required

FAISS IndexPQ relies on k-means convergence over a learn set. turbovec skips this entirely because [`encode.rs`](https://github.com/RyanCodrai/turbovec/blob/main/encode.rs) applies a random rotation and a pre-computed Lloyd-Max codebook. You can delete every `train()` call in your migration.

### Bit-Width Selection

FAISS typically defaults to 8-bit codes. turbovec supports **2-bit** (4 centroids) and **4-bit** (16 centroids). Start with `bit_width=4` for the best recall-speed trade-off, or use `bit_width=2` when you need aggressive compression (16× reduction).

### Per-Vector Scale Storage

turbovec stores a per-vector `f32` scale to reconstruct unbiased inner-product scores. If your vectors have extreme norms, verify that values do not underflow. The library internally stores the norm and handles normalization during encoding, but reasonable input scales are recommended.

### CPU-Only SIMD Execution

FAISS offers GPU backends. turbovec is CPU-only and optimized for NEON on ARM and AVX-512BW on x86. Use turbovec for low-latency CPU search or edge deployments where a GPU is unavailable.

### Built-In Kernel Filtering

FAISS requires you to intersect or post-filter results. turbovec's `allowlist` mask is evaluated inside `search_multi_query_avx2` and `search_multi_query_avx512bw`, guaranteeing that every returned ID belongs to the candidate set.

## Complete Migration Examples

### Python Migration Snippet

```python
import numpy as np
from turbovec import IdMapIndex

dim = 1536
bit_width = 4

# 1. Create index

index = IdMapIndex(dim=dim, bit_width=bit_width)

# 2. Prepare data

vectors = np.random.randn(100_000, dim).astype(np.float32)
ids = np.arange(100_000, dtype=np.uint64) + 1_000_000

# 3. Add with external IDs

index.add_with_ids(vectors, ids)

# 4. Search

query = np.random.randn(1, dim).astype(np.float32)
scores, results = index.search(query, k=10)
print("Top-10 IDs:", results[0])

# 5. Hybrid filtering

candidates = np.array([1_000_123, 1_000_456], dtype=np.uint64)
scores_f, ids_f = index.search(query, k=5, allowlist=candidates)
print("Filtered IDs:", ids_f[0])

```

### Rust Migration Snippet

```rust
use turbovec::IdMapIndex;

fn main() -> anyhow::Result<()> {
    let dim = 1536usize;
    let mut index = IdMapIndex::new(dim, 4);

    // Add vectors with external IDs
    let n = 10_000;
    let vectors = vec![0.1f32; n * dim];
    let ids: Vec<u64> = (0..n as u64).map(|i| i + 1_000_000).collect();
    index.add_with_ids(&vectors, &ids);

    // Query
    let query = vec![0.1f32; dim];
    let (scores, ext_ids) = index.search(&query, 10);
    println!("Top-10 IDs: {:?}", &ext_ids[..10]);
    println!("Scores:    {:?}", &scores[..10]);

    // Persist
    index.write("my_index.tvim")?;
    let loaded = IdMapIndex::load("my_index.tvim")?;
    assert_eq!(loaded.dim(), dim);
    Ok(())
}

```

## Summary

- **Drop the training step:** turbovec does not require k-means learning; instantiate `TurboQuantIndex` and call `add()` immediately.
- **Choose your bit width:** Use `bit_width=4` for 8× compression with strong recall, or `bit_width=2` for 16× compression.
- **Swap FAISS IndexPQ constructors:** Replace `faiss.IndexPQ(d, M, 8)` with `TurboQuantIndex(dim=d, bit_width=4)`.
- **Use `IdMapIndex` for external IDs:** It provides stable `uint64` identifiers and O(1) deletion via `remove(id)`.
- **Filter inside the kernel:** Pass an `allowlist` to `search()` instead of post-filtering results.
- **Persist with `write()` and `load()`:** Both Python and Rust APIs support binary serialization.

## Frequently Asked Questions

### Do I need to train turbovec before adding vectors?

No. turbovec uses a data-oblivious random rotation and a pre-computed Lloyd-Max codebook in [`turbovec/src/encode.rs`](https://github.com/RyanCodrai/turbovec/blob/main/turbovec/src/encode.rs), so you can skip the training phase entirely. Simply create the index and call `add()` or `add_with_ids()`.

### What bit-width should I choose when migrating from FAISS IndexPQ?

Use `bit_width=4` as the default; it provides 8× compression with 16 centroids per sub-vector and higher recall than 2-bit. Choose `bit_width=2` only when you need maximum compression (16×) and can tolerate a small accuracy drop.

### How do I map external IDs in turbovec?

Use `IdMapIndex` instead of `TurboQuantIndex`. Call `add_with_ids(vectors, external_ids)` with a `uint64` ID array, and `search()` will return those external IDs directly. You can also delete by ID using `remove(id)` in O(1) time.

### Is turbovec faster than FAISS FastScan?

Yes, on ARM NEON it beats FAISS FastScan by 12-20%, and on x86 AVX-512BW it matches or exceeds FastScan performance. The speedup comes from hand-written kernels in [`turbovec/src/search.rs`](https://github.com/RyanCodrai/turbovec/blob/main/turbovec/src/search.rs) that avoid explicit codebook decoding during the search loop.