Turbovec Empty Allowlist Handling and Selective Filtering Explained

Turbovec treats empty allowlists as fatal programming errors that trigger an immediate panic with the message "allowlist is empty", while valid allowlists enable efficient selective filtering by creating a boolean mask that automatically deduplicates IDs and limits results to min(k, allowlist.len()) per query.

The IdMapIndex struct in the RyanCodrai/turbovec repository provides high-performance vector search with optional selective filtering through an allowlist mechanism. Understanding how this system handles edge cases—particularly empty allowlists and invalid IDs—is critical for building robust applications that leverage approximate nearest neighbor search with constrained subsets.

Core Implementation in IdMapIndex

The search_with_allowlist Method

The selective filtering logic resides in turbovec/src/id_map.rs within the search_with_allowlist method (lines 191-199). This function accepts an optional slice of u64 IDs that restricts the search domain to a specific subset of the index. When allowlist is Some(ids), the system performs three distinct validation and transformation steps before delegating to the underlying quantized search kernel.

Empty Allowlist Protection

The first defensive check ensures the allowlist contains at least one valid entry. At line 209, the code asserts:

assert!(!ids.is_empty(), "allowlist is empty");

If a caller passes an empty vector, the program panics immediately with the exact message "allowlist is empty". This behavior is verified by the test empty_allowlist_panics in turbovec/tests/filtering.rs (lines 76-86), confirming that empty filters are treated as programming errors rather than silent no-ops.

ID Validation and Mask Creation

After confirming the list is non-empty, the system validates each ID against the internal id_to_slot hash map. At line 214, the code handles missing entries:

None => panic!("id {id} in allowlist is not present in index"),

This strict validation prevents silent mismatches where users might accidentally filter on IDs that were never added to the index. The test unknown_id_in_allowlist_panics (lines 88-99 in turbovec/tests/filtering.rs) exercises this panic path.

For valid IDs, the corresponding slots are marked in a Vec<bool> mask. Because this mask uses boolean indexing, duplicate IDs in the input allowlist are automatically deduplicated—only unique slots are marked as true before the mask is passed to search_with_mask.

The documentation at lines 193-196 clarifies that after deduplication, the kernel returns at most min(k, allowlist.len()) results per query, ensuring the result set never exceeds the constrained domain size.

Python Bindings and Error Translation

The Python API in turbovec-python/src/lib.rs (lines 249-267) exposes identical semantics through the search_with_allowlist method. Rather than crashing the interpreter, Rust panics are caught and translated into appropriate Python exceptions:

• Empty allowlists raise ValueError with the message "allowlist is empty"
• Unknown IDs raise KeyError indicating which specific ID is not present in the index

This translation layer allows Python developers to handle filtering errors gracefully using standard exception handling patterns.

Practical Code Examples

Rust Usage with Allowlists

The following example demonstrates safe allowlist usage in Rust, including the panic behavior for empty lists:

use turbovec::IdMapIndex;
use turbovec::utils::gaussian_normalized;

let dim = 64;
let data = gaussian_normalized(10, dim, 0x1234);
let ids: Vec<u64> = (0..10).collect();
let mut idx = IdMapIndex::new(dim, 4).unwrap();
idx.add_with_ids(&data, &ids).unwrap();

let query = gaussian_normalized(1, dim, 0x5678);
let allowlist = vec![ids[2], ids[5], ids[7]];
let (scores, results) = idx.search_with_allowlist(&query, 5, Some(&allowlist));

println!("Top scores: {:?}", &scores[..results.len()]);
println!("Returned ids (filtered): {:?}", results);

// This will panic with "allowlist is empty"
// let empty: Vec<u64> = vec![];
// let _ = idx.search_with_allowlist(&query, 3, Some(&empty));

Python Integration

Python users can leverage the same filtering capabilities with NumPy arrays, catching validation errors as exceptions:

import turbovec
import numpy as np

idx = turbovec.IdMapIndex(dim=64, bit_width=4)
data = np.random.randn(10, 64).astype(np.float32)
ids = np.arange(10, dtype=np.uint64)
idx.add_with_ids(data, ids)

query = np.random.randn(1, 64).astype(np.float32)
allowlist = np.array([2, 5, 7], dtype=np.uint64)

scores, result_ids = idx.search_with_allowlist(query, k=5, allowlist=allowlist)
print("Filtered ids:", result_ids)

# Handle empty allowlist safely

try:
    idx.search_with_allowlist(query, k=3, allowlist=np.array([], dtype=np.uint64))
except ValueError as e:
    print("Caught error:", e)  # -> "allowlist is empty"

Summary

• Empty allowlists trigger immediate panics via assert!(!ids.is_empty()) at line 209 of turbovec/src/id_map.rs, preventing undefined behavior in downstream search kernels.
• Unknown IDs cause descriptive panics indicating exactly which identifier is missing from the id_to_slot map, as implemented at line 214.
• Duplicate IDs are harmless—the boolean mask construction automatically deduplicates entries before executing the search.
• Result counts are bounded by min(k, allowlist.len()) after deduplication, ensuring realistic expectations for filtered search cardinality.
• Python exceptions map directly to Rust panics, with ValueError for empty lists and KeyError for invalid IDs in turbovec-python/src/lib.rs.

Frequently Asked Questions

What happens if I pass an empty allowlist to Turbovec?

The system panics immediately with the message "allowlist is empty" in Rust, or raises a ValueError in Python. This strict validation at line 209 of turbovec/src/id_map.rs treats empty filters as programming errors rather than returning zero results, helping developers catch logic errors early.

How does Turbovec handle duplicate IDs in an allowlist?

Duplicate IDs are automatically deduplicated during mask creation. The system builds a Vec<bool> mask where each index represents a slot in the index, so multiple references to the same ID result in a single true value. This ensures the search kernel only processes unique vectors regardless of input redundancy.

What is the maximum number of results returned when using an allowlist?

The search returns at most min(k, allowlist.len()) results per query after deduplication. As documented in turbovec/src/id_map.rs (lines 193-196), you cannot receive more results than the number of unique, valid IDs provided in your filter list, even if k is larger than the allowlist size.

Where is the allowlist filtering logic implemented in the source code?

The core implementation resides in turbovec/src/id_map.rs within the search_with_allowlist method (lines 191-199), while the Python bindings that translate errors are located in turbovec-python/src/lib.rs (lines 249-267). Unit tests verifying empty list and unknown ID behavior are available in turbovec/tests/filtering.rs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}