# Turbovec Empty Allowlist Handling and Selective Filtering Explained

> Understand Turbovec's empty allowlist error handling and how selective filtering efficiently deduplicates IDs using boolean masks for optimized results. Learn how to leverage Turbovec's features.

- Repository: [Ryan Codrai/turbovec](https://github.com/RyanCodrai/turbovec)
- Tags: deep-dive
- Published: 2026-07-18

---

**Turbovec treats empty allowlists as fatal programming errors that trigger an immediate panic with the message "allowlist is empty", while valid allowlists enable efficient selective filtering by creating a boolean mask that automatically deduplicates IDs and limits results to `min(k, allowlist.len())` per query.**

The `IdMapIndex` struct in the [RyanCodrai/turbovec](https://github.com/RyanCodrai/turbovec) repository provides high-performance vector search with optional selective filtering through an allowlist mechanism. Understanding how this system handles edge cases—particularly empty allowlists and invalid IDs—is critical for building robust applications that leverage approximate nearest neighbor search with constrained subsets.

## Core Implementation in IdMapIndex

### The search_with_allowlist Method

The selective filtering logic resides in [`turbovec/src/id_map.rs`](https://github.com/RyanCodrai/turbovec/blob/main/turbovec/src/id_map.rs) within the `search_with_allowlist` method (lines 191-199). This function accepts an optional slice of `u64` IDs that restricts the search domain to a specific subset of the index. When `allowlist` is `Some(ids)`, the system performs three distinct validation and transformation steps before delegating to the underlying quantized search kernel.

### Empty Allowlist Protection

The first defensive check ensures the allowlist contains at least one valid entry. At line 209, the code asserts:

```rust
assert!(!ids.is_empty(), "allowlist is empty");

```

If a caller passes an empty vector, the program panics immediately with the exact message *"allowlist is empty"*. This behavior is verified by the test `empty_allowlist_panics` in [`turbovec/tests/filtering.rs`](https://github.com/RyanCodrai/turbovec/blob/main/turbovec/tests/filtering.rs) (lines 76-86), confirming that empty filters are treated as programming errors rather than silent no-ops.

### ID Validation and Mask Creation

After confirming the list is non-empty, the system validates each ID against the internal `id_to_slot` hash map. At line 214, the code handles missing entries:

```rust
None => panic!("id {id} in allowlist is not present in index"),

```

This strict validation prevents silent mismatches where users might accidentally filter on IDs that were never added to the index. The test `unknown_id_in_allowlist_panics` (lines 88-99 in [`turbovec/tests/filtering.rs`](https://github.com/RyanCodrai/turbovec/blob/main/turbovec/tests/filtering.rs)) exercises this panic path.

For valid IDs, the corresponding slots are marked in a `Vec<bool>` mask. Because this mask uses boolean indexing, duplicate IDs in the input allowlist are automatically deduplicated—only unique slots are marked as `true` before the mask is passed to `search_with_mask`.

The documentation at lines 193-196 clarifies that after deduplication, the kernel returns at most `min(k, allowlist.len())` results per query, ensuring the result set never exceeds the constrained domain size.

## Python Bindings and Error Translation

The Python API in [`turbovec-python/src/lib.rs`](https://github.com/RyanCodrai/turbovec/blob/main/turbovec-python/src/lib.rs) (lines 249-267) exposes identical semantics through the `search_with_allowlist` method. Rather than crashing the interpreter, Rust panics are caught and translated into appropriate Python exceptions:

- **Empty allowlists** raise `ValueError` with the message *"allowlist is empty"*
- **Unknown IDs** raise `KeyError` indicating which specific ID is not present in the index

This translation layer allows Python developers to handle filtering errors gracefully using standard exception handling patterns.

## Practical Code Examples

### Rust Usage with Allowlists

The following example demonstrates safe allowlist usage in Rust, including the panic behavior for empty lists:

```rust
use turbovec::IdMapIndex;
use turbovec::utils::gaussian_normalized;

let dim = 64;
let data = gaussian_normalized(10, dim, 0x1234);
let ids: Vec<u64> = (0..10).collect();
let mut idx = IdMapIndex::new(dim, 4).unwrap();
idx.add_with_ids(&data, &ids).unwrap();

let query = gaussian_normalized(1, dim, 0x5678);
let allowlist = vec![ids[2], ids[5], ids[7]];
let (scores, results) = idx.search_with_allowlist(&query, 5, Some(&allowlist));

println!("Top scores: {:?}", &scores[..results.len()]);
println!("Returned ids (filtered): {:?}", results);

// This will panic with "allowlist is empty"
// let empty: Vec<u64> = vec![];
// let _ = idx.search_with_allowlist(&query, 3, Some(&empty));

```

### Python Integration

Python users can leverage the same filtering capabilities with NumPy arrays, catching validation errors as exceptions:

```python
import turbovec
import numpy as np

idx = turbovec.IdMapIndex(dim=64, bit_width=4)
data = np.random.randn(10, 64).astype(np.float32)
ids = np.arange(10, dtype=np.uint64)
idx.add_with_ids(data, ids)

query = np.random.randn(1, 64).astype(np.float32)
allowlist = np.array([2, 5, 7], dtype=np.uint64)

scores, result_ids = idx.search_with_allowlist(query, k=5, allowlist=allowlist)
print("Filtered ids:", result_ids)

# Handle empty allowlist safely

try:
    idx.search_with_allowlist(query, k=3, allowlist=np.array([], dtype=np.uint64))
except ValueError as e:
    print("Caught error:", e)  # -> "allowlist is empty"

```

## Summary

- **Empty allowlists trigger immediate panics** via `assert!(!ids.is_empty())` at line 209 of [`turbovec/src/id_map.rs`](https://github.com/RyanCodrai/turbovec/blob/main/turbovec/src/id_map.rs), preventing undefined behavior in downstream search kernels.
- **Unknown IDs cause descriptive panics** indicating exactly which identifier is missing from the `id_to_slot` map, as implemented at line 214.
- **Duplicate IDs are harmless**—the boolean mask construction automatically deduplicates entries before executing the search.
- **Result counts are bounded** by `min(k, allowlist.len())` after deduplication, ensuring realistic expectations for filtered search cardinality.
- **Python exceptions map directly** to Rust panics, with `ValueError` for empty lists and `KeyError` for invalid IDs in [`turbovec-python/src/lib.rs`](https://github.com/RyanCodrai/turbovec/blob/main/turbovec-python/src/lib.rs).

## Frequently Asked Questions

### What happens if I pass an empty allowlist to Turbovec?

The system panics immediately with the message *"allowlist is empty"* in Rust, or raises a `ValueError` in Python. This strict validation at line 209 of [`turbovec/src/id_map.rs`](https://github.com/RyanCodrai/turbovec/blob/main/turbovec/src/id_map.rs) treats empty filters as programming errors rather than returning zero results, helping developers catch logic errors early.

### How does Turbovec handle duplicate IDs in an allowlist?

Duplicate IDs are automatically deduplicated during mask creation. The system builds a `Vec<bool>` mask where each index represents a slot in the index, so multiple references to the same ID result in a single `true` value. This ensures the search kernel only processes unique vectors regardless of input redundancy.

### What is the maximum number of results returned when using an allowlist?

The search returns at most `min(k, allowlist.len())` results per query after deduplication. As documented in [`turbovec/src/id_map.rs`](https://github.com/RyanCodrai/turbovec/blob/main/turbovec/src/id_map.rs) (lines 193-196), you cannot receive more results than the number of unique, valid IDs provided in your filter list, even if `k` is larger than the allowlist size.

### Where is the allowlist filtering logic implemented in the source code?

The core implementation resides in [`turbovec/src/id_map.rs`](https://github.com/RyanCodrai/turbovec/blob/main/turbovec/src/id_map.rs) within the `search_with_allowlist` method (lines 191-199), while the Python bindings that translate errors are located in [`turbovec-python/src/lib.rs`](https://github.com/RyanCodrai/turbovec/blob/main/turbovec-python/src/lib.rs) (lines 249-267). Unit tests verifying empty list and unknown ID behavior are available in [`turbovec/tests/filtering.rs`](https://github.com/RyanCodrai/turbovec/blob/main/turbovec/tests/filtering.rs).