SIMD Kernel Optimizations in Turbovec: x86 AVX2, AVX‑512, and ARM NEON Implementations

Turbovec uses hand‑written SIMD kernels—AVX2 and AVX‑512BW on x86_64, NEON on AArch64—to accelerate 4‑bit compressed vector search, ensuring bit‑identical scoring across architectures through fused multiply‑add flushes and vectorized top‑k heap updates.

Turbovec is a high‑throughput vector search library that exploits Single Instruction Multiple Data (SIMD) instructions to score quantized vectors. According to the RyanCodrai/turbovec source code, the library implements architecture‑specific kernels that decode nibble‑packed codes, accumulate partial sums, and maintain top‑k results using native vector registers. These SIMD kernel optimizations enable the library to process multiple queries per clock cycle while guaranteeing numerical consistency between x86 and ARM platforms.

x86 SIMD Kernel Optimizations

The x86_64 implementation in turbovec/src/search.rs provides two distinct kernel paths selected by CPU capability: AVX2 for broad compatibility and AVX‑512BW for maximum throughput.

AVX2 Kernel (256‑bit)

The AVX2 kernel, declared with #[target_feature(enable = "avx2", enable = "fma")], resides at lines 66‑80 of turbovec/src/search.rs. This implementation processes four queries simultaneously using 256‑bit YMM registers.

Key optimization techniques include:

• Nibble masking – Uses _mm256_set1_epi8(0x0F) to isolate low and high nibbles of each compressed byte.
• Vectorized lookup – Loads 32‑byte lookup tables (LUTs) per query via _mm256_loadu_si256 and shuffles values with _mm256_shuffle_epi8.
• 16‑bit accumulation – Collects partial scores in 16‑bit integer lanes to defer floating‑point conversion.
• Fused multiply‑add flush – Every FLUSH_EVERY groups converts 16‑bit lanes to f32 and applies _mm256_fmadd_ps to compute scale * partial + bias in a single instruction.
• Top‑k pruning – Employs _mm256_cmp_ps for vectorized comparisons that filter scores before they enter the heap.

AVX‑512BW Kernel (512‑bit)

The AVX‑512 variant, located at lines 96‑108 of turbovec/src/search.rs, extends the AVX2 strategy to 512‑bit ZMM registers.

Key enhancements over AVX2:

• Dual‑block processing – Scores two compressed blocks per inner iteration using 512‑bit shuffles (_mm512_shuffle_epi8).
• Per‑nibble masking – Maintains accuracy with _mm512_set1_epi8 masks for simultaneous high/low nibble extraction.
• Bit‑identical arithmetic – Reuses the same FMAD flush pattern (scale * partial + bias) as AVX2, ensuring cross‑kernel reproducibility.

ARM NEON SIMD Kernel Optimizations

On AArch64, Turbovec leverages 128‑bit NEON vectors through the score_4bit_block_neon function (lines 345‑360 of turbovec/src/search.rs).

The NEON pipeline mirrors the x86 approach but adapts to ARM’s instruction set:

1. Table lookup – Loads 16‑byte LUT halves with vld1q_u8 and decodes nibbles via vqtbl1q_u8, indexed by both low (c & mask) and high (c >> 4) nibbles.
2. Widening accumulation – Accumulates 8‑bit LUT results into 16‑bit lanes using vaddw_u8 to avoid overflow.
3. Single‑step FMAD – Widens to 32‑bit integers, converts to f32, and executes vfmaq_f32 to fuse the scale‑multiply‑add operation.
4. Top‑k update – neon_block_topk_update (and helpers scan_groups_neon, score_4query_block_neon) uses vmaxq_f32‑style comparisons to prune candidates vector‑wide.

Cross‑Architecture Consistency

Both x86 and ARM kernels target bit‑identical results despite differing register widths. The NEON flushing sequence (scale * partial + bias) deliberately mirrors the AVX2/AVX‑512 FMAD pattern so that identical encoded LUTs produce identical floating‑point scores on Intel, AMD, and ARM processors. This architectural parity ensures deterministic search results regardless of deployment hardware.

Runtime Kernel Selection and Usage

Turbovec automatically dispatches to the optimal kernel using standard Rust feature detection:

• x86 – is_x86_feature_detected! probes for AVX‑512BW, falling back to AVX2, then scalar.
• ARM – std::arch::is_aarch64_feature_detected! confirms NEON availability.

Users can override automatic selection by setting the TURBOVEC_REQUIRE_SIMD environment variable to a comma‑separated list of required features.

// Force AVX2 on a server that also supports AVX-512
std::env::set_var("TURBOVEC_REQUIRE_SIMD", "avx2");
let results = turbovec::search(&codes, &queries, k);

Minimal Integration Example

use turbovec::{search, encode, Codebook};

fn main() {
    // Train a 4-bit codebook (SIMD-agnostic)
    let cb = Codebook::train(&data, 4).unwrap();
    
    // Encode vectors: uses NEON on ARM, AVX2/AVX-512 on x86
    let encoded = encode(&cb, &data);
    
    // Search automatically selects AVX-512BW, AVX2, or NEON
    let top_k = search(&encoded, &queries, 10);
}

Summary

• x86 kernels in turbovec/src/search.rs provide AVX2 (256‑bit) and AVX‑512BW (512‑bit) paths, both using _mm256/512_shuffle_epi8 for nibble lookup and fused multiply‑add flushes for accumulation.
• ARM kernels employ 128‑bit NEON vectors with vqtbl1q_u8 table lookups and vfmaq_f32 fused operations, implemented in score_4bit_block_neon and related helpers.
• Bit‑identical arithmetic across architectures ensures that scale * partial + bias produces the same results on x86 and ARM.
• Automatic dispatch via Rust’s feature detection selects the highest available SIMD level, with manual override via TURBOVEC_REQUIRE_SIMD.

Frequently Asked Questions

What SIMD instructions does Turbovec use for vector search on x86?

Turbovec utilizes AVX2 with FMA (_mm256_fmadd_ps, _mm256_shuffle_epi8) for 256‑bit processing and AVX‑512BW (_mm512_shuffle_epi8) for 512‑bit processing. Both kernels reside in turbovec/src/search.rs and perform nibble‑mask lookups followed by fused multiply‑add flushes to score compressed vectors.

How does Turbovec optimize vector search on ARM processors?

On AArch64, Turbovec uses NEON intrinsics through functions like score_4bit_block_neon and neon_block_topk_update. The implementation leverages vqtbl1q_u8 for LUT‑based nibble decoding and vfmaq_f32 for fused multiply‑add accumulation, processing 128‑bit vectors to maintain parity with the x86 kernels.

Can I force Turbovec to use a specific SIMD kernel?

Yes. Set the environment variable TURBOVEC_REQUIRE_SIMD to a comma‑separated list of target features (e.g., "avx2" or "neon"). The library will dispatch only to kernels matching those features and fall back to scalar code if the requirements are unmet.

Why does Turbovec use 16‑bit integer accumulators before the final FMAD flush?

The kernels accumulate partial scores in 16‑bit integers (vaddw_u8 on NEON, _mm256_add_epi16 on AVX2) to maximize throughput while avoiding overflow. These accumulators are widened to 32‑bit and converted to f32 only during the periodic flush, at which point the fused multiply‑add applies the scale and bias. This pattern reduces floating‑point pressure and ensures bit‑identical results across x86 and ARM.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}