Turbovec SIMD Intrinsics for ARM vs x86 Search: NEON, AVX2, and AVX-512BW Kernels Explained

Turbovec leverages ARM NEON intrinsics including vqtbl1q_u8 and vfmaq_f32 for 128-bit aarch64 operations, while x86 platforms utilize AVX2 (_mm256_shuffle_epi8, _mm256_fmadd_ps) or AVX-512BW (_mm512_shuffle_epi8, _mm512_inserti64x4) to accelerate 4-bit quantized vector similarity search through lookup-table-based dot products.

The RyanCodrai/turbovec repository implements high-throughput approximate nearest neighbor search via hand-optimized SIMD kernels written in Rust. By utilizing std::arch intrinsics, turbovec achieves bit-identical scoring across ARM and x86 architectures through a 4-bit quantization scheme that relies on nibble-splitting and lookup table (LUT) accumulation. Understanding these turbovec SIMD intrinsics for ARM vs x86 search reveals how the library maximizes throughput on each platform while maintaining identical numerical results.

ARM NEON Intrinsics (aarch64)

On ARM 64-bit targets, turbovec implements the score_4bit_block_neon function in turbovec/src/search.rs using 128-bit NEON registers. This kernel processes 32-byte code blocks by splitting each byte into high and low 4-bit nibbles, using table lookup instructions that are uniquely efficient on ARM.

The NEON implementation relies on the following key intrinsics:

• vdupq_n_u8 / vdupq_n_f32 – Broadcast scalar values (such as the 0x0F nibble mask and per-query scales) across all lanes of a 128-bit vector.
• vld1q_u8 / vld1q_f32 – Load 16-byte or 16-float chunks from the blocked codes and lookup tables.
• vqtbl1q_u8 – Perform byte-wise table lookups in a single instruction, indexing the 16-byte LUT with the 4-bit nibbles extracted from the codes.
• vaddw_u8 – Widen and add 8-bit values into 16-bit accumulators to prevent overflow during dot product accumulation.
• vfmaq_f32 – Execute fused multiply-add operations when converting final 16-bit integer accumulators to 32-bit floats and applying per-vector scaling factors.

This approach loads 32-byte code blocks, splits nibbles using vandq_u8 and vshrq_n_u8, accumulates via vaddq_u8 and vaddw_u8, then finishes with vfmaq_f32 for the final scaling step.

#[cfg(target_arch = "aarch64")]
unsafe fn score_4bit_block_neon(
    blocked_codes: &[u8],
    uint8_luts:   &[u8],
    /* … other args … */
) {
    use std::arch::aarch64::*;

    let mask = vdupq_n_u8(0x0F);
    let v_scale = vdupq_n_f32(scale);

    for batch in 0..n_batches {
        // Load LUTs and code bytes (32 bytes each)
        let lut_hi = vld1q_u8(lp);
        let lut_lo = vld1q_u8(lp.add(16));
        let c0 = vld1q_u8(cp);
        let c1 = vld1q_u8(cp.add(16));

        // Table-lookup + nibble split
        let s0 = vaddq_u8(vqtbl1q_u8(lut_lo, vandq_u8(c0, mask)),
                         vqtbl1q_u8(lut_hi, vshrq_n_u8(c0, 4)));
        let lo = vcvtq_f32_u32(vmovl_u16(vget_low_u16(acc[i])));
        fa[i*2] = vfmaq_f32(fa[i*2], v_scale, lo);
    }
}

x86 AVX2 Intrinsics (x86_64)

For x86 platforms without AVX-512 support, turbovec provides the search_multi_query_avx2 function in turbovec/src/search.rs. This kernel processes 256-bit vectors (32 bytes) per iteration, using AVX2's powerful byte-shuffle capabilities to perform the LUT lookups that ARM handles with vqtbl1q_u8.

Critical AVX2 intrinsics include:

• _mm256_set1_epi8 – Broadcast the 0x0F mask across all 32 lanes of a 256-bit vector.
• _mm256_loadu_si256 – Load unaligned 256-bit chunks of packed codes and lookup tables.
• _mm256_and_si256 and _mm256_srli_epi16 – Extract low and high nibbles by masking and shifting.
• _mm256_shuffle_epi8 – Perform 32 parallel byte lookups into the LUT using the extracted nibbles as indices.
• _mm256_add_epi16 – Accumulate partial dot products into 16-bit integer vectors.
• _mm256_cvtepu16_epi32 and _mm256_cvtepi32_ps – Widen 16-bit integers to 32-bit floats.
• _mm256_fmadd_ps – Apply per-query scaling factors using fused multiply-add operations.

The AVX2 kernel follows the same nibble-splitting logic as NEON but processes twice the data width per instruction.

#[cfg(target_arch = "x86_64")]
#[target_feature(enable = "avx2,fma")]
unsafe fn search_multi_query_avx2(/* … */) {
    use std::arch::x86_64::*;

    let nibble_mask = _mm256_set1_epi8(0x0F);
    for batch in 0..n_batches {
        let mut accus = [[_mm256_setzero_si256(); 4]; 4];
        for g in g_start..g_end {
            let cp = codes_base.add((b * n_byte_groups + g) * BLOCK);
            let codes_v = _mm256_loadu_si256(cp as *const __m256i);
            let clo = _mm256_and_si256(codes_v, nibble_mask);
            let chi = _mm256_and_si256(_mm256_srli_epi16(codes_v, 4), nibble_mask);

            for qi in 0..4 {
                let lut = _mm256_loadu_si256(luts[qi].as_ptr().add(g * 32) as *const __m256i);
                let res0 = _mm256_shuffle_epi8(lut, clo);
                let res1 = _mm256_shuffle_epi8(lut, chi);
                accus[qi][0] = _mm256_add_epi16(accus[qi][0], res0);
            }
        }
        let f0 = _mm256_cvtepi32_ps(_mm256_cvtepu16_epi32(_mm256_castsi256_si128(dis0)));
        fa[qi][0] = _mm256_fmadd_ps(v_scales[qi], f0, fa[qi][0]);
    }
}

x86 AVX-512BW Intrinsics (x86_64)

The search_multi_query_avx512bw function in turbovec/src/search.rs targets AVX-512BW-capable processors, processing 512-bit vectors by fusing two 256-bit blocks into a single register. This approach halves the number of loop iterations compared to the AVX2 implementation while maintaining identical numerical precision.

Key AVX-512BW intrinsics include:

• _mm512_inserti64x4 and _mm512_castsi256_si512 – Combine two 256-bit code blocks into one 512-bit register (zmm).
• _mm512_set1_epi8 – Broadcast the nibble mask across 64 lanes.
• _mm512_and_si512 and _mm512_srli_epi16 – Extract low and high nibbles at 512-bit width.
• _mm512_shuffle_epi8 – Perform 64 parallel byte lookups (AVX-512BW extends the AVX2 shuffle to 512 bits).
• _mm512_broadcast_i64x4 – Broadcast 256-bit LUT data into 512-bit registers for the shuffle operation.
• _mm512_add_epi16 – Accumulate into 512-bit vectors.
• _mm512_extracti64x4_epi64 – Split the 512-bit accumulator back into 256-bit halves for final conversion using AVX2 helpers.

This kernel utilizes the wider vector width to process paired blocks simultaneously, significantly improving instructions-per-cycle throughput on server-grade x86 hardware.

#[cfg(target_arch = "x86_64")]
#[target_feature(enable = "avx2,fma,avx512f,avx512bw")]
unsafe fn search_multi_query_avx512bw(/* … */) {
    use std::arch::x86_64::*;

    let mask512 = _mm512_set1_epi8(0x0F);
    for batch in 0..n_batches {
        let mut accus = [[_mm512_setzero_si512(); 4]; 4];
        for gp in gp_start..gp_end {
            let codes_a = _mm512_inserti64x4(
                _mm512_castsi256_si512(_mm256_loadu_si256(cp0_a as *const __m256i)),
                _mm256_loadu_si256(cp1_a as *const __m256i),
                1,
            );

            let clo = _mm512_and_si512(codes_a, mask512);
            let chi = _mm512_and_si512(_mm512_srli_epi16(codes_a, 4), mask512);

            for qi in 0..4 {
                let lut_a = _mm512_broadcast_i64x4(
                    _mm256_loadu_si256(luts[qi].as_ptr().add(g0 * 32) as *const __m256i)
                );
                let res0_a = _mm512_shuffle_epi8(lut_a, clo);
                accus[qi][0] = _mm512_add_epi16(accus[qi][0], res0_a);
            }
        }
        let low_half = _mm512_extracti64x4_epi64(accus[0][0], 0);
        avx2_batch_flush_to_fa(low_half, &mut fa[qi]);
    }
}

Core Data Flow and Supporting Files

The SIMD kernels consume a specific data layout defined in turbovec/src/pack.rs, which implements a FAISS-style perm0-interleaved block format required for efficient LUT-based scoring. Query-specific lookup tables are constructed in turbovec/src/encode.rs via the build_query_neon_lut_from_slice function, preparing the 16-byte tables that both ARM and x86 kernels index during search.

Turbovec selects the appropriate kernel at compile time using #[cfg(target_arch = "aarch64")] for ARM and #[cfg(target_arch = "x86_64")] for x86, with runtime feature detection (is_x86_feature_detected!) choosing between AVX2 and AVX-512BW paths on x86 platforms.

Summary

• ARM NEON uses 128-bit vectors with vqtbl1q_u8 for table lookups and vfmaq_f32 for fused multiply-add scaling in the score_4bit_block_neon function.
• x86 AVX2 processes 256-bit vectors using _mm256_shuffle_epi8 for parallel byte lookups and _mm256_fmadd_ps for final scaling in search_multi_query_avx2.
• x86 AVX-512BW doubles throughput via 512-bit operations in search_multi_query_avx512bw, utilizing _mm512_inserti64x4 to process paired blocks and _mm512_shuffle_epi8 for 64-lane parallel lookups.
• All three kernels implement identical 4-bit nibble-splitting logic and accumulate into 16-bit integers before converting to 32-bit floats, ensuring bit-identical results across architectures.
• The implementation resides primarily in turbovec/src/search.rs, with data layout support in turbovec/src/pack.rs and LUT generation in turbovec/src/encode.rs.

Frequently Asked Questions

Why does turbovec use different intrinsics for ARM vs x86 search?

ARM NEON and x86 AVX2/AVX-512BW offer fundamentally different instruction sets for parallel table lookups. NEON provides vqtbl1q_u8 for 16-byte table lookups natively, while AVX2 relies on _mm256_shuffle_epi8 with careful nibble extraction. The library maps the same 4-bit quantization algorithm to each architecture's most efficient instructions rather than forcing a common denominator approach, maximizing throughput on both platforms.

What is the performance difference between AVX2 and AVX-512BW in turbovec?

The AVX-512BW kernel processes two 256-bit blocks simultaneously within 512-bit registers, reducing loop iteration count by half compared to AVX2. By using _mm512_inserti64x4 to pair blocks and _mm512_shuffle_epi8 to perform 64 parallel byte lookups, the AVX-512BW implementation achieves higher instructions-per-cycle throughput on compatible hardware, though the actual speedup depends on CPU-specific frequency scaling and thermal constraints.

How does turbovec maintain bit-identical results across ARM and x86?

All three kernels follow the same numerical pipeline: split 4-bit nibbles, index into query-specific LUTs using byte-wise shuffles, accumulate into 16-bit integers to prevent overflow, then convert to 32-bit floats and apply fused multiply-add scaling. By using fixed-point integer accumulation until the final scaling step—implemented via vfmaq_f32 on ARM and _mm256_fmadd_ps or _mm512_fmadd_ps on x86—turbovec ensures deterministic, architecture-independent similarity scores.

Which source file contains the main search implementations?

turbovec/src/search.rs contains all three SIMD kernels: score_4bit_block_neon for ARM, search_multi_query_avx2 for 256-bit x86, and search_multi_query_avx512bw for 512-bit x86. Supporting logic for LUT construction resides in turbovec/src/encode.rs, while the required FAISS-style block interleaving format is defined in turbovec/src/pack.rs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}