# How to Enable AVX2-Only CPU Backend for KTransformers Inference

> Enable KTransformers AVX2-only CPU backend for faster inference. Learn how to set the CPUINFER_CPU_INSTRUCT environment variable for Intel Haswell+ and AMD Zen+ processors.

- Repository: [kvcache.ai/ktransformers](https://github.com/kvcache-ai/ktransformers)
- Tags: how-to-guide
- Published: 2026-07-26

---

**Set the `CPUINFER_CPU_INSTRUCT` environment variable to `AVX2` before building or running KTransformers to force the AVX2-only CPU backend**, which supports BF16, FP8, GPTQ_INT4, and RAWINT4 quantization on Intel Haswell+ or AMD Zen+ processors without requiring AVX512.

KTransformers (kvcache-ai/ktransformers) provides optimized CPU inference for large language models through its *kt-kernel* C++ layer. For machines lacking AVX512 or AMX instructions, the framework offers a dedicated AVX2-only backend that automatically handles modern quantization formats while maximizing compatibility with older hardware.

## Verify CPU Capability

Before building, confirm your processor supports the required AVX2 instructions using the provided diagnostic script.

### Check CPU Features with check_cpu_features.py

Run the helper script located at [`kt-kernel/scripts/check_cpu_features.py`](https://github.com/kvcache-ai/ktransformers/blob/main/kt-kernel/scripts/check_cpu_features.py) to detect available instruction sets:

```bash
python kt-kernel/scripts/check_cpu_features.py

```

If your CPU supports AVX2 but lacks AVX512/AMX, the script outputs **"AVX2 Support (fallback)"**, indicating you should proceed with the AVX2-only configuration.

## Configure the Build Environment

The AVX2 backend is controlled at compile time through environment variables that modify the CMake configuration in [`kt-kernel/setup.py`](https://github.com/kvcache-ai/ktransformers/blob/main/kt-kernel/setup.py) (lines 139 and 556).

### Set CPUINFER_CPU_INSTRUCT=AVX2

Export the instruction-set flag **before** invoking any build commands:

```bash
export CPUINFER_CPU_INSTRUCT=AVX2

```

When this variable is set, the build system in [`setup.py`](https://github.com/kvcache-ai/ktransformers/blob/main/setup.py) disables AVX512/AMX flags and selects AVX2-specific compilation options (e.g., `-DLLAMA_AVX2=ON`), ensuring the kernel compiles only AVX2 code paths.

### Build the Kernel with AVX2 Flags

With the environment variable exported, compile the kernel using the provided install script or pip:

```bash
cd kt-kernel
bash install.sh

# Alternative: pip install -e .

```

The [`install.sh`](https://github.com/kvcache-ai/ktransformers/blob/main/install.sh) wrapper respects the `CPUINFER_CPU_INSTRUCT` value and passes the appropriate flags to CMake, producing binaries that target the AVX2 instruction set exclusively.

## Runtime Backend Selection

Once compiled, the Python utilities in [`kt-kernel/python/utils/amx.py`](https://github.com/kvcache-ai/ktransformers/blob/main/kt-kernel/python/utils/amx.py) expose the AVX2-specific backend objects: **`AVX2BF16_MOE`**, **`AVX2FP8_MOE`**, **`AVX2GPTQInt4_MOE`**, and **`AVX2RawInt4_MOE`**. These classes are automatically selected at runtime when higher-level instruction sets are unavailable.

Run inference using any supported quantization method:

```bash
python -m ktransformers.infer \
    --model facebook/opt-13b \
    --kt-method BF16

```

Valid `--kt-method` values for AVX2 include **BF16**, **FP8**, **GPTQ_INT4**, and **RAWINT4**, all of which route through the optimized AVX2 implementations.

## Force AVX2 on AVX512-Capable Systems

To compile the AVX2-only backend even on machines that support AVX512 or AMX, maintain the environment variable setting:

```bash
export CPUINFER_CPU_INSTRUCT=AVX2
bash install.sh  # Re-compile to overwrite previous builds

```

This configuration ignores richer instruction sets and ensures the kernel uses only AVX2 code paths, which can be useful for testing compatibility or deploying to heterogeneous hardware environments.

## Summary

- **Environment Variable**: Set `CPUINFER_CPU_INSTRUCT=AVX2` before building to force AVX2-only compilation.
- **Key Files**: Configuration logic resides in [`kt-kernel/setup.py`](https://github.com/kvcache-ai/ktransformers/blob/main/kt-kernel/setup.py) (lines 139, 556); runtime objects are exposed in [`kt-kernel/python/utils/amx.py`](https://github.com/kvcache-ai/ktransformers/blob/main/kt-kernel/python/utils/amx.py).
- **Verification**: Use [`kt-kernel/scripts/check_cpu_features.py`](https://github.com/kvcache-ai/ktransformers/blob/main/kt-kernel/scripts/check_cpu_features.py) to confirm AVX2 support.
- **Build Process**: Run [`kt-kernel/install.sh`](https://github.com/kvcache-ai/ktransformers/blob/main/kt-kernel/install.sh) or `pip install` after setting the environment variable.
- **Supported Formats**: BF16, FP8, GPTQ_INT4, and RAWINT4 all operate via the AVX2 backend.

## Frequently Asked Questions

### What CPUs support the KTransformers AVX2 backend?

Any processor supporting the AVX2 instruction set works, including Intel Haswell (4th Gen Core, 2013+) and AMD Zen-based chips. Run [`kt-kernel/scripts/check_cpu_features.py`](https://github.com/kvcache-ai/ktransformers/blob/main/kt-kernel/scripts/check_cpu_features.py) to verify your specific CPU flags before building.

### Can I run quantized models with the AVX2-only backend?

Yes. The AVX2 backend supports **GPTQ_INT4**, **RAWINT4**, **FP8**, and **BF16** quantization formats. These map to the `AVX2GPTQInt4_MOE`, `AVX2RawInt4_MOE`, `AVX2FP8_MOE`, and `AVX2BF16_MOE` runtime classes respectively.

### How do I verify that AVX2 is actually being used during inference?

Check that `CPUINFER_CPU_INSTRUCT` was set to `AVX2` during the build phase. The build system in [`setup.py`](https://github.com/kvcache-ai/ktransformers/blob/main/setup.py) prints the selected instruction set during compilation. At runtime, the presence of AVX2-specific backend objects in [`kt-kernel/python/utils/amx.py`](https://github.com/kvcache-ai/ktransformers/blob/main/kt-kernel/python/utils/amx.py) confirms the compiled kernel is active.

### Do I need to rebuild if I want to switch between AVX2 and AVX512?

Yes. The instruction set selection occurs at compile time via CMake flags in [`setup.py`](https://github.com/kvcache-ai/ktransformers/blob/main/setup.py). To switch backends, set or unset `CPUINFER_CPU_INSTRUCT=AVX2` and re-run `bash install.sh` to generate new binaries targeting the desired instruction set.