# Debug kt-kernel Installation Issues Related to CPU Variant Detection

> Debug kt-kernel installation errors caused by CPU variant detection mismatches. Learn how to resolve link errors and performance issues for optimal compilation.

- Repository: [kvcache.ai/ktransformers](https://github.com/kvcache-ai/ktransformers)
- Tags: how-to-guide
- Published: 2026-07-20

---

**kt-kernel selects instruction-set variants (AMX, AVX-512, AVX2) at build time using a dual-detection system, and mismatches between detected capabilities and compiled binaries cause link errors, illegal instruction crashes, or silent performance degradation.**

The `kt-kernel` component of the [kvcache-ai/ktransformers](https://github.com/kvcache-ai/ktransformers) repository compiles platform-specific optimized kernels for x86_64 CPUs. Because these kernels rely on advanced instruction sets like **AMX** (Advanced Matrix Extensions) and **AVX-512**, the build system must accurately detect CPU capabilities. When detection fails or the build environment differs from the runtime environment, installation breaks with cryptic errors. This guide explains how to diagnose and fix these CPU variant mismatches using the actual detection mechanisms in the source code.

## How CPU Variant Detection Works

`kt-kernel` uses two complementary detection mechanisms that operate at different stages of the build process.

### Runtime Feature Probe

The Python helper script [`kt-kernel/scripts/check_cpu_features.py`](https://github.com/kvcache-ai/ktransformers/blob/main/kt-kernel/scripts/check_cpu_features.py) analyzes `/proc/cpuinfo` at runtime to determine which instruction-set flags the CPU actually supports. According to lines 100-106 of this script, it prints a clear recommendation for which CMake flags you should use. This is your primary diagnostic tool when debugging installation failures.

### CMake Compile-Time Detection

During the configuration phase, `kt-kernel/cmake/DetectCPU.cmake` probes the compiler and system to set cache variables. Lines 130-138 contain the critical logic:

```cmake
if(NOT DEFINED KTRANSFORMERS_CPU_USE_AMX AND HAS_AMX)
    set(KTRANSFORMERS_CPU_USE_AMX ON CACHE BOOL "Enable AMX" FORCE)
endif()

```

Similar blocks handle AVX-512 detection. If these checks fail, the variables remain `OFF` and the build silently falls back to AVX2.

## Recognizing CPU Variant Mismatch Symptoms

When the compiled binary's instruction set does not match the host CPU capabilities, you will encounter one of these specific failure modes:

- **Link-time errors** such as `undefined reference to '__builtin_ia32_amx_*'` — the binary was compiled expecting AMX instructions that the linker cannot resolve for the target CPU.
- **Runtime aborts** reporting `illegal instruction (core dumped)` — the binary contains AVX-512F, BF16, VNNI, or VBMI instructions that the CPU does not recognize.
- **Silent performance degradation** — the library loads successfully but executes AVX2 code paths despite the CPU supporting AVX-512, resulting in significantly slower inference.

## Step-by-Step Debugging Workflow

Follow this exact sequence to identify and resolve the root cause of variant-related installation failures.

### 1. Run the Runtime CPU Probe

Execute the Python detection script to establish baseline capabilities:

```bash
python3 kt-kernel/scripts/check_cpu_features.py

```

The output reports:
- **CPU Model** (e.g., "Intel Xeon Gold 6348")
- **AMX Support** status for `amx_tile`, `amx_int8`, and `amx_bf16`
- **AVX-512 Support** for `avx512f`, `avx512_bf16`, `avx512_vnni`, and `avx512_vbmi`
- **Recommendation** block specifying exact CMake flags needed

### 2. Inspect CMake Detection Logic

Open `kt-kernel/cmake/DetectCPU.cmake` and verify that the detection logic around lines 130-138 correctly identified your CPU features. If the script failed to detect AMX or AVX-512 despite the CPU probe showing support, your Linux kernel may be hiding these flags (common on kernels older than 5.10).

### 3. Force the Correct Variant

If the CPU probe confirms support but CMake missed it, manually override the cache variables:

```bash
cmake -S . -B build \
    -DKTRANSFORMERS_CPU_USE_AMX=ON \
    -DKTRANSFORMERS_CPU_USE_AMX_AVX512=ON \
    -DKTRANSFORMERS_CPU_ARCH=x86_64

```

If the probe shows missing extensions, you must either use a pre-built AVX2 wheel, upgrade your hardware/firmware, or remove explicit `-mavx512*` flags from any custom build scripts.

### 4. Build and Verify Symbols

Compile the library and inspect the generated symbols to confirm the correct variant was baked in:

```bash
cmake --build build -j$(nproc)
nm -D build/libktkernel.so | grep vpdpbusds

```

The presence of `vpdpbusds` (VNNI) or AMX-related symbols confirms the high-performance path was compiled.

### 5. Validate the Installation

Re-run the CPU probe after installation to ensure the "Recommendation" section now shows a checkmark path matching your compiled variant.

## Common Pitfalls and Solutions

| Symptom | Root Cause | Solution |
|---------|------------|----------|
| "Illegal instruction" on startup | Binary built for AMX/AVX-512 but CPU lacks flags | Run [`check_cpu_features.py`](https://github.com/kvcache-ai/ktransformers/blob/main/check_cpu_features.py) → confirm missing flags → rebuild without `-DKTRANSFORMERS_CPU_USE_AMX*` or `-DKTRANSFORMERS_CPU_USE_AMX_AVX512` |
| CMake reports "AMX not detected" on known-AMX CPU | Linux kernel < 5.10 hides AMX in `/proc/cpuinfo` | Upgrade kernel or manually set `-DKTRANSFORMERS_CPU_USE_AMX=ON` |
| No AVX-512 symbols in compiled library | Stale CMake cache from previous configuration | Delete [`CMakeCache.txt`](https://github.com/kvcache-ai/ktransformers/blob/main/CMakeCache.txt) and `build/` directory, then reconfigure |
| AVX2-level performance on AVX-512 CPU | Missing `avx512_bf16` or `avx512_vbmi` caused fallback | Verify flags via probe; if present, force `-DKTRANSFORMERS_CPU_USE_AMX_AVX512=ON` |

## Key Source Files

Understanding these specific files helps you trace detection logic:

- **[`kt-kernel/scripts/check_cpu_features.py`](https://github.com/kvcache-ai/ktransformers/blob/main/kt-kernel/scripts/check_cpu_features.py)** — Runtime detector that parses `/proc/cpuinfo` and recommends CMake flags.
- **`kt-kernel/cmake/DetectCPU.cmake`** — CMake module that sets `KTRANSFORMERS_CPU_USE_AMX` and related cache variables during configuration.
- **`kt-kernel/cmake/KTransformersConfig.cmake`** — Generated configuration exposing CPU variant options to downstream builds.
- **[`setup.py`](https://github.com/kvcache-ai/ktransformers/blob/main/setup.py)** — Python entry point that forwards CMake flags when building from source.
- **[`kt-kernel/ext_bindings.cpp`](https://github.com/kvcache-ai/ktransformers/blob/main/kt-kernel/ext_bindings.cpp)** — Implementation of the `CPUInfer` class compiled with the selected CPU variant.

## Summary

- **Always run** [`kt-kernel/scripts/check_cpu_features.py`](https://github.com/kvcache-ai/ktransformers/blob/main/kt-kernel/scripts/check_cpu_features.py) before building to establish your CPU's actual capabilities.
- **CMake detection** in `DetectCPU.cmake` sets `KTRANSFORMERS_CPU_USE_AMX` and `KTRANSFORMERS_CPU_USE_AMX_AVX512`; manually override these when the kernel hides CPU flags.
- **Verify the build** using `nm -D` to inspect for AVX-512 VNNI or AMX symbols, ensuring the optimized kernels were actually compiled.
- **Delete stale caches** ([`CMakeCache.txt`](https://github.com/kvcache-ai/ktransformers/blob/main/CMakeCache.txt)) when switching between CPU variants to prevent silent fallbacks to AVX2.

## Frequently Asked Questions

### How do I know if my CPU supports AMX for kt-kernel?

Run the provided probe script: `python3 kt-kernel/scripts/check_cpu_features.py`. The script checks for `amx_tile`, `amx_int8`, and `amx_bf16` flags in `/proc/cpuinfo`. If you see checkmarks for these flags, your CPU supports AMX. Note that older Linux kernels (below 5.10) may hide these flags even if the hardware supports them.

### Why does kt-kernel crash with "illegal instruction" after installation?

This error occurs when the binary was compiled for a CPU variant (such as AVX-512 or AMX) that your current hardware does not support. Run the CPU feature probe to identify the mismatch, then rebuild from source without the unsupported flags (e.g., remove `-DKTRANSFORMERS_CPU_USE_AMX=ON`) or install the AVX2-only wheel instead.

### Can I force kt-kernel to build for AVX-512 even if CMake doesn't detect it?

Yes. If you know your CPU supports AVX-512 but CMake fails to detect it (often due to `/proc/cpuinfo` formatting issues), manually set the cache variables: `cmake -DKTRANSFORMERS_CPU_USE_AMX_AVX512=ON -DKTRANSFORMERS_CPU_USE_AMX=ON ..`. Always verify the resulting binary contains AVX-512 symbols using `nm -D build/libktkernel.so | grep avx512` to confirm the override worked.

### What should I do if CMake detects AMX but the build still fails?

If CMake sets `KTRANSFORMERS_CPU_USE_AMX=ON` but compilation fails with undefined references to `__builtin_ia32_amx_*`, your compiler toolchain likely lacks AMX intrinsics support. Ensure you are using GCC 11+ or Clang 14+. Alternatively, disable AMX by setting `-DKTRANSFORMERS_CPU_USE_AMX=OFF` and rely on AVX-512 or AVX2 instead.