# How to Run BitNet Inference on GPU Using CUDA Kernels: A Complete Guide

> Learn how to run BitNet inference on GPU with CUDA kernels. Follow our guide to compile custom INT2 CUDA kernels and execute the two phase pipeline for faster model performance.

- Repository: [Microsoft/BitNet](https://github.com/microsoft/BitNet)
- Tags: how-to-guide
- Published: 2026-03-13

---

**Run BitNet inference on NVIDIA GPUs by compiling the custom INT2 CUDA kernel from `bitnet_kernels.cu`, preparing separate FP16 and INT2 checkpoints, and executing the two-phase pipeline via [`gpu/generate.py`](https://github.com/microsoft/BitNet/blob/main/gpu/generate.py) with `use_kernel=True`.**

BitNet inference on GPU relies on a specialized CUDA kernel for INT2-quantized matrix multiplication, implemented in the **microsoft/BitNet** repository. This architecture splits generation into a high-precision prefill phase and a kernel-accelerated decode phase, delivering significant throughput improvements on compatible hardware.

## Compiling the INT2 CUDA Kernel

The custom kernel resides in `gpu/bitnet_kernels/bitnet_kernels.cu` and must be compiled into a shared library before use. The repository provides an automated compilation script for this purpose.

### Building libbitnet.so

Navigate to the kernel directory and execute the compilation script:

```bash
cd gpu/bitnet_kernels
bash compile.sh          # Produces libbitnet.so in the same directory

```

The [`compile.sh`](https://github.com/microsoft/BitNet/blob/main/compile.sh) script targets `compute_80` (Ampere architecture) by default【L1-L2】. If you are running on a different GPU architecture, modify the `-gencode` flags in [`compile.sh`](https://github.com/microsoft/BitNet/blob/main/compile.sh) before building.

Once compiled, the shared library is loaded dynamically via Python's `ctypes` mechanism. In [`gpu/model.py`](https://github.com/microsoft/BitNet/blob/main/gpu/model.py), the loading occurs through:

```python
ctypes.CDLL('bitnet_kernels/libbitnet.so')

```【L18-L20】

## Preparing Model Checkpoints for GPU Inference

BitNet requires two distinct checkpoint formats to support the dual-phase inference strategy:

| Phase | Checkpoint File | Format | Purpose |
|-------|----------------|--------|---------|
| **Prefill** | `model_state_fp16.pt` | FP16 | Processes prompt tokens with standard precision |
| **Decode** | `model_state_int2.pt` | INT2 | Executes token generation via CUDA kernel |

### Converting HuggingFace Models

Use the conversion utilities in the `utils/` directory to generate both checkpoints from a standard model:

```bash
python utils/convert-hf-to-gguf-bitnet.py \
    --hf-repo facebook/opt-125m \
    --out-dir ./ckpt

```

This creates the INT2-quantized weights that match the memory layout expected by `BitLinearKernel` (see [`utils/convert-helper-bitnet.py`](https://github.com/microsoft/BitNet/blob/main/utils/convert-helper-bitnet.py) for quantization details). Ensure both `.pt` files reside in your checkpoint directory before running inference.

## Running BitNet Inference with CUDA Acceleration

The high-level entry point [`gpu/generate.py`](https://github.com/microsoft/BitNet/blob/main/gpu/generate.py) orchestrates the two-phase workflow, automatically selecting the appropriate precision and kernel activation for each generation stage.

### Understanding the Two-Phase Architecture

`FastGen.build()` constructs two separate `Transformer` instances:

```python
model_args_prefill = fast.ModelArgs(use_kernel=False)   # FP16, standard linear

model_args_decode = fast.ModelArgs(use_kernel=True)     # INT2, kernel-backed

```【L56-L58】

The `use_kernel` boolean propagates through `Attention` and `FeedForward` layers, which dynamically select the linear layer implementation:

```python
Linear = BitLinearKernel if use_kernel else BitLinear

```【L106-L108】

In [`gpu/model.py`](https://github.com/microsoft/BitNet/blob/main/gpu/model.py), the `BitLinearKernel` class quantizes inputs to INT8, computes per-batch scaling factors, and invokes the low-level `bitnet_int8xint2_linear` wrapper function【L21-L36】【L54-L75】.

### Command-Line Execution

Execute the complete pipeline with a single command after building the kernel and preparing checkpoints:

```bash

# 1. Build the kernel

cd gpu/bitnet_kernels && bash compile.sh && cd ../../

# 2. Run inference

python gpu/generate.py \
    --ckpt-dir ./ckpt \
    --device cuda:0

```

The script automatically loads both checkpoints, initializes the KV cache, and compiles optimized CUDA graphs for minimal launch overhead during generation.

### Programmatic Python API

For integration into custom applications, use the `FastGen` class directly:

```python
import torch
from gpu import generate as gen

# Initialize the dual-phase generator

fast = gen.FastGen.build(
    ckpt_dir="./ckpt",
    gen_args=gen.GenArgs(gen_length=64, gen_bsz=1, prompt_length=128),
    device="cuda:0",
)

# Tokenize input

prompt = fast.tokenizer.encode("What is the capital of France?", bos=False, eos=False)

# Execute generation with CUDA graphs enabled

stats, answers = fast.generate_all(
    [prompt], 
    use_cuda_graphs=True, 
    use_sampling=False
)

print("Answer:", fast.tokenizer.decode(answers[0]))

```

The `use_kernel=True` flag is baked into the decode model constructed by `FastGen.build()`, ensuring the INT2 kernel handles all token generation steps while the FP16 model processes the initial prompt.

## Key Components and Source Code Architecture

| Component | Source Location | Function |
|-----------|----------------|----------|
| **`bitnet_int8xint2_linear`** | [`gpu/model.py`](https://github.com/microsoft/BitNet/blob/main/gpu/model.py)【L21-L36】 | Marshals tensors to the CUDA kernel, handling shape conversion and stream management |
| **`BitLinearKernel`** | [`gpu/model.py`](https://github.com/microsoft/BitNet/blob/main/gpu/model.py)【L54-L75】 | PyTorch module that quantizes activations and invokes the INT2 GEMM |
| **`FastGen`** | [`gpu/generate.py`](https://github.com/microsoft/BitNet/blob/main/gpu/generate.py)【L56-L99】 | Orchestrates prefill/decode models, manages CUDA graphs, and handles token generation |
| **CUDA Graphs** | [`gpu/generate.py`](https://github.com/microsoft/BitNet/blob/main/gpu/generate.py)【L34-L46】【L91-L103】 | Captures static forward-pass graphs to eliminate CPU launch overhead |

The architecture intentionally separates prefill (compute-bound, FP16) from decode (memory-bound, INT2), maximizing both prompt processing speed and per-token generation throughput.

## Summary

- **Compile the kernel** using [`gpu/bitnet_kernels/compile.sh`](https://github.com/microsoft/BitNet/blob/main/gpu/bitnet_kernels/compile.sh) to produce `libbitnet.so` for your GPU architecture.
- **Prepare dual checkpoints**: FP16 for prefill (`model_state_fp16.pt`) and INT2 for decode (`model_state_int2.pt`).
- **Enable kernel acceleration** by setting `use_kernel=True` in `ModelArgs` for the decode phase, automatically handled by `FastGen`.
- **Execute via** [`gpu/generate.py`](https://github.com/microsoft/BitNet/blob/main/gpu/generate.py) or the `FastGen.build()` Python API, ensuring all tensors reside on the same CUDA device.
- **Monitor memory usage**: The INT2 kernel reduces bandwidth but requires KV cache storage of shape `(1, seq_len, kv_heads, 1, head_dim)`.

## Frequently Asked Questions

### What NVIDIA GPUs support BitNet CUDA inference?

The default compilation targets Ampere architecture (`compute_80`, SM 8.0) found in A100, RTX 3090, and RTX A6000 GPUs. Modify the `-gencode` flags in [`compile.sh`](https://github.com/microsoft/BitNet/blob/main/compile.sh) to target other architectures like Turing (SM 7.5) or Ada Lovelace (SM 8.9), though performance optimizations are specifically tuned for Ampere's INT2 tensor core capabilities.

### Why does BitNet use separate checkpoints for prefill and decode phases?

The prefill phase processes large prompt contexts where FP16 precision maintains model accuracy and stability, while the decode phase generates single tokens sequentially where the INT2 kernel's memory bandwidth reduction provides maximum throughput. This hybrid approach, implemented in `FastGen`, balances accuracy and speed without requiring dynamic quantization during generation.

### How do I troubleshoot "libbitnet.so not found" errors?

Ensure you have run `bash compile.sh` from within the `gpu/bitnet_kernels` directory and that the resulting `libbitnet.so` file exists in that location. The `ctypes.CDLL` call in [`gpu/model.py`](https://github.com/microsoft/BitNet/blob/main/gpu/model.py) expects a relative path from the execution directory; run your Python script from the repository root or modify the path in line 20 of [`model.py`](https://github.com/microsoft/BitNet/blob/main/model.py) to use an absolute path to the shared library.

### Can I use the CUDA kernel for models other than the default BitNet architecture?

The `BitLinearKernel` class in [`gpu/model.py`](https://github.com/microsoft/BitNet/blob/main/gpu/model.py) expects specific INT2 weight layouts produced by the conversion scripts in [`utils/convert-helper-bitnet.py`](https://github.com/microsoft/BitNet/blob/main/utils/convert-helper-bitnet.py). While the underlying `bitnet_kernels.cu` implements a generic INT8xINT2 GEMM, the PyTorch wrapper assumes BitNet-specific quantization scales and group dimensions. Adapting other models requires modifying the weight preparation logic in the conversion utilities.