How to Run BitNet Inference on GPU Using CUDA Kernels: A Complete Guide

Run BitNet inference on NVIDIA GPUs by compiling the custom INT2 CUDA kernel from bitnet_kernels.cu, preparing separate FP16 and INT2 checkpoints, and executing the two-phase pipeline via gpu/generate.py with use_kernel=True.

BitNet inference on GPU relies on a specialized CUDA kernel for INT2-quantized matrix multiplication, implemented in the microsoft/BitNet repository. This architecture splits generation into a high-precision prefill phase and a kernel-accelerated decode phase, delivering significant throughput improvements on compatible hardware.

Compiling the INT2 CUDA Kernel

The custom kernel resides in gpu/bitnet_kernels/bitnet_kernels.cu and must be compiled into a shared library before use. The repository provides an automated compilation script for this purpose.

Building libbitnet.so

Navigate to the kernel directory and execute the compilation script:

cd gpu/bitnet_kernels
bash compile.sh          # Produces libbitnet.so in the same directory

The compile.sh script targets compute_80 (Ampere architecture) by default【L1-L2】. If you are running on a different GPU architecture, modify the -gencode flags in compile.sh before building.

Once compiled, the shared library is loaded dynamically via Python's ctypes mechanism. In gpu/model.py, the loading occurs through:

ctypes.CDLL('bitnet_kernels/libbitnet.so')

```【L18-L20】

## Preparing Model Checkpoints for GPU Inference

BitNet requires two distinct checkpoint formats to support the dual-phase inference strategy:

| Phase | Checkpoint File | Format | Purpose |
|-------|----------------|--------|---------|
| **Prefill** | `model_state_fp16.pt` | FP16 | Processes prompt tokens with standard precision |
| **Decode** | `model_state_int2.pt` | INT2 | Executes token generation via CUDA kernel |

### Converting HuggingFace Models

Use the conversion utilities in the `utils/` directory to generate both checkpoints from a standard model:

```bash
python utils/convert-hf-to-gguf-bitnet.py \
    --hf-repo facebook/opt-125m \
    --out-dir ./ckpt

This creates the INT2-quantized weights that match the memory layout expected by BitLinearKernel (see utils/convert-helper-bitnet.py for quantization details). Ensure both .pt files reside in your checkpoint directory before running inference.

Running BitNet Inference with CUDA Acceleration

The high-level entry point gpu/generate.py orchestrates the two-phase workflow, automatically selecting the appropriate precision and kernel activation for each generation stage.

Understanding the Two-Phase Architecture

FastGen.build() constructs two separate Transformer instances:

model_args_prefill = fast.ModelArgs(use_kernel=False)   # FP16, standard linear

model_args_decode = fast.ModelArgs(use_kernel=True)     # INT2, kernel-backed

```【L56-L58】

The `use_kernel` boolean propagates through `Attention` and `FeedForward` layers, which dynamically select the linear layer implementation:

```python
Linear = BitLinearKernel if use_kernel else BitLinear

```【L106-L108】

In [`gpu/model.py`](https://github.com/microsoft/BitNet/blob/main/gpu/model.py), the `BitLinearKernel` class quantizes inputs to INT8, computes per-batch scaling factors, and invokes the low-level `bitnet_int8xint2_linear` wrapper function【L21-L36】【L54-L75】.

### Command-Line Execution

Execute the complete pipeline with a single command after building the kernel and preparing checkpoints:

```bash

# 1. Build the kernel

cd gpu/bitnet_kernels && bash compile.sh && cd ../../

# 2. Run inference

python gpu/generate.py \
    --ckpt-dir ./ckpt \
    --device cuda:0

The script automatically loads both checkpoints, initializes the KV cache, and compiles optimized CUDA graphs for minimal launch overhead during generation.

Programmatic Python API

For integration into custom applications, use the FastGen class directly:

import torch
from gpu import generate as gen

# Initialize the dual-phase generator

fast = gen.FastGen.build(
    ckpt_dir="./ckpt",
    gen_args=gen.GenArgs(gen_length=64, gen_bsz=1, prompt_length=128),
    device="cuda:0",
)

# Tokenize input

prompt = fast.tokenizer.encode("What is the capital of France?", bos=False, eos=False)

# Execute generation with CUDA graphs enabled

stats, answers = fast.generate_all(
    [prompt], 
    use_cuda_graphs=True, 
    use_sampling=False
)

print("Answer:", fast.tokenizer.decode(answers[0]))

The use_kernel=True flag is baked into the decode model constructed by FastGen.build(), ensuring the INT2 kernel handles all token generation steps while the FP16 model processes the initial prompt.

Key Components and Source Code Architecture

Component	Source Location	Function
bitnet_int8xint2_linear	gpu/model.py【L21-L36】	Marshals tensors to the CUDA kernel, handling shape conversion and stream management
BitLinearKernel	gpu/model.py【L54-L75】	PyTorch module that quantizes activations and invokes the INT2 GEMM
FastGen	gpu/generate.py【L56-L99】	Orchestrates prefill/decode models, manages CUDA graphs, and handles token generation
CUDA Graphs	gpu/generate.py【L34-L46】【L91-L103】	Captures static forward-pass graphs to eliminate CPU launch overhead

The architecture intentionally separates prefill (compute-bound, FP16) from decode (memory-bound, INT2), maximizing both prompt processing speed and per-token generation throughput.

Summary

• Compile the kernel using gpu/bitnet_kernels/compile.sh to produce libbitnet.so for your GPU architecture.
• Prepare dual checkpoints: FP16 for prefill (model_state_fp16.pt) and INT2 for decode (model_state_int2.pt).
• Enable kernel acceleration by setting use_kernel=True in ModelArgs for the decode phase, automatically handled by FastGen.
• Execute via gpu/generate.py or the FastGen.build() Python API, ensuring all tensors reside on the same CUDA device.
• Monitor memory usage: The INT2 kernel reduces bandwidth but requires KV cache storage of shape (1, seq_len, kv_heads, 1, head_dim).

Frequently Asked Questions

What NVIDIA GPUs support BitNet CUDA inference?

The default compilation targets Ampere architecture (compute_80, SM 8.0) found in A100, RTX 3090, and RTX A6000 GPUs. Modify the -gencode flags in compile.sh to target other architectures like Turing (SM 7.5) or Ada Lovelace (SM 8.9), though performance optimizations are specifically tuned for Ampere's INT2 tensor core capabilities.

Why does BitNet use separate checkpoints for prefill and decode phases?

The prefill phase processes large prompt contexts where FP16 precision maintains model accuracy and stability, while the decode phase generates single tokens sequentially where the INT2 kernel's memory bandwidth reduction provides maximum throughput. This hybrid approach, implemented in FastGen, balances accuracy and speed without requiring dynamic quantization during generation.

How do I troubleshoot "libbitnet.so not found" errors?

Ensure you have run bash compile.sh from within the gpu/bitnet_kernels directory and that the resulting libbitnet.so file exists in that location. The ctypes.CDLL call in gpu/model.py expects a relative path from the execution directory; run your Python script from the repository root or modify the path in line 20 of model.py to use an absolute path to the shared library.

Can I use the CUDA kernel for models other than the default BitNet architecture?

The BitLinearKernel class in gpu/model.py expects specific INT2 weight layouts produced by the conversion scripts in utils/convert-helper-bitnet.py. While the underlying bitnet_kernels.cu implements a generic INT8xINT2 GEMM, the PyTorch wrapper assumes BitNet-specific quantization scales and group dimensions. Adapting other models requires modifying the weight preparation logic in the conversion utilities.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}