# How Unsloth's FastLoRA Kernel Optimizes Training Speed and VRAM Usage

> Discover how Unsloth's FastLoRA kernel optimizes deep learning training by reducing VRAM by 30% and boosting speed 2x through fused operations and on-the-fly de-quantization.

- Repository: [Unsloth AI/unsloth](https://github.com/unslothai/unsloth)
- Tags: deep-dive
- Published: 2026-03-20

---

**Unsloth's FastLoRA kernel fuses base weight multiplication with low-rank LoRA updates into a single custom autograd operation, eliminating intermediate tensors and reducing VRAM usage by up to 30% while delivering 1.5–2× training speedups through on-the-fly de-quantization and specialized GEMV shortcuts.**

The `unslothai/unsloth` library reimplements standard PEFT LoRA layers as highly optimized CUDA kernels. Unlike conventional implementations that materialize full weight matrices in GPU memory, FastLoRA keeps quantized weights compressed and computes LoRA adjustments in fused kernels, drastically cutting memory allocations and kernel launch overhead.

## Fused LoRA Computation: The Core Speedup

The FastLoRA implementation replaces standard PyTorch linear layers with custom autograd functions located in [`unsloth/kernels/fast_lora.py`](https://github.com/unslothai/unsloth/blob/main/unsloth/kernels/fast_lora.py). Three primary classes handle different transformer components: **`LoRA_MLP`** for feed-forward networks, **`LoRA_QKV`** for attention projections, and **`LoRA_W`** for output projections.

These classes implement **`torch_amp_custom_fwd`** and **`torch_amp_custom_bwd`** decorators to automatically handle automatic mixed precision (Amp) contexts. The forward pass never materializes the full LoRA-augmented weight matrix. Instead, it computes `X @ (W + A @ B)` by performing the base matrix multiplication and the LoRA contribution separately, then fuses them with a single `addmm` or `addmv` call—eliminating intermediate activation tensors that would otherwise consume VRAM.

```python
class LoRA_MLP(torch.autograd.Function):
    @torch_amp_custom_fwd
    def forward(ctx, X, W, W_quant, lora_A, lora_B, ...):
        # De-quantize on-the-fly and fused matmul

        output = matmul_lora(X, W, W_quant, lora_A, lora_B)
        return output
    
    @torch_amp_custom_bwd
    def backward(ctx, grad_output):
        # Optimized gradient computation for base + LoRA weights

        ...

```

## VRAM Reduction: Global Buffers and Fast De-Quantization

Memory efficiency in FastLoRA relies on **global buffers** and **on-the-fly de-quantization** implemented in [`unsloth/kernels/utils.py`](https://github.com/unslothai/unsloth/blob/main/unsloth/kernels/utils.py). The system allocates reusable GPU buffers (`WEIGHT_BUFFERS` and `ABSMAX_BUFFERS`, defined at lines 21–34) once per device, then reuses them across forward passes. This prevents the repeated `torch.empty` allocations that typically fragment GPU memory during training.

The **`fast_dequantize`** function handles 4-bit (NF4) and FP8 weights without permanent de-compression. When weights are quantized, only the specific slice required for the current operation is de-quantized using Triton kernels like `cdequantize_blockwise_fp32`. The rest remains in compressed storage, cutting the weight memory footprint by roughly 30% compared to standard de-quantization approaches that expand full matrices before multiplication.

```python
def fast_dequantize(W, quant_state=None, out=None, use_global_buffer=False):
    if isinstance(W, Float8Tensor):
        return W.dequantize()
    if quant_state is None:
        return W
    # Triton-driven NF4 de-quantization using global buffers

    ...

```

## Computational Shortcuts: GEMV vs GEMM

FastLoRA automatically detects batch dimensions to select the optimal compute kernel. In [`unsloth/kernels/utils.py`](https://github.com/unslothai/unsloth/blob/main/unsloth/kernels/utils.py), the **`fast_linear_forward`** function checks if `X.shape[1] == 1` (indicating a single-token sequence in decoder-only models). When this condition is met, it routes to **`fast_gemv`** instead of a full GEMM operation.

This **batched GEMV shortcut** skips the extra batch dimension overhead, reducing operation count substantially during inference and training on single-token steps. For longer sequences, it falls back to standard matrix multiplication, ensuring optimal performance across varying sequence lengths without manual tuning.

```python
def fast_linear_forward(proj, X, temp_lora=None, out=None):
    W, W_quant, lora_A, lora_B, ... = get_lora_parameters_bias(proj)
    if X.shape[1] == 1:
        out = fast_gemv(X, W, W_quant, out=out)  # Vector-matrix multiply

    else:
        out = torch_matmul(X, W.t())            # Standard GEMM

    # Fused LoRA addition via single addmm/addmv call

```

## Implementation Example: Using FastLoRA in Training

The kernels integrate seamlessly into standard Hugging Face workflows. The `FastLoraModel` class automatically applies these optimizations when loading models with quantization enabled.

```python
import torch
from unsloth import FastLoraModel

# Load with 4-bit quantization

model = FastLoraModel.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    load_in_4bit=True,
    device_map="auto"
)

# Attach LoRA adapter

model.add_adapter(name="default", r=8, alpha=16)

# Forward pass uses FastLoRA kernels automatically

inputs = torch.randint(0, model.config.vocab_size, (1, 1)).to(model.device)
logits = model(inputs)

# Backpropagation triggers optimized custom backward

loss = logits.mean()
loss.backward()

```

During both forward and backward passes, `get_lora_parameters` fetches base weights, quantization states, and LoRA matrices without copying data. When adapters are disabled or merged, the function returns `None` for LoRA components, allowing the code to bypass all LoRA-specific computation and de-quantization overhead entirely.

## Summary

- **Custom autograd functions** (`LoRA_MLP`, `LoRA_QKV`, `LoRA_W`) in [`fast_lora.py`](https://github.com/unslothai/unsloth/blob/main/fast_lora.py) fuse base and LoRA computations into single kernel calls, eliminating intermediate tensors.
- **Global buffers** (`WEIGHT_BUFFERS`, `ABSMAX_BUFFERS`) eliminate repeated memory allocations, while **fast de-quantization** keeps 4-bit/FP8 weights compressed until the moment of computation.
- **GEMV shortcuts** automatically replace GEMM operations when `seq_len == 1`, significantly reducing compute overhead for single-token processing.
- **Mixed-precision decorators** (`torch_amp_custom_fwd`, `torch_amp_custom_bwd`) ensure all kernels run at optimal precision without manual casting.
- Compared to standard PEFT implementations, FastLoRA achieves **1.5–2× speedups** and **~30% VRAM reduction** through these fused, memory-efficient operations.

## Frequently Asked Questions

### How does FastLoRA reduce VRAM usage compared to standard PEFT LoRA?

FastLoRA reduces VRAM by avoiding the materialization of full de-quantized weight matrices. The **`fast_dequantize`** function in [`utils.py`](https://github.com/unslothai/unsloth/blob/main/utils.py) decompresses only the specific weight slices needed for the current operation using Triton kernels, while **global buffers** prevent repeated allocation of temporary tensors. This keeps 4-bit and FP8 weights in their compressed forms throughout training, eliminating the permanent 16-bit/32-bit weight copies that standard implementations require.

### What is the difference between `fast_gemv` and standard GEMM in Unsloth?

The **`fast_gemv`** function, defined in [`unsloth/kernels/utils.py`](https://github.com/unslothai/unsloth/blob/main/unsloth/kernels/utils.py), is optimized for vector-matrix multiplication when the sequence length equals one (`seq_len == 1`). Instead of performing a full batched GEMM operation that includes unnecessary batch dimension overhead, `fast_gemv` calls specialized CUDA kernels (such as BitsAndBytes 4-bit GEMM primitives) directly on the quantized weight matrix. This shortcut significantly reduces operation count and latency during autoregressive generation and single-token training steps.

### Does FastLoRA support mixed-precision training?

Yes, all FastLoRA kernels are **mixed-precision aware** through the `torch_amp_custom_fwd` and `torch_amp_custom_bwd` decorators applied to `LoRA_MLP`, `LoRA_QKV`, and `LoRA_W` in [`fast_lora.py`](https://github.com/unslothai/unsloth/blob/main/fast_lora.py). These decorators automatically respect PyTorch's autocasting context, allowing the kernels to run in FP16 or BF16 when enabled while maintaining FP32 master weights for numerical stability. This integration eliminates manual casting code and ensures maximum throughput on modern Ampere or newer GPUs.

### Which model components use the `LoRA_MLP` and `LoRA_QKV` kernels?

According to the source code in [`fast_lora.py`](https://github.com/unslothai/unsloth/blob/main/fast_lora.py), **`LoRA_MLP`** handles LoRA-augmented feed-forward layers (typically SwiGLU or GeGLU activations), while **`LoRA_QKV`** manages the Query, Key, and Value projections in attention mechanisms. The **`LoRA_W`** class applies to output projection layers. These kernels are dispatched automatically based on layer type during model initialization, requiring no manual configuration from the user.