# How to Handle LoRA Loading and Parameter Application in ComfyUI

> Master ComfyUI LoRA loading and parameter application with LoRAAdapter. Learn to parse tensors and merge deltas or inject additive paths for optimal model performance.

- Repository: [Comfy Org/ComfyUI](https://github.com/Comfy-Org/ComfyUI)
- Tags: how-to-guide
- Published: 2026-02-26

---

**To handle LoRA loading and parameter application in ComfyUI, use the `LoRAAdapter` class to parse checkpoint tensors, then merge deltas via `calculate_weight()` for standard models or inject additive paths via `bypass_forward()` for quantized formats.**

ComfyUI implements a flexible **weight-adapter architecture** that plugs Low-Rank Adaptation (LoRA) modules into PyTorch layers without modifying original weights. This system supports multiple checkpoint formats—Diffusers, Kohya, and custom variants—while accommodating both standard float models and quantized formats like GGUF.

## Understanding the LoRA Architecture

The ComfyUI LoRA system operates through three core phases: discovering tensors in the checkpoint, building an adapter instance, and merging the low-rank delta into base weights.

### The Weight-Adapter Pattern

At the heart of the implementation is the `LoRAAdapter` class defined in [`comfy/weight_adapter/lora.py`](https://github.com/Comfy-Org/ComfyUI/blob/main/comfy/weight_adapter/lora.py). This class encapsulates the logic for parsing variable naming conventions (e.g., `*.lora_up.weight`, `*_lora.up.weight`, `*.lora_B.weight`) and constructing a tuple of `(up, down, alpha, mid, dora_scale, reshape)` tensors. The base implementation in [`comfy/weight_adapter/base.py`](https://github.com/Comfy-Org/ComfyUI/blob/main/comfy/weight_adapter/base.py) provides abstract classes `WeightAdapterBase` and `WeightAdapterTrainBase`, along with utilities like `weight_decompose()` (lines 75-107) for Dora scaling and `pad_tensor_to_shape()` (lines 10-25) for reshape operations.

### Two Operation Modes

ComfyUI supports dual execution strategies depending on model accessibility:

- **Weight-modification mode**: Directly updates the base weight tensor during model loading using `calculate_weight()`. This is invoked internally via `comfy/model_management.apply_weight_adapters()`.
- **Bypass-forward mode**: Computes additive deltas during inference for quantized models where raw weights are inaccessible. This uses `adapter.h(x, base_out)` to calculate `Δy = up(down(x)) * (alpha / rank)`, then adds the result to the base forward output via `WeightAdapterBase.bypass_forward()`.

## Loading LoRA Checkpoints

The entry point for LoRA integration is `LoRAAdapter.load()`, located at lines 47-66 of [`comfy/weight_adapter/lora.py`](https://github.com/Comfy-Org/ComfyUI/blob/main/comfy/weight_adapter/lora.py). This method handles the exhaustive name-matching logic required to support various training tool outputs.

```python
import torch
from comfy.weight_adapter.lora import LoRAAdapter

# Load checkpoint containing LoRA tensors

ckpt = torch.load("my_lora.safetensors", map_location="cpu")
lora_state = ckpt["state_dict"] if "state_dict" in ckpt else ckpt

# Build adapter for a specific layer

# x is the fully-qualified module name, e.g., "model.diffusion_model.input_blocks.0.0"

adapter = LoRAAdapter.load(
    x="model.diffusion_model.input_blocks.0.0",
    lora=lora_state,
    alpha=1.0,               # fallback alpha if not stored in checkpoint

    dora_scale=None,         # optional Dora scaling tensor

    loaded_keys=set()        # populated with consumed keys

)

```

The `load()` method returns `None` if no matching LoRA keys exist for the specified layer, allowing graceful iteration over model parameters.

## Applying LoRA Parameters to Model Weights

Once loaded, the adapter merges low-rank deltas through device-cast tensor operations and optional decomposition handling.

### Weight Modification Mode (Standard)

For standard models with accessible weight tensors, use `calculate_weight()` (lines 35-86 in [`comfy/weight_adapter/lora.py`](https://github.com/Comfy-Org/ComfyUI/blob/main/comfy/weight_adapter/lora.py)) to produce an updated parameter tensor:

```python

# orig_weight is the tensor from the base model's state_dict

new_weight = adapter.calculate_weight(
    weight=orig_weight,
    key="model.diffusion_model.input_blocks.0.0.weight",
    strength=1.0,
    strength_model=1.0,
    offset=0,
    function=torch.nn.functional.tanh,   # post-processing function

    intermediate_dtype=torch.float32
)

# Replace entry in model state dict

state_dict["model.diffusion_model.input_blocks.0.0.weight"] = new_weight

```

The method performs several critical operations:
- **Device casting**: Uses `comfy.model_management.cast_to_device` to align LoRA matrices with the base weight's device and dtype.
- **Alpha scaling**: Computes `alpha = v[2] / mat2.shape[0]` to achieve the classic LoRA scale `α / r`.
- **Tucker decomposition**: Folds optional `mid` tensors into the delta when present.
- **Dora integration**: Applies `weight_decompose()` if `dora_scale` is provided, performing weight-dependent rescaling before delta addition.
- **Reshape padding**: Expands weights to target shapes using `pad_tensor_to_shape()` when reshape entries exist.

### Bypass Forward Mode (Quantized)

For quantized formats where raw weights are unreadable, ComfyUI injects the adapter into the forward pass. The `BypassForwardHook` infrastructure sets adapter attributes like `is_conv`, `conv_dim`, and `multiplier`, then delegates to `bypass_forward()`:

```python

# Configure adapter for convolutional or linear layers

adapter.is_conv = isinstance(module, torch.nn.Conv2d)
adapter.conv_dim = 2
adapter.kw_dict = {"stride": module.stride, "padding": module.padding}
adapter.in_channels = module.in_channels
adapter.out_channels = module.out_channels
adapter.multiplier = 1.0

# During inference, the hook executes:

output = adapter.bypass_forward(module.forward, x)

```

Internally, `LoRAAdapter.h()` (lines 88-121) constructs the additive path:
- For linear layers: `F.linear(x, down_weight)` followed by optional `mid` processing, then `F.linear(..., up_weight)`.
- For convolutions: Applies `F.conv*d` operations with reshaped kernels (lines 74-99).

The computed delta is scaled by `alpha / rank * multiplier` and added to the base forward output.

## End-to-End Implementation Example

The following workflow demonstrates complete LoRA integration for a Stable Diffusion model:

```python
import torch
from comfy.model_management import load_models
from comfy.weight_adapter.lora import LoRAAdapter

# 1. Load base model checkpoint

base_sd = torch.load("sd_v1.5.ckpt", map_location="cpu")
base_state = base_sd["state_dict"] if "state_dict" in base_sd else base_sd

# 2. Load LoRA checkpoint

lora_sd = torch.load("my_lora.safetensors", map_location="cpu")
lora_state = lora_sd["state_dict"] if "state_dict" in lora_sd else lora_sd

# 3. Attach and apply LoRA adapters

for name, weight in base_state.items():
    layer_name = name.rsplit(".", 1)[0]  # Strip ".weight" suffix

    adapter = LoRAAdapter.load(
        x=layer_name,
        lora=lora_state,
        alpha=1.0,
        dora_scale=None,
        loaded_keys=set()
    )
    if adapter:
        merged_weight = adapter.calculate_weight(
            weight,
            key=name,
            strength=1.0,
            strength_model=1.0,
            offset=0,
            function=lambda x: x,
            intermediate_dtype=torch.float32
        )
        base_state[name] = merged_weight

# 4. Build model from merged state dict

model = load_models(base_state)

```

## Summary

- **`LoRAAdapter.load()`** in [`comfy/weight_adapter/lora.py`](https://github.com/Comfy-Org/ComfyUI/blob/main/comfy/weight_adapter/lora.py) handles multi-format checkpoint parsing, supporting Diffusers, Kohya, and custom naming conventions.
- **`calculate_weight()`** merges LoRA deltas into base weights during model loading, supporting alpha scaling, Dora rescaling, and Tucker decomposition.
- **`bypass_forward()`** enables LoRA application for quantized models by computing additive deltas during inference rather than modifying frozen weights.
- **Utility functions** in [`comfy/weight_adapter/base.py`](https://github.com/Comfy-Org/ComfyUI/blob/main/comfy/weight_adapter/base.py) provide tensor padding (`pad_tensor_to_shape`) and Dora weight decomposition (`weight_decompose`).

## Frequently Asked Questions

### How does ComfyUI handle different LoRA checkpoint formats?

ComfyUI normalizes format variations through exhaustive key matching in `LoRAAdapter.load()` (lines 47-66 of [`comfy/weight_adapter/lora.py`](https://github.com/Comfy-Org/ComfyUI/blob/main/comfy/weight_adapter/lora.py)). The method recognizes naming patterns including `*.lora_up.weight`, `*_lora.up.weight`, and `*.lora_B.weight`, ensuring compatibility with checkpoints produced by Diffusers, Kohya-ss, and other training frameworks.

### Can I use LoRA with quantized models in ComfyUI?

Yes. When the base model uses quantized formats like GGUF where weight tensors are inaccessible, ComfyUI switches to **bypass-forward mode**. Instead of modifying weights, the system injects `LoRAAdapter.h()` into the forward pass to compute `Δy = up(down(x)) * (alpha / rank)` and adds this delta to the layer output via `WeightAdapterBase.bypass_forward()`.

### What is the purpose of the alpha parameter in LoRA loading?

The **alpha** parameter controls the scaling magnitude of the low-rank update. In `calculate_weight()`, ComfyUI computes the effective scale as `alpha / rank` (where rank is the dimension of the down-projection matrix). If the checkpoint stores alpha values, these are extracted from the state dict; otherwise, the fallback value passed to `LoRAAdapter.load()` is used.

### How does Dora scaling integrate with LoRA application?

**Dora** (Weight-Decomposed Low-Rank Adaptation) introduces a learnable magnitude vector applied to the base weight direction. When `dora_scale` is provided to `LoRAAdapter.load()`, the `calculate_weight()` method invokes `weight_decompose()` (lines 75-107 in [`comfy/weight_adapter/base.py`](https://github.com/Comfy-Org/ComfyUI/blob/main/comfy/weight_adapter/base.py)) to rescale the weight matrix before adding the low-rank delta, preserving the magnitude adaptation across the adapted layers.