How to Handle LoRA Loading and Parameter Application in ComfyUI

To handle LoRA loading and parameter application in ComfyUI, use the LoRAAdapter class to parse checkpoint tensors, then merge deltas via calculate_weight() for standard models or inject additive paths via bypass_forward() for quantized formats.

ComfyUI implements a flexible weight-adapter architecture that plugs Low-Rank Adaptation (LoRA) modules into PyTorch layers without modifying original weights. This system supports multiple checkpoint formats—Diffusers, Kohya, and custom variants—while accommodating both standard float models and quantized formats like GGUF.

Understanding the LoRA Architecture

The ComfyUI LoRA system operates through three core phases: discovering tensors in the checkpoint, building an adapter instance, and merging the low-rank delta into base weights.

The Weight-Adapter Pattern

At the heart of the implementation is the LoRAAdapter class defined in comfy/weight_adapter/lora.py. This class encapsulates the logic for parsing variable naming conventions (e.g., *.lora_up.weight, *_lora.up.weight, *.lora_B.weight) and constructing a tuple of (up, down, alpha, mid, dora_scale, reshape) tensors. The base implementation in comfy/weight_adapter/base.py provides abstract classes WeightAdapterBase and WeightAdapterTrainBase, along with utilities like weight_decompose() (lines 75-107) for Dora scaling and pad_tensor_to_shape() (lines 10-25) for reshape operations.

Two Operation Modes

ComfyUI supports dual execution strategies depending on model accessibility:

• Weight-modification mode: Directly updates the base weight tensor during model loading using calculate_weight(). This is invoked internally via comfy/model_management.apply_weight_adapters().
• Bypass-forward mode: Computes additive deltas during inference for quantized models where raw weights are inaccessible. This uses adapter.h(x, base_out) to calculate Δy = up(down(x)) * (alpha / rank), then adds the result to the base forward output via WeightAdapterBase.bypass_forward().

Loading LoRA Checkpoints

The entry point for LoRA integration is LoRAAdapter.load(), located at lines 47-66 of comfy/weight_adapter/lora.py. This method handles the exhaustive name-matching logic required to support various training tool outputs.

import torch
from comfy.weight_adapter.lora import LoRAAdapter

# Load checkpoint containing LoRA tensors

ckpt = torch.load("my_lora.safetensors", map_location="cpu")
lora_state = ckpt["state_dict"] if "state_dict" in ckpt else ckpt

# Build adapter for a specific layer

# x is the fully-qualified module name, e.g., "model.diffusion_model.input_blocks.0.0"

adapter = LoRAAdapter.load(
    x="model.diffusion_model.input_blocks.0.0",
    lora=lora_state,
    alpha=1.0,               # fallback alpha if not stored in checkpoint

    dora_scale=None,         # optional Dora scaling tensor

    loaded_keys=set()        # populated with consumed keys

)

The load() method returns None if no matching LoRA keys exist for the specified layer, allowing graceful iteration over model parameters.

Applying LoRA Parameters to Model Weights

Once loaded, the adapter merges low-rank deltas through device-cast tensor operations and optional decomposition handling.

Weight Modification Mode (Standard)

For standard models with accessible weight tensors, use calculate_weight() (lines 35-86 in comfy/weight_adapter/lora.py) to produce an updated parameter tensor:

# orig_weight is the tensor from the base model's state_dict

new_weight = adapter.calculate_weight(
    weight=orig_weight,
    key="model.diffusion_model.input_blocks.0.0.weight",
    strength=1.0,
    strength_model=1.0,
    offset=0,
    function=torch.nn.functional.tanh,   # post-processing function

    intermediate_dtype=torch.float32
)

# Replace entry in model state dict

state_dict["model.diffusion_model.input_blocks.0.0.weight"] = new_weight

The method performs several critical operations:

• Device casting: Uses comfy.model_management.cast_to_device to align LoRA matrices with the base weight's device and dtype.
• Alpha scaling: Computes alpha = v[2] / mat2.shape[0] to achieve the classic LoRA scale α / r.
• Tucker decomposition: Folds optional mid tensors into the delta when present.
• Dora integration: Applies weight_decompose() if dora_scale is provided, performing weight-dependent rescaling before delta addition.
• Reshape padding: Expands weights to target shapes using pad_tensor_to_shape() when reshape entries exist.

Bypass Forward Mode (Quantized)

For quantized formats where raw weights are unreadable, ComfyUI injects the adapter into the forward pass. The BypassForwardHook infrastructure sets adapter attributes like is_conv, conv_dim, and multiplier, then delegates to bypass_forward():

# Configure adapter for convolutional or linear layers

adapter.is_conv = isinstance(module, torch.nn.Conv2d)
adapter.conv_dim = 2
adapter.kw_dict = {"stride": module.stride, "padding": module.padding}
adapter.in_channels = module.in_channels
adapter.out_channels = module.out_channels
adapter.multiplier = 1.0

# During inference, the hook executes:

output = adapter.bypass_forward(module.forward, x)

Internally, LoRAAdapter.h() (lines 88-121) constructs the additive path:

• For linear layers: F.linear(x, down_weight) followed by optional mid processing, then F.linear(..., up_weight).
• For convolutions: Applies F.conv*d operations with reshaped kernels (lines 74-99).

The computed delta is scaled by alpha / rank * multiplier and added to the base forward output.

End-to-End Implementation Example

The following workflow demonstrates complete LoRA integration for a Stable Diffusion model:

import torch
from comfy.model_management import load_models
from comfy.weight_adapter.lora import LoRAAdapter

# 1. Load base model checkpoint

base_sd = torch.load("sd_v1.5.ckpt", map_location="cpu")
base_state = base_sd["state_dict"] if "state_dict" in base_sd else base_sd

# 2. Load LoRA checkpoint

lora_sd = torch.load("my_lora.safetensors", map_location="cpu")
lora_state = lora_sd["state_dict"] if "state_dict" in lora_sd else lora_sd

# 3. Attach and apply LoRA adapters

for name, weight in base_state.items():
    layer_name = name.rsplit(".", 1)[0]  # Strip ".weight" suffix

    adapter = LoRAAdapter.load(
        x=layer_name,
        lora=lora_state,
        alpha=1.0,
        dora_scale=None,
        loaded_keys=set()
    )
    if adapter:
        merged_weight = adapter.calculate_weight(
            weight,
            key=name,
            strength=1.0,
            strength_model=1.0,
            offset=0,
            function=lambda x: x,
            intermediate_dtype=torch.float32
        )
        base_state[name] = merged_weight

# 4. Build model from merged state dict

model = load_models(base_state)

Summary

• LoRAAdapter.load() in comfy/weight_adapter/lora.py handles multi-format checkpoint parsing, supporting Diffusers, Kohya, and custom naming conventions.
• calculate_weight() merges LoRA deltas into base weights during model loading, supporting alpha scaling, Dora rescaling, and Tucker decomposition.
• bypass_forward() enables LoRA application for quantized models by computing additive deltas during inference rather than modifying frozen weights.
• Utility functions in comfy/weight_adapter/base.py provide tensor padding (pad_tensor_to_shape) and Dora weight decomposition (weight_decompose).

Frequently Asked Questions

How does ComfyUI handle different LoRA checkpoint formats?

ComfyUI normalizes format variations through exhaustive key matching in LoRAAdapter.load() (lines 47-66 of comfy/weight_adapter/lora.py). The method recognizes naming patterns including *.lora_up.weight, *_lora.up.weight, and *.lora_B.weight, ensuring compatibility with checkpoints produced by Diffusers, Kohya-ss, and other training frameworks.

Can I use LoRA with quantized models in ComfyUI?

Yes. When the base model uses quantized formats like GGUF where weight tensors are inaccessible, ComfyUI switches to bypass-forward mode. Instead of modifying weights, the system injects LoRAAdapter.h() into the forward pass to compute Δy = up(down(x)) * (alpha / rank) and adds this delta to the layer output via WeightAdapterBase.bypass_forward().

What is the purpose of the alpha parameter in LoRA loading?

The alpha parameter controls the scaling magnitude of the low-rank update. In calculate_weight(), ComfyUI computes the effective scale as alpha / rank (where rank is the dimension of the down-projection matrix). If the checkpoint stores alpha values, these are extracted from the state dict; otherwise, the fallback value passed to LoRAAdapter.load() is used.

How does Dora scaling integrate with LoRA application?

Dora (Weight-Decomposed Low-Rank Adaptation) introduces a learnable magnitude vector applied to the base weight direction. When dora_scale is provided to LoRAAdapter.load(), the calculate_weight() method invokes weight_decompose() (lines 75-107 in comfy/weight_adapter/base.py) to rescale the weight matrix before adding the low-rank delta, preserving the magnitude adaptation across the adapted layers.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}