How to Implement ControlNet and T2I-Adapter Preprocessing in ComfyUI: A Complete Technical Guide

ControlNet and T2I-Adapter preprocessing in ComfyUI is implemented through callable lambdas attached during model loading in comfy/controlnet.py, which transform hint images during the get_control execution before they reach the diffusion model.

ControlNet and T2I-Adapter preprocessing pipelines in ComfyUI enable precise visual conditioning for diffusion models by transforming hint images before inference. This guide examines the implementation details in the Comfy-Org/ComfyUI repository, focusing on how comfy/controlnet.py manages preprocessing logic for both ControlNet checkpoints and T2I-Adapter models.

How ControlNet Preprocessing Works in ComfyUI

The ControlNet preprocessing pipeline centers on the ControlNet class and its preprocess_image callable. When you load a ControlNet checkpoint through nodes like ControlNetLoader or DiffControlNetLoader (defined in nodes.py lines 33-66), the system invokes load_controlnet in comfy/controlnet.py to instantiate the model and attach the appropriate preprocessing function.

Preprocessing Function Selection in load_controlnet_sd35

The preprocessing lambda is selected based on the checkpoint type during loading. In load_controlnet_sd35 (lines 61-68 of comfy/controlnet.py), the code detects whether the model expects Canny edges, depth maps, or raw input, then constructs the appropriate transformation:

# From comfy/controlnet.py

preprocess_image = lambda a: a                # default – no change

if canny_cnet:
    preprocess_image = lambda a: (a * 255 * 0.5 + 0.5)   # map [-1,1] → [0,1] for Canny

elif depth_cnet:
    preprocess_image = lambda a: 1.0 - a                # invert depth

control = ControlNetSD35(..., preprocess_image=preprocess_image)

This lambda is stored in the ControlNet instance via the constructor (lines 201-217) and remains dormant until inference time.

Application Point in ControlNet.get_control

The preprocessing occurs inside ControlNet.get_control at lines 244-251. First, the hint image is upscaled to match the latent space dimensions using the compression ratio, then the stored preprocess_image callable is applied:

# From ControlNet.get_control (comfy/controlnet.py)

self.cond_hint = comfy.utils.common_upscale(
    self.cond_hint_original,
    x_noisy.shape[-1] * compression_ratio,
    x_noisy.shape[-2] * compression_ratio,
    self.upscale_algorithm,
    "center")
self.cond_hint = self.preprocess_image(self.cond_hint)   # Preprocessing executed here

After preprocessing, the tensor optionally passes through VAE encoding and concatenation steps (lines 254-263) before reaching the underlying control model.

T2I-Adapter Preprocessing Architecture

T2I-Adapters follow a distinct preprocessing path implemented in the T2IAdapter class (starting at line 851 in comfy/controlnet.py). Unlike ControlNet models that rely on simple lambdas, T2I-Adapters perform structural transformations including dimension rounding and channel collapsing within their get_control method.

The T2IAdapter.get_control Method

The preprocessing flow in T2IAdapter.get_control (lines 86-98) handles spatial alignment and tensor formatting before adapter inference:

# Inside T2IAdapter.get_control (comfy/controlnet.py)

width, height = self.scale_image_to(
    x_noisy.shape[3] * self.compression_ratio,
    x_noisy.shape[2] * self.compression_ratio)
self.cond_hint = comfy.utils.common_upscale(
    self.cond_hint_original, width, height,
    self.upscale_algorithm, "center").float().to(self.device)

# Collapse to single channel if required

if self.channels_in == 1 and self.cond_hint.shape[1] > 1:
    self.cond_hint = torch.mean(self.cond_hint, 1, keepdim=True)

# Run adapter inference

self.t2i_model.to(x_noisy.dtype)
self.t2i_model.to(self.device)
self.control_input = self.t2i_model(self.cond_hint.to(x_noisy.dtype))

The scale_image_to method (lines 63-68) ensures dimensions are multiples of the adapter's internal unshuffle_amount, preventing alignment errors during downsampling operations.

Loading Adapter Checkpoints

The load_t2i_adapter function (lines 9-15 and 58-66) inspects checkpoint keys to instantiate the correct adapter architecture (Adapter or Adapter_light from comfy/t2i_adapter/adapter.py):

# From load_t2i_adapter (comfy/controlnet.py)

if 'adapter' in t2i_data:
    t2i_data = t2i_data['adapter']

# Architecture detection

if "body.0.in_conv.weight" in keys:
    model_ad = comfy.t2i_adapter.adapter.Adapter_light(...)
elif 'conv_in.weight' in keys:
    model_ad = comfy.t2i_adapter.adapter.Adapter(...)

return T2IAdapter(model_ad, model_ad.input_channels,
                 compression_ratio, upscale_algorithm)

Implementing ControlNet and T2I-Adapter Preprocessing in Practice

Loading and Applying ControlNet

To implement ControlNet preprocessing programmatically, load the checkpoint and apply it through ControlNetApplyAdvanced:

import folder_paths, comfy

# Load ControlNet checkpoint

cn_path = folder_paths.get_full_path_or_raise("controlnet", "canny_controlnet.pth")
control_net = comfy.controlnet.load_controlnet(cn_path)  # Uses load_controlnet_sd35 internally

# Load hint image (H, W, 3) → (1, 3, H, W)

hint_image = comfy.utils.load_image("canny_map.png")

# Apply with strength scheduling

apply_node = comfy.nodes.ControlNetApplyAdvanced()
positive, negative = apply_node.apply_controlnet(
    positive, negative,
    control_net, hint_image,
    strength=1.0,
    start_percent=0.0,
    end_percent=1.0)

Loading and Applying T2I-Adapter

T2I-Adapters use identical application interfaces despite different internal preprocessing:

import folder_paths, comfy

# Load adapter checkpoint

adapter_path = folder_paths.get_full_path_or_raise("controlnet", "t2i_adapter_sd15.pth")
t2i_adapter = comfy.controlnet.load_t2i_adapter(adapter_path)

# Apply through the same node interface

apply_node = comfy.nodes.ControlNetApplyAdvanced()
positive, negative = apply_node.apply_controlnet(
    positive, negative,
    t2i_adapter, hint_image,
    strength=0.7,
    start_percent=0.0,
    end_percent=1.0)

The T2IAdapter object contains the preprocessing described in section 2, so ControlNetApplyAdvanced (lines 998-1030 in nodes.py) works without modification.

Combining Multiple Control Mechanisms

You can stack ControlNet and T2I-Adapter instances using set_previous_controlnet, merged internally by ControlNet.control_merge (lines 555-595):

# Load distinct control mechanisms

cn_canny = comfy.controlnet.load_controlnet(canny_path)
cn_adapter = comfy.controlnet.load_t2i_adapter(adapter_path)

# Chain adapters: second argument treats first as previous_controlnet

cn_canny.set_previous_controlnet(cn_adapter)

# Single application merges both controls

positive, negative = apply_node.apply_controlnet(
    positive, negative,
    cn_canny, hint_image,
    strength=1.0,
    start_percent=0.0,
    end_percent=1.0)

Critical Source Files for Preprocessing Logic

• comfy/controlnet.py: Contains ControlBase, ControlNet, T2IAdapter, all loader functions (load_controlnet, load_t2i_adapter), preprocessing lambdas, and merging logic.
• nodes.py: Defines UI nodes including ControlNetLoader, DiffControlNetLoader, and ControlNetApplyAdvanced that expose preprocessing classes to the node graph.
• comfy/t2i_adapter/adapter.py: Implements lightweight adapter architectures (Adapter, Adapter_light) used by load_t2i_adapter.
• comfy_extras/nodes_replacements.py: Compatibility shim for legacy T2IAdapterLoader node names.

Summary

• ControlNet preprocessing relies on lambdas attached during loading in load_controlnet_sd35, applied in ControlNet.get_control after upscaling but before model inference.
• T2I-Adapter preprocessing handles dimension rounding via scale_image_to and optional channel collapsing within T2IAdapter.get_control, producing conditioning tensors through the adapter model.
• Both systems share the compression ratio parameter to align hint images with latent space dimensions.
• ControlNetApplyAdvanced provides a unified interface for applying either mechanism, supporting strength scheduling and temporal control.
• Multiple controls can be chained via set_previous_controlnet and merged through control_merge respecting individual strengths and masks.

Frequently Asked Questions

Where is ControlNet preprocessing defined in ComfyUI?

ControlNet preprocessing is defined in comfy/controlnet.py, specifically within the load_controlnet_sd35 function (lines 61-68) where preprocessing lambdas are selected, and in ControlNet.get_control (lines 244-251) where these functions are executed on the upscaled hint image.

How does T2I-Adapter preprocessing differ from ControlNet?

T2I-Adapter preprocessing, implemented in T2IAdapter.get_control (lines 86-98), includes additional structural transformations such as dimension rounding to unshuffle_amount multiples and channel collapsing for single-channel adapters, whereas ControlNet preprocessing typically involves simple value transformations like scaling or inversion through lambdas.

Can I customize the preprocessing lambda for ControlNet models?

Yes, you can modify the preprocess_image lambda passed to the ControlNet constructor in comfy/controlnet.py. The default loaders in load_controlnet_sd35 select from predefined lambdas (identity, Canny scaling, or depth inversion), but you can extend this logic or instantiate ControlNet directly with a custom callable.

What is the compression_ratio parameter used for?

The compression_ratio parameter determines the spatial scaling factor between the latent space and the hint image resolution. Both ControlNet and T2I-Adapter use this value in common_upscale calls to ensure the conditioning image matches the dimensions of the noisy latent tensor (x_noisy) before preprocessing and model inference occur.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}