Using ControlNet with Sana for Controllable Image Generation: A Complete Implementation Guide

Sana's ControlNet integration enables spatial conditioning via the SanaControlNetPipeline class, which injects VAE-encoded control signals (such as HED edge maps) into the diffusion sampling loop alongside text embeddings to guide image generation.

The NVlabs/Sana repository extends its efficient text-to-image diffusion backbone with a ControlNet-compatible pipeline that accepts spatial priors like scribbles or edge maps. This implementation, found in app/sana_controlnet_pipeline.py, orchestrates the conversion of reference images into latent control signals that condition the generation process without requiring model retraining.

Architecture Overview

Sana's ControlNet architecture consists of three integrated components that transform a reference image into a conditioning signal:

Component	Role	Key Source Location
SanaControlNetPipeline	Wraps the diffusion model, VAE, and text encoder to orchestrate the sampling loop with control signal injection	app/sana_controlnet_pipeline.py (lines 98‑156)
Control Signal Generation	Creates raster maps (e.g., HED-based scribbles) from reference images using edge detection and post-processing	tools/controlnet/utils.py (lines 37‑83)
Signal Transformation	Resizes and normalizes maps to match latent spatial resolution expected by the diffusion model	tools/controlnet/utils.py (lines 15‑34)

During inference, the control map undergoes VAE encoding via vae_encode and passes as control_signal inside model_kwargs (see the forward method at lines 289‑303). The diffusion sampler then conditions on both the text embedding (caption_embs) and the control latent, enabling sketch-to-image or edge-to-image synthesis.

Preparing the Control Signal

Edge Detection with HED

The pipeline generates spatial conditioning using the HED (Holistically-nested Edge Detection) detector. The get_scribble_map function in tools/controlnet/utils.py (lines 52‑82) processes reference images through non-maximum suppression, Gaussian blur, and line thickness adjustment:

control_signal = get_scribble_map(
    input_image=ref_image,
    det="Scribble_HED",                 # HED edge detector

    detect_resolution=int(hw.min()),      # matches target latent size

    thickness=sketch_thickness,
)

This creates a binary edge map that preserves the structural composition of the reference image while allowing adjustable stroke thickness for different artistic effects.

Transformation and Normalization

Before injection into the diffusion model, the transform_control_signal function (lines 15‑34 in tools/controlnet/utils.py) converts the map to a 3-channel RGB tensor, applies BICUBIC resizing, center-cropping, and normalization to the [-1, 1] range:

control_signal = transform_control_signal(control_signal, hw) \
                 .to(self.device).to(self.weight_dtype)

This ensures the control signal dimensions match the latent spatial resolution (HW) expected by the VAE encoder.

Pipeline Initialization

The SanaControlNetPipeline class loads configuration via pyrallis and constructs the VAE, tokenizer, and core Sana model through the build_model function:

from app.sana_controlnet_pipeline import SanaControlNetPipeline

pipe = SanaControlNetPipeline(args.config)          # loads YAML configuration

pipe.from_pretrained(pipe.args.model_path)           # loads checkpoint weights

The pipeline supports both Flow-Euler (deterministic ODE solver) and DPM-Solver (multistep stochastic) samplers, selected via the vis_sampler configuration parameter.

The Inference Workflow

Step 1: Control Map Generation and Encoding

The reference image first converts to a scribble map, then encodes to latent space using the VAE:

control_signal_latent = vae_encode(
    self.config.vae.vae_type, self.vae,
    control_signal, self.config.vae.sample_posterior, self.device
)

This latent representation allows the diffusion model to attend to spatial features at the same resolution as the noise latents.

Step 2: Conditioned Sampling

The pipeline injects the control latent via model_kwargs["control_signal"] during the sampling loop (lines 289‑303). Depending on the vis_sampler setting, the pipeline executes either:

• Flow-Euler (FlowEuler) — A deterministic ODE solver implemented at lines 310‑322
• DPM-Solver (DPMS) — A multistep stochastic solver implemented at lines 322‑342

Both samplers receive the conditioning tuple (condition=caption_embs, uncondition=null_y) plus the control signal. The guidance type (classifier-free or classifier-free_PAG) is selected automatically via guidance_type_select (line 45).

Step 3: Decoding and Post-Processing

After sampling completes, the latent tensor decodes to pixel space and resizes to match original reference dimensions:

sample = vae_decode(self.config.vae.vae_type, self.vae, sample)
sample = resize_and_crop_tensor(sample, self.ori_width, self.ori_height)

Complete Implementation Example

The following script demonstrates end-to-end usage of ControlNet with Sana, mirroring the CLI implementation in app/app_sana_controlnet_hed.py (lines 16‑70) without the multi-GPU launch logic:

import argparse
from pathlib import Path
from PIL import Image
import torch

# ----------------------------------------------------------------------

# 1️⃣  Argument parsing (mirrors the CLI in app_sana_controlnet_hed.py)

# ----------------------------------------------------------------------

parser = argparse.ArgumentParser()
parser.add_argument("--config", type=str, default="configs/sana_config/1024ms/Sana_1600M_img1024.yaml")
parser.add_argument("--model_path", type=str, default="output/Sana_D20/SANA.pth")
parser.add_argument("--ref_image", type=str, required=True, help="Path to the conditioning image")
parser.add_argument("--prompt", type=str, default="A photorealistic portrait")
parser.add_argument("--sketch_thickness", type=int, default=2)
parser.add_argument("--steps", type=int, default=30)
args = parser.parse_args()

# ----------------------------------------------------------------------

# 2️⃣  Build the pipeline

# ----------------------------------------------------------------------

from app.sana_controlnet_pipeline import SanaControlNetPipeline
pipe = SanaControlNetPipeline(args.config)
pipe.from_pretrained(args.model_path)          # load weights

pipe.eval()
pipe.to(torch.device("cuda" if torch.cuda.is_available() else "cpu"))

# ----------------------------------------------------------------------

# 3️⃣  Load reference image (any format Pillow understands)

# ----------------------------------------------------------------------

ref_img = Image.open(args.ref_image).convert("RGB")

# ----------------------------------------------------------------------

# 4️⃣  Run inference

# ----------------------------------------------------------------------

output_tensor = pipe.forward(
    prompt=args.prompt,
    ref_image=ref_img,
    num_inference_steps=args.steps,
    sketch_thickness=args.sketch_thickness,
)

# ----------------------------------------------------------------------

# 5️⃣  Convert tensor → PIL image & save

# ----------------------------------------------------------------------

output_img = (output_tensor.squeeze(0).cpu().numpy().transpose(1, 2, 0) * 127.5 + 127.5).astype("uint8")
Image.fromarray(output_img).save("generated.png")
print("✅ Image saved as generated.png")

Key Source Files and Methods

Understanding the ControlNet implementation requires familiarity with these specific source files:

Path	Purpose	Key Functions/Classes
app/sana_controlnet_pipeline.py	Core pipeline class and sampler orchestration	SanaControlNetPipeline, forward (lines 289‑303), Flow-Euler (lines 310‑322), DPM-Solver (lines 322‑342)
tools/controlnet/utils.py	Control map generation and preprocessing	get_scribble_map (lines 52‑82), transform_control_signal (lines 15‑34)
tools/controlnet/annotator/util.py	Low-level image utilities for HED detection	HWC3, resize_image, nms
app/app_sana_controlnet_hed.py	Reference CLI implementation	Command-line interface and inference orchestration

Summary

• SanaControlNetPipeline extends the base diffusion model with spatial conditioning capabilities through the control_signal parameter in model_kwargs.
• Control signals originate from HED edge detection (get_scribble_map) and undergo BICUBIC resizing, center-cropping, and normalization before VAE encoding.
• Dual sampler support allows selection between deterministic Flow-Euler (lines 310‑322) and stochastic DPM-Solver (lines 322‑342) based on the vis_sampler configuration.
• Latent encoding occurs via vae_encode, creating a compatible tensor that the diffusion model attends to alongside text embeddings during the forward pass (lines 289‑303).
• Guidance selection happens automatically via guidance_type_select (line 45), supporting both standard classifier-free guidance and Perturbed Attention Guidance (PAG).

Frequently Asked Questions

What is the difference between Flow-Euler and DPM-Solver in Sana ControlNet?

Flow-Euler implements a deterministic ODE solver (lines 310‑322 in app/sana_controlnet_pipeline.py) that follows a fixed trajectory through the diffusion noise schedule, producing consistent outputs for identical seeds. DPM-Solver (lines 322‑342) utilizes a multistep stochastic approach that can achieve comparable quality in fewer steps but introduces variability between runs. According to the NVlabs/Sana source code, Flow-Euler is preferred for reproducible sketch-to-image workflows, while DPM-Solver offers faster convergence for iterative refinement.

How does the HED detector process control images in Sana?

The Holistically-nested Edge Detection (HED) processing occurs in tools/controlnet/utils.py through the get_scribble_map function (lines 52‑82). This implementation applies non-maximum suppression to thin edge lines, optional Gaussian blur for anti-aliasing, and configurable stroke thickness adjustment. The detector first resizes the input to detect_resolution, computes gradient-based edges, then maps these to a binary raster suitable for diffusion conditioning.

Can I use custom control signals other than HED edge maps with SanaControlNetPipeline?

Yes. While the reference implementation in app/app_sana_controlnet_hed.py demonstrates HED-based scribble detection, the SanaControlNetPipeline.forward method accepts any tensor as control_signal provided it undergoes proper preprocessing through transform_control_signal (lines 15‑34) and VAE encoding. You can replace get_scribble_map with custom depth maps, semantic segmentation masks, or pose keypoints, ensuring the final tensor matches the expected [batch, channels, height, width] dimensions and [-1, 1] normalization range.

What hardware requirements are needed for running ControlNet with Sana?

The pipeline requires CUDA-capable hardware for the vae_encode and diffusion sampling operations, though the specific VRAM requirements depend on the model configuration (e.g., 1024ms vs 512ms). The pipe.to() method handles device placement, and the implementation supports automatic mixed precision via weight_dtype configuration. According to the source code in app/sana_controlnet_pipeline.py, the control signal processing occurs on the same device as the model weights to avoid latency from cross-device tensor transfers.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}