# Using ControlNet with Sana for Controllable Image Generation: A Complete Implementation Guide

> Unlock controllable image generation with ControlNet and Sana. This guide shows how to use SanaControlNetPipeline to inject VAE-encoded control signals for precise diffusion sampling. Integrate spatial conditioning easily.

- Repository: [NVIDIA Research Projects/Sana](https://github.com/NVlabs/Sana)
- Tags: how-to-guide
- Published: 2026-05-19

---

**Sana's ControlNet integration enables spatial conditioning via the `SanaControlNetPipeline` class, which injects VAE-encoded control signals (such as HED edge maps) into the diffusion sampling loop alongside text embeddings to guide image generation.**

The NVlabs/Sana repository extends its efficient text-to-image diffusion backbone with a ControlNet-compatible pipeline that accepts spatial priors like scribbles or edge maps. This implementation, found in [`app/sana_controlnet_pipeline.py`](https://github.com/NVlabs/Sana/blob/main/app/sana_controlnet_pipeline.py), orchestrates the conversion of reference images into latent control signals that condition the generation process without requiring model retraining.

## Architecture Overview

Sana's ControlNet architecture consists of three integrated components that transform a reference image into a conditioning signal:

| Component | Role | Key Source Location |
|-----------|------|---------------------|
| **SanaControlNetPipeline** | Wraps the diffusion model, VAE, and text encoder to orchestrate the sampling loop with control signal injection | [`app/sana_controlnet_pipeline.py`](https://github.com/NVlabs/Sana/blob/main/app/sana_controlnet_pipeline.py) (lines 98‑156) |
| **Control Signal Generation** | Creates raster maps (e.g., HED-based scribbles) from reference images using edge detection and post-processing | [`tools/controlnet/utils.py`](https://github.com/NVlabs/Sana/blob/main/tools/controlnet/utils.py) (lines 37‑83) |
| **Signal Transformation** | Resizes and normalizes maps to match latent spatial resolution expected by the diffusion model | [`tools/controlnet/utils.py`](https://github.com/NVlabs/Sana/blob/main/tools/controlnet/utils.py) (lines 15‑34) |

During inference, the control map undergoes VAE encoding via `vae_encode` and passes as `control_signal` inside `model_kwargs` (see the `forward` method at lines 289‑303). The diffusion sampler then conditions on both the text embedding (`caption_embs`) and the control latent, enabling sketch-to-image or edge-to-image synthesis.

## Preparing the Control Signal

### Edge Detection with HED

The pipeline generates spatial conditioning using the **HED** (Holistically-nested Edge Detection) detector. The `get_scribble_map` function in [`tools/controlnet/utils.py`](https://github.com/NVlabs/Sana/blob/main/tools/controlnet/utils.py) (lines 52‑82) processes reference images through non-maximum suppression, Gaussian blur, and line thickness adjustment:

```python
control_signal = get_scribble_map(
    input_image=ref_image,
    det="Scribble_HED",                 # HED edge detector

    detect_resolution=int(hw.min()),      # matches target latent size

    thickness=sketch_thickness,
)

```

This creates a binary edge map that preserves the structural composition of the reference image while allowing adjustable stroke thickness for different artistic effects.

### Transformation and Normalization

Before injection into the diffusion model, the `transform_control_signal` function (lines 15‑34 in [`tools/controlnet/utils.py`](https://github.com/NVlabs/Sana/blob/main/tools/controlnet/utils.py)) converts the map to a 3-channel RGB tensor, applies **BICUBIC** resizing, center-cropping, and normalization to the `[-1, 1]` range:

```python
control_signal = transform_control_signal(control_signal, hw) \
                 .to(self.device).to(self.weight_dtype)

```

This ensures the control signal dimensions match the latent spatial resolution (HW) expected by the VAE encoder.

## Pipeline Initialization

The `SanaControlNetPipeline` class loads configuration via `pyrallis` and constructs the VAE, tokenizer, and core Sana model through the `build_model` function:

```python
from app.sana_controlnet_pipeline import SanaControlNetPipeline

pipe = SanaControlNetPipeline(args.config)          # loads YAML configuration

pipe.from_pretrained(pipe.args.model_path)           # loads checkpoint weights

```

The pipeline supports both **Flow-Euler** (deterministic ODE solver) and **DPM-Solver** (multistep stochastic) samplers, selected via the `vis_sampler` configuration parameter.

## The Inference Workflow

### Step 1: Control Map Generation and Encoding

The reference image first converts to a scribble map, then encodes to latent space using the VAE:

```python
control_signal_latent = vae_encode(
    self.config.vae.vae_type, self.vae,
    control_signal, self.config.vae.sample_posterior, self.device
)

```

This latent representation allows the diffusion model to attend to spatial features at the same resolution as the noise latents.

### Step 2: Conditioned Sampling

The pipeline injects the control latent via `model_kwargs["control_signal"]` during the sampling loop (lines 289‑303). Depending on the `vis_sampler` setting, the pipeline executes either:

- **Flow-Euler** (`FlowEuler`) — A deterministic ODE solver implemented at lines 310‑322
- **DPM-Solver** (`DPMS`) — A multistep stochastic solver implemented at lines 322‑342

Both samplers receive the conditioning tuple (`condition=caption_embs`, `uncondition=null_y`) plus the control signal. The guidance type (`classifier-free` or `classifier-free_PAG`) is selected automatically via `guidance_type_select` (line 45).

### Step 3: Decoding and Post-Processing

After sampling completes, the latent tensor decodes to pixel space and resizes to match original reference dimensions:

```python
sample = vae_decode(self.config.vae.vae_type, self.vae, sample)
sample = resize_and_crop_tensor(sample, self.ori_width, self.ori_height)

```

## Complete Implementation Example

The following script demonstrates end-to-end usage of ControlNet with Sana, mirroring the CLI implementation in [`app/app_sana_controlnet_hed.py`](https://github.com/NVlabs/Sana/blob/main/app/app_sana_controlnet_hed.py) (lines 16‑70) without the multi-GPU launch logic:

```python
import argparse
from pathlib import Path
from PIL import Image
import torch

# ----------------------------------------------------------------------

# 1️⃣  Argument parsing (mirrors the CLI in app_sana_controlnet_hed.py)

# ----------------------------------------------------------------------

parser = argparse.ArgumentParser()
parser.add_argument("--config", type=str, default="configs/sana_config/1024ms/Sana_1600M_img1024.yaml")
parser.add_argument("--model_path", type=str, default="output/Sana_D20/SANA.pth")
parser.add_argument("--ref_image", type=str, required=True, help="Path to the conditioning image")
parser.add_argument("--prompt", type=str, default="A photorealistic portrait")
parser.add_argument("--sketch_thickness", type=int, default=2)
parser.add_argument("--steps", type=int, default=30)
args = parser.parse_args()

# ----------------------------------------------------------------------

# 2️⃣  Build the pipeline

# ----------------------------------------------------------------------

from app.sana_controlnet_pipeline import SanaControlNetPipeline
pipe = SanaControlNetPipeline(args.config)
pipe.from_pretrained(args.model_path)          # load weights

pipe.eval()
pipe.to(torch.device("cuda" if torch.cuda.is_available() else "cpu"))

# ----------------------------------------------------------------------

# 3️⃣  Load reference image (any format Pillow understands)

# ----------------------------------------------------------------------

ref_img = Image.open(args.ref_image).convert("RGB")

# ----------------------------------------------------------------------

# 4️⃣  Run inference

# ----------------------------------------------------------------------

output_tensor = pipe.forward(
    prompt=args.prompt,
    ref_image=ref_img,
    num_inference_steps=args.steps,
    sketch_thickness=args.sketch_thickness,
)

# ----------------------------------------------------------------------

# 5️⃣  Convert tensor → PIL image & save

# ----------------------------------------------------------------------

output_img = (output_tensor.squeeze(0).cpu().numpy().transpose(1, 2, 0) * 127.5 + 127.5).astype("uint8")
Image.fromarray(output_img).save("generated.png")
print("✅ Image saved as generated.png")

```

## Key Source Files and Methods

Understanding the ControlNet implementation requires familiarity with these specific source files:

| Path | Purpose | Key Functions/Classes |
|------|---------|----------------------|
| [`app/sana_controlnet_pipeline.py`](https://github.com/NVlabs/Sana/blob/main/app/sana_controlnet_pipeline.py) | Core pipeline class and sampler orchestration | `SanaControlNetPipeline`, `forward` (lines 289‑303), Flow-Euler (lines 310‑322), DPM-Solver (lines 322‑342) |
| [`tools/controlnet/utils.py`](https://github.com/NVlabs/Sana/blob/main/tools/controlnet/utils.py) | Control map generation and preprocessing | `get_scribble_map` (lines 52‑82), `transform_control_signal` (lines 15‑34) |
| [`tools/controlnet/annotator/util.py`](https://github.com/NVlabs/Sana/blob/main/tools/controlnet/annotator/util.py) | Low-level image utilities for HED detection | `HWC3`, `resize_image`, `nms` |
| [`app/app_sana_controlnet_hed.py`](https://github.com/NVlabs/Sana/blob/main/app/app_sana_controlnet_hed.py) | Reference CLI implementation | Command-line interface and inference orchestration |

## Summary

- **SanaControlNetPipeline** extends the base diffusion model with spatial conditioning capabilities through the `control_signal` parameter in `model_kwargs`.
- **Control signals** originate from HED edge detection (`get_scribble_map`) and undergo BICUBIC resizing, center-cropping, and normalization before VAE encoding.
- **Dual sampler support** allows selection between deterministic Flow-Euler (lines 310‑322) and stochastic DPM-Solver (lines 322‑342) based on the `vis_sampler` configuration.
- **Latent encoding** occurs via `vae_encode`, creating a compatible tensor that the diffusion model attends to alongside text embeddings during the `forward` pass (lines 289‑303).
- **Guidance selection** happens automatically via `guidance_type_select` (line 45), supporting both standard classifier-free guidance and Perturbed Attention Guidance (PAG).

## Frequently Asked Questions

### What is the difference between Flow-Euler and DPM-Solver in Sana ControlNet?

**Flow-Euler** implements a deterministic ODE solver (lines 310‑322 in [`app/sana_controlnet_pipeline.py`](https://github.com/NVlabs/Sana/blob/main/app/sana_controlnet_pipeline.py)) that follows a fixed trajectory through the diffusion noise schedule, producing consistent outputs for identical seeds. **DPM-Solver** (lines 322‑342) utilizes a multistep stochastic approach that can achieve comparable quality in fewer steps but introduces variability between runs. According to the NVlabs/Sana source code, Flow-Euler is preferred for reproducible sketch-to-image workflows, while DPM-Solver offers faster convergence for iterative refinement.

### How does the HED detector process control images in Sana?

The Holistically-nested Edge Detection (HED) processing occurs in [`tools/controlnet/utils.py`](https://github.com/NVlabs/Sana/blob/main/tools/controlnet/utils.py) through the `get_scribble_map` function (lines 52‑82). This implementation applies non-maximum suppression to thin edge lines, optional Gaussian blur for anti-aliasing, and configurable stroke thickness adjustment. The detector first resizes the input to `detect_resolution`, computes gradient-based edges, then maps these to a binary raster suitable for diffusion conditioning.

### Can I use custom control signals other than HED edge maps with SanaControlNetPipeline?

Yes. While the reference implementation in [`app/app_sana_controlnet_hed.py`](https://github.com/NVlabs/Sana/blob/main/app/app_sana_controlnet_hed.py) demonstrates HED-based scribble detection, the `SanaControlNetPipeline.forward` method accepts any tensor as `control_signal` provided it undergoes proper preprocessing through `transform_control_signal` (lines 15‑34) and VAE encoding. You can replace `get_scribble_map` with custom depth maps, semantic segmentation masks, or pose keypoints, ensuring the final tensor matches the expected `[batch, channels, height, width]` dimensions and `[-1, 1]` normalization range.

### What hardware requirements are needed for running ControlNet with Sana?

The pipeline requires CUDA-capable hardware for the `vae_encode` and diffusion sampling operations, though the specific VRAM requirements depend on the model configuration (e.g., 1024ms vs 512ms). The `pipe.to()` method handles device placement, and the implementation supports automatic mixed precision via `weight_dtype` configuration. According to the source code in [`app/sana_controlnet_pipeline.py`](https://github.com/NVlabs/Sana/blob/main/app/sana_controlnet_pipeline.py), the control signal processing occurs on the same device as the model weights to avoid latency from cross-device tensor transfers.