# How to Use Diffusion Models for Image Generation with AILIA Models

> Generate high-quality images from text with AILIA diffusion models. Explore stable diffusion and SDXL-Turbo pipelines for automatic text encoding, denoising, and decoding.

- Repository: [axinc-ai/ailia-models](https://github.com/axinc-ai/ailia-models)
- Tags: tutorial
- Published: 2026-02-26

---

**You can generate high-quality images from text prompts using the ready-to-run Stable Diffusion and SDXL-Turbo pipelines in the axinc-ai/ailia-models repository, which handle text encoding, iterative denoising, and VAE decoding automatically.**

The axinc-ai/ailia-models repository provides complete, production-ready implementations of state-of-the-art diffusion models for image generation. These pipelines encapsulate the full generation workflow—from CLIP text encoding through UNet denoising to VAE decoding—allowing you to create images via simple command-line interfaces or direct Python integration.

## Understanding the Diffusion Pipeline Architecture

The diffusion pipelines in this repository follow the standard latent diffusion architecture established by Stable Diffusion and SDXL-Turbo. Each component is implemented as a distinct ONNX model loaded through the AILIA SDK or ONNX Runtime.

### Core Components

The pipelines consist of five interconnected components:

- **Text Encoder (CLIP)**: Converts natural language prompts into latent text embeddings. Implemented in [`stable-diffusion-txt2img.py`](https://github.com/axinc-ai/ailia-models/blob/main/stable-diffusion-txt2img.py) via `FrozenCLIPEmbedder` and in [`sdxl-turbo.py`](https://github.com/axinc-ai/ailia-models/blob/main/sdxl-turbo.py) through dual text encoders (lines 13-31).

- **UNet (Denoising Network)**: Predicts noise residuals for latent images at specific timesteps, conditioned on text embeddings. Loaded via `ailia.Net` or `onnxruntime.InferenceSession` using paths like `WEIGHT_UNET_PATH` in [`sdxl-turbo.py`](https://github.com/axinc-ai/ailia-models/blob/main/sdxl-turbo.py).

- **VAE Encoder/Decoder**: Transforms between pixel space and the lower-dimensional latent space where diffusion occurs. The decoder (`vae_decoder`) converts final latents to RGB images in both pipelines.

- **Scheduler**: Manages timestep scheduling and reverse diffusion steps. Uses `EulerAncestralDiscreteScheduler` in SDXL-Turbo and custom DDIM helpers (`make_ddim_timesteps`, `make_ddim_sampling_parameters`) in [`stable-diffusion-txt2img.py`](https://github.com/axinc-ai/ailia-models/blob/main/stable-diffusion-txt2img.py).

- **Model Wrappers**: High-level classes (`StableDiffusionXL`, `StableDiffusionXLImg2Img` in `sdxl-turbo/df/pipelines/`) that orchestrate component interactions.

### The Generation Process

The diffusion process follows these discrete steps:

1. **Encode the prompt** using CLIP to produce text embedding `c`.
2. **Sample random latent** `z` from Gaussian noise with shape `[C, H/f, W/f]`.
3. **Iteratively denoise** `z` for *N* timesteps using the UNet, scheduler, and classifier-free guidance scale.
4. **Decode the final latent** through the VAE decoder to produce the RGB output image.

## Running Stable Diffusion for Text-to-Image Generation

The [`stable-diffusion-txt2img.py`](https://github.com/axinc-ai/ailia-models/blob/main/stable-diffusion-txt2img.py) script provides a complete CLI for generating images from text prompts using Stable Diffusion v1.4 or custom checkpoints.

### Installation Requirements

Install the required dependency for text tokenization:

```bash
pip3 install transformers

```

### Basic CLI Usage

Generate an image with a single command:

```bash
python3 diffusion/stable-diffusion-txt2img/stable-diffusion-txt2img.py \
    --input "a photograph of an astronaut riding a horse" \
    --savepath results/astro.png

```

The script automatically downloads required ONNX models from the remote bucket on first run.

### Common Configuration Options

| Flag | Description |
|------|-------------|
| `--n_iter N` | Repeat generation N times (creates result grid) |
| `--n_samples S` | Generate S samples per iteration |
| `--steps K` | Number of DDIM sampling steps (default: 50) |
| `--scale V` | Classifier-free guidance scale (default: 7.5) |
| `--sampler {PLMS,DDIM,"DPM++ 2M Kerras"}` | Sampling algorithm selection |
| `--onnx` | Execute via ONNX Runtime instead of AILIA SDK |
| `--legacy` | Use three-file legacy model layout (diffusion_emb, diffusion_mid, diffusion_out) |
| `--sd {default,basil_mix}` | Switch to Basil-Mix checkpoint |
| `--vae {default,vae-ft-mse}` | Use fine-tuned VAE for higher quality |

## Using SDXL-Turbo for Faster Inference

The [`sdxl-turbo.py`](https://github.com/axinc-ai/ailia-models/blob/main/sdxl-turbo.py) script implements the SDXL-Turbo architecture, which generates high-quality images in as few as 1-4 steps compared to the 50 steps required by standard Stable Diffusion.

### Text-to-Image Generation

Run SDXL-Turbo with minimal configuration:

```bash
python3 diffusion/sdxl-turbo/sdxl-turbo.py \
    --input "little cute gremlin wearing a jacket, cinematic, vivid colors" \
    --savepath results/gremlin.png

```

### Image-to-Image Generation

Supply an initial image to guide the generation process:

```bash
python3 diffusion/sdxl-turbo/sdxl-turbo.py \
    --input "cat wizard, Gandalf, fantasy, Pixar style" \
    --init_image samples/cat.png \
    --savepath results/cat_wizard.png

```

### SDXL-Turbo Specific Options

| Flag | Description |
|------|-------------|
| `--init_image PATH` | Source image for img2img pipeline |
| `--seed N` | Fixed random seed for reproducible outputs |
| `--disable_ailia_tokenizer` | Use Hugging Face tokenizer instead of AILIA's bundled version |

The SDXL-Turbo implementation uses dual text encoders and the `EulerAncestralDiscreteScheduler` defined in [`diffusion/sdxl-turbo/df/schedulers/scheduling_euler_ancestral_discrete.py`](https://github.com/axinc-ai/ailia-models/blob/main/diffusion/sdxl-turbo/df/schedulers/scheduling_euler_ancestral_discrete.py).

## Integrating Diffusion Models Programmatically

Beyond CLI usage, you can import the pipeline components directly into your Python applications.

### Importing from Stable Diffusion

Access helper functions and the main entry point from [`stable-diffusion-txt2img.py`](https://github.com/axinc-ai/ailia-models/blob/main/stable-diffusion-txt2img.py):

```python
from diffusion.stable_diffusion_txt2img.stable_diffusion_txt2img import (
    make_ddim_timesteps,
    make_ddim_sampling_parameters,
    recognize_from_text,
    main as sd_main,
)

# Run generation without CLI parsing

sd_main()

```

### Using the StableDiffusionXL Class

For SDXL-Turbo, instantiate the `StableDiffusionXL` class directly from the `df` sub-package:

```python
from diffusion.sdxl_turbo.df import StableDiffusionXL

pipe = StableDiffusionXL(
    vae_decoder=vae_decoder,
    text_encoder=text_encoder,
    text_encoder_2=text_encoder_2,
    tokenizer=tokenizer,
    tokenizer_2=tokenizer_2,
    unet=unet,
    scheduler=scheduler,
    use_onnx=False,
)

image = pipe.forward(
    prompt="a futuristic city at sunset",
    num_inference_steps=2
)

```

The class definition resides in [`diffusion/sdxl-turbo/df/pipelines/stable_diffusion_xl.py`](https://github.com/axinc-ai/ailia-models/blob/main/diffusion/sdxl-turbo/df/pipelines/stable_diffusion_xl.py), with the img2img variant available in [`stable_diffusion_xl_img2img.py`](https://github.com/axinc-ai/ailia-models/blob/main/stable_diffusion_xl_img2img.py).

## Summary

- The **axinc-ai/ailia-models** repository provides complete diffusion pipelines for both standard Stable Diffusion and high-speed SDXL-Turbo generation.
- Each pipeline consists of five core components: **CLIP text encoder**, **UNet denoiser**, **VAE encoder/decoder**, **scheduler**, and **model wrappers**.
- Run text-to-image generation via [`stable-diffusion-txt2img.py`](https://github.com/axinc-ai/ailia-models/blob/main/stable-diffusion-txt2img.py) or [`sdxl-turbo.py`](https://github.com/axinc-ai/ailia-models/blob/main/sdxl-turbo.py) with automatic model downloading and support for both AILIA SDK and ONNX Runtime backends.
- Implement image-to-image generation using the `--init_image` flag in SDXL-Turbo or programmatically via the `StableDiffusionXLImg2Img` class.
- Integrate diffusion capabilities into custom applications by importing helper functions from the scripts or instantiating the `StableDiffusionXL` class directly.

## Frequently Asked Questions

### What is the difference between Stable Diffusion and SDXL-Turbo in the ailia-models repository?

**Stable Diffusion** (v1.4/Basil-Mix) requires approximately 50 inference steps to generate high-quality images and uses a single text encoder. **SDXL-Turbo** generates comparable quality in 1-4 steps using a distilled UNet and dual text encoders (CLIP-L and OpenCLIP-G), making it significantly faster for real-time applications. Both are implemented in the repository with identical CLI interfaces but reside in different directories (`stable-diffusion-txt2img` vs `sdxl-turbo`).

### Do I need the AILIA SDK to run these diffusion models?

No. While the scripts default to the AILIA SDK for optimized inference, you can run both pipelines entirely with **ONNX Runtime** by adding the `--onnx` flag to any command. This allows execution on CPU or CUDA without installing the proprietary AILIA SDK, though performance characteristics may differ depending on your hardware and ONNX Runtime version.

### How do I perform image-to-image generation with these models?

For **SDXL-Turbo**, use the `--init_image` flag followed by the path to your source image: `python3 diffusion/sdxl-turbo/sdxl-turbo.py --input "prompt" --init_image source.jpg`. Programmatically, import `StableDiffusionXLImg2Img` from `diffusion.sdxl_turbo.df` and pass the `image` parameter to the `forward()` method. The standard Stable Diffusion pipeline in the repository focuses on text-to-image, though the underlying architecture supports img2img via the VAE encoder component.

### What hardware requirements are needed for reasonable inference speeds?

Both pipelines run on CPU via ONNX Runtime, but a **CUDA-capable GPU with at least 8GB VRAM** is recommended for production use. SDXL-Turbo specifically benefits from GPU acceleration due to its larger UNet and dual text encoders. The AILIA SDK provides optimized backends for various edge devices including ARM processors and NVIDIA Jetson platforms, making these models deployable on embedded systems with reduced precision (FP16/INT8) where supported.