How to Work with Autoencoders and Variational Autoencoders (VAEs) for Image Generation

Autoencoders compress images into a latent representation and reconstruct them, while variational autoencoders (VAEs) force that latent space into a standard normal distribution so you can sample new images by decoding random vectors.

Both architectures are fundamental tools for unsupervised learning and generative modeling in computer vision. The Microsoft AI-For-Beginners repository provides complete, runnable implementations in PyTorch and TensorFlow that walk through building convolutional autoencoders and VAEs for image generation tasks.

What Is an Autoencoder vs. a VAE?

An autoencoder consists of two networks: an encoder that squeezes input data into a low-dimensional latent vector, and a decoder that reconstructs the original input from that vector. During training, the network learns to minimize reconstruction loss, typically using binary cross-entropy or MSE.

A variational autoencoder extends this by treating the latent representation as a probability distribution. Instead of outputting a single vector, the encoder predicts mean (μ) and log-variance (log σ²) parameters for each latent dimension. The reparameterization trick samples from this distribution while keeping gradients flowable:

z = μ + ε · σ, where ε ~ N(0, I)

The KL-divergence term in the loss forces these distributions toward a standard normal. After training, you can sample z ~ N(0, I) and feed it through the decoder to generate entirely new images.

Core VAE Architecture Components

The PyTorch implementation in lessons/4-ComputerVision/09-Autoencoders/AutoEncodersPyTorch.ipynb defines three essential modules:

Component	Purpose	Output Shape (MNIST 28×28)
VAEEncoder	Convolutional feature extraction → μ and log σ²	μ (latent_dim), logvar (latent_dim)
Reparameterization	Sample z with differentiable randomness	z (latent_dim)
VAEDecoder	Transpose convolutions reconstruct image	x̂ (1, 28, 28)

The TensorFlow notebook AutoencodersTF.ipynb follows identical logic using tf.keras.layers.

Building a Convolutional VAE in PyTorch

Below is the complete implementation extracted from the Microsoft repository's PyTorch notebook. This VAE uses 2D convolutions for encoding and transposed convolutions for decoding, trained on MNIST with a 2-dimensional latent space for visualization.

Encoder: Predicting Latent Distribution Parameters

import torch
import torch.nn as nn
import torch.nn.functional as F

latent_dim = 2
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

class VAEEncoder(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv = nn.Sequential(
            nn.Conv2d(1, 32, 3, stride=2, padding=1),   # 28 → 14

            nn.ReLU(),
            nn.Conv2d(32, 64, 3, stride=2, padding=1),  # 14 → 7

            nn.ReLU(),
        )
        self.fc_mu = nn.Linear(64 * 7 * 7, latent_dim)
        self.fc_logvar = nn.Linear(64 * 7 * 7, latent_dim)

    def forward(self, x):
        x = self.conv(x)
        x = x.view(x.size(0), -1)
        mu = self.fc_mu(x)
        logvar = self.fc_logvar(x)
        return mu, logvar

Decoder: Reconstructing Images from Latent Vectors

class VAEDecoder(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc = nn.Linear(latent_dim, 64 * 7 * 7)
        self.deconv = nn.Sequential(
            nn.ConvTranspose2d(64, 32, 3, stride=2, padding=1, output_padding=1),  # 7 → 14

            nn.ReLU(),
            nn.ConvTranspose2d(32, 1, 3, stride=2, padding=1, output_padding=1),   # 14 → 28

            nn.Sigmoid(),
        )

    def forward(self, z):
        x = self.fc(z)
        x = x.view(-1, 64, 7, 7)
        x = self.deconv(x)
        return x

Full VAE with Reparameterization

class VAE(nn.Module):
    def __init__(self):
        super().__init__()
        self.encoder = VAEEncoder()
        self.decoder = VAEDecoder()

    def reparameterize(self, mu, logvar):
        std = torch.exp(0.5 * logvar)
        eps = torch.randn_like(std)
        return mu + eps * std

    def forward(self, x):
        mu, logvar = self.encoder(x)
        z = self.reparameterize(mu, logvar)
        recon = self.decoder(z)
        return recon, mu, logvar

Training a VAE for Image Generation

The combined loss function balances reconstruction quality against latent space regularization:

def loss_fn(recon, x, mu, logvar):
    recon_loss = F.binary_cross_entropy(recon, x, reduction='sum')
    kl = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())
    return recon_loss + kl

Reconstruction loss ensures decoded images match inputs. KL-divergence penalizes deviations from N(0, I), creating a smooth, continuous latent space where interpolation yields meaningful image blends.

Training Loop

model = VAE().to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

for epoch in range(10):
    for batch, _ in train_loader:  # MNIST: [B, 1, 28, 28]

        batch = batch.to(device)
        recon, mu, logvar = model(batch)
        loss = loss_fn(recon, batch, mu, logvar)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    print(f'Epoch {epoch}: loss={loss.item():.2f}')

Generating New Images with a Trained VAE

Once training converges, sample from the prior distribution and decode:

with torch.no_grad():
    # Sample 64 latent vectors from standard normal

    z = torch.randn(64, latent_dim).to(device)
    samples = model.decoder(z).cpu()
    # samples shape: [64, 1, 28, 28] — new handwritten digits

Because the KL term constrained the latent space, these samples resemble the training distribution even though the model never saw them during training.

TensorFlow/Keras Implementation

The repository's AutoencodersTF.ipynb implements the identical architecture using tf.keras:

• Encoder: Dense layers or Conv2D → Flatten → separate Dense heads for μ and log σ
• Reparameterization: Lambda layer implementing z = mu + exp(log_var * 0.5) * epsilon
• Decoder: Dense → Reshape → Conv2DTranspose → Sigmoid
• Loss: binary_crossentropy + KL term added via add_loss

Both frameworks achieve equivalent results; choose based on your existing pipeline.

Key Source Files in AI-For-Beginners

File Path	Contents
lessons/4-ComputerVision/09-Autoencoders/AutoEncodersPyTorch.ipynb	Full PyTorch VAE with CNN encoder/decoder, training, and generation
lessons/4-ComputerVision/09-Autoencoders/AutoencodersTF.ipynb	TensorFlow/Keras equivalent using functional and sequential APIs
lessons/4-ComputerVision/09-Autoencoders/README.md	Mathematical foundations: ELBO, KL divergence, and reparameterization trick

The PyTorch notebook's implementation (cells 965–1036) specifically defines VAEEncoder, VAEDecoder, and VAEAutoEncoder classes used in the code examples above.

Practical Tips for Training VAEs

• Start with latent_dim=2 for visualization; increase to 20–50 for complex datasets
• Use batch normalization in deeper encoders for stable training
• Monitor KL divergence separately — if it collapses to zero, the model ignores the latent variable (posterior collapse)
• Anneal the KL weight progressively if reconstruction quality suffers early in training
• Try β-VAE (multiply KL by β > 1) for more disentangled latent representations

Summary

• Autoencoders learn compressed representations via encoder-decoder reconstruction
• VAEs add probabilistic constraints: encoder outputs μ and log σ², trained with reconstruction loss plus KL-divergence
• The reparameterization trick enables gradient flow through random sampling: z = μ + ε·σ
• After training, sample z ~ N(0, I) and decode to generate new images
• Microsoft AI-For-Beginners provides complete PyTorch and TensorFlow notebooks for hands-on learning

Frequently Asked Questions

What is the difference between an autoencoder and a variational autoencoder?

An autoencoder learns a deterministic compression of input data, mapping each image to a single point in latent space. A variational autoencoder learns a distribution over latent codes, producing similar images from nearby points and enabling generation by sampling from the prior distribution.

Why does VAE training require the reparameterization trick?

Backpropagation cannot flow through random sampling operations. The reparameterization trick moves the randomness to an auxiliary noise variable ε, allowing gradients to propagate through the deterministic operations μ + ε·σ that compute the latent sample z.

What causes VAE posterior collapse and how can I prevent it?

Posterior collapse occurs when the KL term drives encoder outputs to match the prior regardless of input, causing the decoder to ignore the latent code. Solutions include KL annealing (starting with zero weight and increasing), free bits (capping minimum KL), or using more powerful decoders. The AI-For-Beginners notebooks use β-weighting to balance these terms.

Can VAEs generate high-resolution images comparable to GANs?

Standard VAEs produce blurrier outputs than GANs because the likelihood objective averages over modes. For sharper images, combine VAEs with adversarial training (VAEGAN), use hierarchical architectures (NVAE), or adopt diffusion-based decoders. The repository's basic implementation is best suited for understanding fundamentals and generating MNIST-level resolutions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}