How to Implement Variational Autoencoders for Image Generation Using Microsoft AI-For-Beginners

You can implement a Variational Autoencoder for image generation by building an encoder that predicts latent distribution parameters, applying the reparameterization trick to sample latent codes, and training a decoder to minimize both reconstruction error and KL divergence.

The Microsoft AI-For-Beginners repository provides production-ready implementations of Variational Autoencoders in both PyTorch and TensorFlow. This guide walks through the complete implementation details found in lessons/4-ComputerVision/09-Autoencoders/, demonstrating how to construct the encoder-decoder architecture, compute the evidence lower bound (ELBO), and generate novel images from learned latent distributions.

What Is a Variational Autoencoder?

A Variational Autoencoder is a generative model that learns a probabilistic mapping between data space and a lower-dimensional latent space. Unlike standard autoencoders that learn deterministic encodings, VAEs enforce a Gaussian prior on the latent distribution through the reparameterization trick, enabling smooth interpolation and novel sample generation.

The architecture consists of three components: an encoder that outputs mean (μ) and log-variance (log σ²) parameters, a sampling layer that draws latent vectors z ~ N(μ, σ²), and a decoder that reconstructs images from z.

Key Files in the Repository

The VAE implementations are located in lessons/4-ComputerVision/09-Autoencoders/:

• AutoEncodersPyTorch.ipynb – Complete PyTorch implementation with custom forward methods and manual loss computation.
• AutoencodersTF.ipynb – TensorFlow/Keras implementation using the Functional API and add_loss for KL divergence.
• images/vae.png – Architecture diagram illustrating the probabilistic encoder-decoder flow.

PyTorch Implementation

The PyTorch notebook defines the VAE as a single module combining encoder and decoder networks with explicit loss calculations.

Encoder Architecture

The encoder compresses 28×28 MNIST images into latent distribution parameters. In AutoEncodersPyTorch.ipynb, the encoder uses fully-connected layers to output μ and log σ²:

class Encoder(nn.Module):
    def __init__(self, latent_dim):
        super().__init__()
        self.fc1 = nn.Linear(784, 512)
        self.fc_mu = nn.Linear(512, latent_dim)
        self.fc_logvar = nn.Linear(512, latent_dim)
    
    def forward(self, x):
        x = x.view(-1, 784)
        h = F.relu(self.fc1(x))
        mu = self.fc_mu(h)
        logvar = self.fc_logvar(h)
        return mu, logvar

The Reparameterization Trick

To enable backpropagation through random sampling, the implementation uses the reparameterization trick. This samples epsilon from a standard normal distribution and scales it by the learned standard deviation:

def reparameterize(self, mu, logvar):
    std = torch.exp(0.5 * logvar)
    eps = torch.randn_like(std)
    return mu + eps * std

This operation moves sampling to an independent noise variable while maintaining differentiable pathways for gradient descent.

Decoder Architecture

The decoder reverses the process, mapping latent vectors back to image space:

class Decoder(nn.Module):
    def __init__(self, latent_dim):
        super().__init__()
        self.fc1 = nn.Linear(latent_dim, 512)
        self.fc2 = nn.Linear(512, 784)
    
    def forward(self, z):
        h = F.relu(self.fc1(z))
        return torch.sigmoid(self.fc2(h))

VAE Loss Function

The total loss combines binary cross-entropy reconstruction loss with the KL divergence between the learned distribution and the prior:

def loss_function(recon_x, x, mu, logvar):
    BCE = F.binary_cross_entropy(recon_x, x.view(-1, 784), reduction='sum')
    KLD = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())
    return BCE + KLD

TensorFlow/Keras Implementation

The TensorFlow implementation in AutoencodersTF.ipynb uses the Functional API to construct the VAE as a Model subclass with built-in loss tracking.

Model Definition with add_loss

Instead of returning distribution parameters as separate outputs, the encoder appends the KL divergence directly to the model using add_loss:

class VAE(tf.keras.Model):
    def __init__(self, latent_dim):
        super(VAE, self).__init__()
        self.latent_dim = latent_dim
        self.encoder = tf.keras.Sequential([
            tf.keras.layers.Flatten(),
            tf.keras.layers.Dense(512, activation='relu'),
            tf.keras.layers.Dense(latent_dim * 2)  # mu and log_var

        ])
        self.decoder = tf.keras.Sequential([
            tf.keras.layers.Dense(512, activation='relu'),
            tf.keras.layers.Dense(784, activation='sigmoid'),
            tf.keras.layers.Reshape((28, 28))
        ])
    
    def reparameterize(self, mean, logvar):
        eps = tf.random.normal(shape=mean.shape)
        return eps * tf.exp(logvar * 0.5) + mean
    
    def call(self, x):
        mean_logvar = self.encoder(x)
        mean, logvar = tf.split(mean_logvar, num_or_size_splits=2, axis=1)
        z = self.reparameterize(mean, logvar)
        recon_x = self.decoder(z)
        
        # Add KL divergence as internal loss

        kl_loss = -0.5 * tf.reduce_mean(1 + logvar - tf.square(mean) - tf.exp(logvar))
        self.add_loss(kl_loss)
        return recon_x

This approach automatically includes the KL term during model.fit() without requiring explicit loss function definitions in the training loop.

Training and Image Generation

Both implementations follow the same training protocol using the MNIST dataset.

Data Preparation

Load and normalize MNIST to [0, 1] range:

# PyTorch

transform = transforms.ToTensor()
train_loader = DataLoader(
    datasets.MNIST('./data', train=True, download=True, transform=transform),
    batch_size=128, shuffle=True
)

# TensorFlow

(x_train, _), _ = tf.keras.datasets.mnist.load_data()
x_train = x_train.astype('float32') / 255.0

Training Loop

For PyTorch, manually optimize the ELBO:

optimizer = optim.Adam(model.parameters(), lr=1e-3)

def train(epoch):
    model.train()
    for batch_idx, (data, _) in enumerate(train_loader):
        optimizer.zero_grad()
        mu, logvar = model.encode(data.view(-1, 784))
        z = model.reparameterize(mu, logvar)
        recon_batch = model.decode(z)
        loss = loss_function(recon_batch, data, mu, logvar)
        loss.backward()
        optimizer.step()

In TensorFlow, compilation handles the optimization:

vae = VAE(latent_dim=20)
vae.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=1e-3), loss='binary_crossentropy')
vae.fit(x_train, x_train, epochs=50, batch_size=128)

Sampling New Images

To generate images, sample from the prior distribution N(0, I) and decode:

# PyTorch

with torch.no_grad():
    z = torch.randn(64, latent_dim)
    sample = model.decode(z).view(64, 1, 28, 28)

# TensorFlow

z = tf.random.normal(shape=(64, latent_dim))
samples = vae.decoder(z)

Summary

• VAEs learn probabilistic latent representations by encoding inputs into Gaussian parameters (μ, σ²) rather than fixed vectors.
• The reparameterization trick enables gradient flow through random sampling by separating the stochastic component (ε) from the learned parameters.
• Microsoft AI-For-Beginners provides dual implementations: PyTorch (AutoEncodersPyTorch.ipynb) using explicit loss computation and TensorFlow (AutoencodersTF.ipynb) using add_loss for KL divergence.
• Training minimizes the ELBO, combining reconstruction accuracy (BCE/MSE) with KL divergence regularization.
• Generation occurs by sampling latent vectors from the standard normal prior and passing them through the trained decoder.

Frequently Asked Questions

What is the difference between a standard autoencoder and a VAE?

A standard autoencoder learns deterministic latent codes, which can lead to discontinuities in the latent space and poor generation quality. A VAE learns a probability distribution over latents, enforcing continuity through the KL divergence regularization term. This allows VAEs to generate novel images by sampling from the learned distribution rather than just memorizing training examples.

Why do we use log variance instead of standard deviation directly?

The implementation uses log variance (log σ²) for numerical stability during optimization. Log-space ensures the variance remains positive without requiring additional constraints, and it prevents the exponential term in the KL divergence from exploding during early training stages. The conversion to standard deviation occurs inside the reparameterization function using exp(0.5 * logvar).

How does the β-VAE variant modify the standard implementation?

The β-VAE introduces a weighting factor β on the KL divergence term: Loss = Reconstruction + β × KL. When β > 1, the model learns more disentangled latent representations at the cost of reconstruction quality. To implement this in the Microsoft notebooks, multiply the KLD or kl_loss calculation by a scalar beta value (typically 1-4 for disentanglement).

Can these VAEs generate high-resolution images beyond 28×28 MNIST?

The current implementations use fully-connected layers optimized for 28×28 grayscale images. To generate higher resolution images (e.g., 64×64 or 128×128), you must replace the dense layers with convolutional and transposed convolutional layers in the encoder and decoder respectively. The repository's task notebook (09-Autoencoders.ipynb) explicitly prompts this modification as an exercise for adapting the architecture to CIFAR-10 or custom datasets.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}