How to Use Autoencoders and VAEs for Dimensionality Reduction: A Practical Guide

Autoencoders and Variational Autoencoders (VAEs) compress high-dimensional data into low-dimensional latent vectors by training neural networks to reconstruct their inputs through a bottleneck layer, producing embeddings suitable for visualization, clustering, and downstream machine learning tasks.

The microsoft/AI-For-Beginners repository provides complete implementations demonstrating how to leverage these architectures for effective dimensionality reduction. By forcing data through a compact latent space, these models learn to preserve salient features while discarding noise, outperforming traditional linear methods like PCA on complex datasets.

Understanding the Autoencoder Architecture

An autoencoder is a neural network trained to copy its input to its output. The critical constraint is the bottleneck—a latent layer with fewer neurons than the input dimension—which forces the network to learn a compressed representation.

The Encoder-Bottleneck-Decoder Pipeline

According to the source code in lessons/4-ComputerVision/09-Autoencoders/README.md, the architecture consists of three components:

1. Encoder – A stack of convolutional or fully-connected layers that compresses the input into a latent vector z of size d < input_dim.
2. Bottleneck – The latent vector containing the compressed representation. Its dimensionality directly determines the degree of reduction.
3. Decoder – Mirrors the encoder using transposed convolutions or dense layers to reconstruct the original input from z.

The network minimizes a reconstruction loss (typically Mean Squared Error) measuring pixel-wise or feature-wise differences between input and output. In lessons/4-ComputerVision/09-Autoencoders/AutoencodersTF.ipynb, this architecture is implemented using TensorFlow/Keras convolutional layers for MNIST digit compression.

Why Autoencoders Surpass PCA for Dimensionality Reduction

Autoencoders reduce dimensionality more effectively than Principal Component Analysis because they capture non-linear relationships and spatial hierarchies:

• Non-linear encoding – Activation functions (ReLU, Sigmoid) enable the encoder to map data onto complex manifolds that linear projections cannot represent.
• Spatial awareness – Convolutional layers preserve locality and hierarchical features, whereas PCA treats each pixel or feature independently.

How Variational Autoencoders (VAEs) Enhance Latent Representations

A Variational Autoencoder extends the classic architecture by treating the latent space as a probabilistic distribution rather than a deterministic point. This approach, detailed in lessons/4-ComputerVision/09-Autoencoders/AutoEncodersPyTorch.ipynb, generates smoother, more interpretable embeddings.

Probabilistic Latent Spaces and the Reparameterization Trick

Unlike standard autoencoders that output a fixed vector, the VAE encoder predicts two parameters for each latent dimension:

• Mean (μ) – The center of the Gaussian distribution.
• Log-variance (log σ²) – The uncertainty or spread.

To enable backpropagation through the random sampling process, VAEs employ the reparameterization trick:

z = μ + σ * ε

Here, ε is sampled from a standard normal distribution 𝒩(0, I), while μ and σ are learned parameters. This formulation allows gradients to flow through the stochastic node during training.

The VAE Loss Function: Reconstruction Plus KL Divergence

The VAE optimizes a composite loss function combining two objectives:

1. Reconstruction loss – Identical to classic autoencoders (e.g., MSE or Binary Cross-Entropy), ensuring the decoded output matches the input.
2. KL Divergence – A regularization term that forces the learned distribution to stay close to a unit Gaussian:

KL(N(μ, σ) || N(0, 1)) = -0.5 * Σ(1 + log(σ²) - μ² - σ²)

This regularization encourages a smooth, continuous latent space where similar inputs map to nearby regions, making the embeddings more robust for downstream dimensionality reduction tasks.

Practical Implementation: Extracting Low-Dimensional Embeddings

To use autoencoders and VAEs for dimensionality reduction, you must train the model, extract the latent vectors, and apply them to downstream tasks.

TensorFlow/Keras Convolutional Autoencoder

The following implementation from lessons/4-ComputerVision/09-Autoencoders/AutoencodersTF.ipynb demonstrates building a convolutional autoencoder for image compression:

import tensorflow as tf
from tensorflow.keras import layers, models

def build_encoder(latent_dim=32):
    inputs = layers.Input(shape=(28, 28, 1))
    x = layers.Conv2D(32, 3, activation='relu', padding='same')(inputs)
    x = layers.MaxPooling2D(2, padding='same')(x)
    x = layers.Conv2D(64, 3, activation='relu', padding='same')(x)
    x = layers.MaxPooling2D(2, padding='same')(x)
    x = layers.Flatten()(x)
    latent = layers.Dense(latent_dim, name='latent')(x)
    return models.Model(inputs, latent, name='encoder')

def build_decoder(latent_dim=32, shape_before_flatten=(7, 7, 64)):
    latent_inputs = layers.Input(shape=(latent_dim,))
    x = layers.Dense(7 * 7 * 64)(latent_inputs)
    x = layers.Reshape(shape_before_flatten)(x)
    x = layers.Conv2DTranspose(64, 3, strides=2, activation='relu', padding='same')(x)
    x = layers.Conv2DTranspose(32, 3, strides=2, activation='relu', padding='same')(x)
    outputs = layers.Conv2D(1, 3, activation='sigmoid', padding='same')(x)
    return models.Model(latent_inputs, outputs, name='decoder')

# Build and compile

encoder = build_encoder()
decoder = build_decoder()
autoencoder = models.Model(encoder.input, decoder(encoder.output))
autoencoder.compile(optimizer='adam', loss='mse')

# Train

autoencoder.fit(x_train, x_train, epochs=30, batch_size=256)

# Extract reduced-dimension features

z = encoder.predict(x_test)  # Shape: (n_samples, 32)

PyTorch Variational Autoencoder

For probabilistic embeddings, the PyTorch implementation in lessons/4-ComputerVision/09-Autoencoders/AutoEncodersPyTorch.ipynb defines a fully-connected VAE:

import torch
import torch.nn as nn
import torch.nn.functional as F

class VAE(nn.Module):
    def __init__(self, latent_dim=20):
        super().__init__()
        self.fc1 = nn.Linear(28*28, 400)
        self.fc_mu = nn.Linear(400, latent_dim)
        self.fc_logvar = nn.Linear(400, latent_dim)
        self.fc2 = nn.Linear(latent_dim, 400)
        self.fc3 = nn.Linear(400, 28*28)
    
    def encode(self, x):
        h = F.relu(self.fc1(x))
        return self.fc_mu(h), self.fc_logvar(h)
    
    def reparameterize(self, mu, logvar):
        std = torch.exp(0.5 * logvar)
        eps = torch.randn_like(std)
        return mu + eps * std
    
    def decode(self, z):
        h = F.relu(self.fc2(z))
        return torch.sigmoid(self.fc3(h))
    
    def forward(self, x):
        mu, logvar = self.encode(x.view(-1, 784))
        z = self.reparameterize(mu, logvar)
        return self.decode(z), mu, logvar

def loss_function(recon_x, x, mu, logvar):
    BCE = F.binary_cross_entropy(recon_x, x.view(-1, 784), reduction='sum')
    KLD = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())
    return BCE + KLD

# Training loop

model = VAE()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

for epoch in range(10):
    for batch, _ in train_loader:
        optimizer.zero_grad()
        recon, mu, logvar = model(batch)
        loss = loss_function(recon, batch, mu, logvar)
        loss.backward()
        optimizer.step()

# Extract deterministic embeddings for dimensionality reduction

with torch.no_grad():
    mu, _ = model.encode(x_test.view(-1, 784))
    latent = mu.numpy()  # Use mean as reduced-dimension features

Downstream Dimensionality Reduction Pipeline

Once you have trained your encoder, extract the latent vectors and apply them to visualization or clustering tasks:

# 1. Encode the dataset

z_train = encoder.predict(x_train)  # TensorFlow

# OR

z_train = model.encode(x_train.view(-1, 784))[0].numpy()  # PyTorch

# 2. Visualize with UMAP or t-SNE

import umap
import matplotlib.pyplot as plt

reducer = umap.UMAP(n_components=2, random_state=42)
z_2d = reducer.fit_transform(z_train)

plt.scatter(z_2d[:, 0], z_2d[:, 1], c=y_train, cmap='Spectral', s=5)
plt.title('2D Embedding from Autoencoder Latent Space')
plt.show()

Summary

• Autoencoders learn compressed representations through an encoder-bottleneck-decoder architecture, using reconstruction loss to train the network to preserve essential features in the latent space.
• VAEs extend this approach by modeling the latent space as a probability distribution, using the reparameterization trick and KL divergence to create smooth, continuous embeddings.
• Dimensionality reduction is achieved by extracting the latent vectors (deterministic z for autoencoders, μ for VAEs) and using them as low-dimensional features for visualization, clustering, or downstream classification.
• The microsoft/AI-For-Beginners repository provides complete TensorFlow and PyTorch implementations in lessons/4-ComputerVision/09-Autoencoders/ demonstrating these techniques on standard datasets like MNIST.

Frequently Asked Questions

What is the difference between an autoencoder and a VAE for dimensionality reduction?

A classic autoencoder produces a deterministic latent vector z, while a VAE outputs parameters (mean μ and variance σ) of a probability distribution. For dimensionality reduction, autoencoders provide fixed embeddings, whereas VAEs offer probabilistic embeddings that can be sampled. In practice, use the mean vector μ from a VAE as the deterministic reduced-dimension feature, which tends to produce more robust and smoothly interpolating latent spaces than standard autoencoders.

How do you choose the size of the latent dimension?

The latent dimension represents a trade-off between compression and reconstruction quality. Start with a dimension significantly smaller than your input (e.g., 32 for 784-pixel MNIST images) and monitor the reconstruction loss. If the loss is high, increase the dimension; if you need more aggressive compression, decrease it. According to the implementations in lessons/4-ComputerVision/09-Autoencoders/AutoencodersTF.ipynb, values between 16 and 64 typically work well for image datasets.

Can autoencoders be used for non-image data?

Yes, autoencoders and VAEs are framework-agnostic and work for any high-dimensional data including tabular features, time series, and text embeddings. Replace the convolutional layers in the computer vision examples with fully-connected (dense) layers for tabular data, or use 1D convolutions and LSTM layers for sequential data. The core principle—compressing through a bottleneck and reconstructing—remains identical regardless of input modality.

Why add KL divergence loss in VAEs if we only want dimensionality reduction?

The KL divergence term acts as a regularizer that prevents overfitting to specific training examples and encourages the latent space to follow a standard normal distribution. This creates a smooth, continuous space where similar inputs cluster together and interpolation between points produces meaningful outputs. Without this term, the VAE behaves like a standard autoencoder with a discontinuous latent space, potentially creating "holes" where decoder reconstructions become unreliable.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}