Implementing Generative Adversarial Networks (GANs) for Image Synthesis: A Complete Guide with TensorFlow and PyTorch

Build a working DCGAN for image generation using the Microsoft AI-For-Beginners curriculum with complete, runnable code examples in both TensorFlow/Keras and PyTorch.

The AI-For-Beginners repository by Microsoft provides a comprehensive, beginner-friendly introduction to generative adversarial networks (GANs) for image synthesis. This guide walks through the exact implementation patterns found in the official lesson materials, covering architecture design, the adversarial training loop, and practical code you can run immediately.

What Are GANs and How Do They Generate Images?

A Generative Adversarial Network consists of two neural networks locked in a minimax game: a Generator that creates synthetic images from random noise, and a Discriminator that learns to distinguish real images from fakes. Through adversarial training, the Generator progressively improves its ability to synthesize photorealistic images.

According to the AI-For-Beginners source code, the key educational resources are:

• Theory and architecture documentation: [lessons/4-ComputerVision/10-GANs/README.md](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/10-GANs/README.md)
• TensorFlow/Keras implementation: lessons/4-ComputerVision/10-GANs/GANTF.ipynb
• PyTorch implementation: lessons/4-ComputerVision/10-GANs/GANPyTorch.ipynb

DCGAN Architecture for Image Synthesis

The Deep Convolutional GAN (DCGAN) is the standard architecture for image generation tasks. The Microsoft curriculum implements this pattern with specific layer configurations optimized for 28×28 MNIST images.

Generator Network

The Generator maps a latent vector z (typically 100-dimensional) to a synthetic image through transposed convolutions:

Stage	Operation	Output Shape
Input	Dense projection + reshape	7×7×256
Block 1	Conv2DTranspose (128 filters)	7×7×128
Block 2	Conv2DTranspose (64 filters, stride 2)	14×14×64
Output	Conv2DTranspose (1 channel, stride 2) + Tanh	28×28×1

Discriminator Network

The Discriminator acts as a binary classifier with a mirrored convolutional structure:

Stage	Operation	Output Shape
Input	28×28×1 image	28×28×1
Block 1	Conv2D (64 filters, stride 2) + LeakyReLU	14×14×64
Block 2	Conv2D (128 filters, stride 2) + LeakyReLU	7×7×128
Output	Flatten + Dense + Sigmoid	Scalar probability

TensorFlow/Keras Implementation of GANs for Image Synthesis

The following implementation matches the official GANTF.ipynb notebook structure, using functional and sequential APIs for clarity:

import tensorflow as tf
from tensorflow.keras import layers, models, optimizers
import numpy as np

# Configuration

latent_dim = 100
img_shape = (28, 28, 1)  # MNIST grayscale format

def build_generator():
    """Builds the generator network as defined in AI-For-Beginners."""
    model = models.Sequential(name="Generator")
    
    # Foundation: dense projection to spatial form

    model.add(layers.Dense(7 * 7 * 256, use_bias=False, input_shape=(latent_dim,)))
    model.add(layers.BatchNormalization())
    model.add(layers.ReLU())
    model.add(layers.Reshape((7, 7, 256)))
    
    # Upsampling block 1: 7×7 → 7×7 (feature expansion)

    model.add(layers.Conv2DTranspose(128, 5, strides=1, padding='same', use_bias=False))
    model.add(layers.BatchNormalization())
    model.add(layers.ReLU())
    
    # Upsampling block 2: 7×7 → 14×14

    model.add(layers.Conv2DTranspose(64, 5, strides=2, padding='same', use_bias=False))
    model.add(layers.BatchNormalization())
    model.add(layers.ReLU())
    
    # Output: 14×14 → 28×28 with tanh activation

    model.add(layers.Conv2DTranspose(1, 5, strides=2, padding='same',
                                     activation='tanh', use_bias=False))
    return model

def build_discriminator():
    """Builds the discriminator network with leaky activations."""
    model = models.Sequential(name="Discriminator")
    
    # Downsampling block 1

    model.add(layers.Conv2D(64, 5, strides=2, padding='same',
                            input_shape=img_shape))
    model.add(layers.LeakyReLU(alpha=0.2))
    model.add(layers.Dropout(0.3))
    
    # Downsampling block 2

    model.add(layers.Conv2D(128, 5, strides=2, padding='same'))
    model.add(layers.LeakyReLU(alpha=0.2))
    model.add(layers.Dropout(0.3))
    
    # Classification head

    model.add(layers.Flatten())
    model.add(layers.Dense(1, activation='sigmoid'))
    return model

# Instantiate networks

generator = build_generator()
discriminator = build_discriminator()
discriminator.compile(loss='binary_crossentropy',
                      optimizer=optimizers.Adam(1e-4))

# Combined GAN: generator → discriminator (frozen during generator training)

discriminator.trainable = False
gan_input = layers.Input(shape=(latent_dim,))
gan_output = discriminator(generator(gan_input))
gan = models.Model(gan_input, gan_output, name="GAN")
gan.compile(loss='binary_crossentropy', optimizer=optimizers.Adam(1e-4))

Training Loop for TensorFlow GAN

The adversarial training procedure alternates between discriminator and generator updates:

# Load and preprocess MNIST

(train_images, _) = tf.keras.datasets.mnist.load_data()
train_images = (train_images.astype('float32') - 127.5) / 127.5
train_images = np.expand_dims(train_images, axis=-1)

batch_size, epochs = 64, 10000
half_batch = batch_size // 2

for epoch in range(epochs):
    # --- Discriminator training ---

    # Sample real images

    idx = np.random.randint(0, train_images.shape[0], half_batch)
    real_imgs = train_images[idx]
    
    # Generate fake images from random noise

    noise = np.random.normal(0, 1, (half_batch, latent_dim))
    fake_imgs = generator.predict(noise, verbose=0)
    
    # Train discriminator on both classes

    d_loss_real = discriminator.train_on_batch(real_imgs, np.ones((half_batch, 1)))
    d_loss_fake = discriminator.train_on_batch(fake_imgs, np.zeros((half_batch, 1)))
    d_loss = 0.5 * (d_loss_real + d_loss_fake)
    
    # --- Generator training ---

    # Generate fresh noise samples

    noise = np.random.normal(0, 1, (batch_size, latent_dim))
    
    # Train generator to fool discriminator (target = 1 for "real")

    g_loss = gan.train_on_batch(noise, np.ones((batch_size, 1)))
    
    if epoch % 1000 == 0:
        print(f'Epoch {epoch:5d} | D loss: {d_loss:.4f} | G loss: {g_loss:.4f}')

PyTorch Implementation of GANs for Image Synthesis

The PyTorch version in GANPyTorch.ipynb uses equivalent operations with explicit manual training steps:

import torch
import torch.nn as nn
import torch.optim as optim
from torchvision import datasets, transforms

latent_dim = 100
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

class Generator(nn.Module):
    """PyTorch generator with ConvTranspose2d upsampling."""
    def __init__(self):
        super().__init__()
        self.net = nn.Sequential(
            # Project and reshape

            nn.Linear(latent_dim, 256 * 7 * 7),
            nn.BatchNorm1d(256 * 7 * 7),
            nn.ReLU(True),
            nn.Unflatten(1, (256, 7, 7)),
            
            # 7×7 → 14×14

            nn.ConvTranspose2d(256, 128, 4, stride=2, padding=1),
            nn.BatchNorm2d(128),
            nn.ReLU(True),
            
            # 14×14 → 28×28 (intermediate)

            nn.ConvTranspose2d(128, 64, 4, stride=2, padding=1),
            nn.BatchNorm2d(64),
            nn.ReLU(True),
            
            # Final: slightly adjust to exact 28×28

            nn.ConvTranspose2d(64, 1, 4, stride=2, padding=1),
            nn.Tanh()  # Output range [-1, 1]

        )
    
    def forward(self, z):
        return self.net(z)

class Discriminator(nn.Module):
    """PyTorch discriminator with LeakyReLU activations."""
    def __init__(self):
        super().__init__()
        self.net = nn.Sequential(
            nn.Conv2d(1, 64, 4, stride=2, padding=1),
            nn.LeakyReLU(0.2, inplace=True),
            nn.Dropout2d(0.3),
            
            nn.Conv2d(64, 128, 4, stride=2, padding=1),
            nn.LeakyReLU(0.2, inplace=True),
            nn.Dropout2d(0.3),
            
            nn.Flatten(),
            nn.Linear(128 * 7 * 7, 1),
            nn.Sigmoid()
        )
    
    def forward(self, img):
        return self.net(img)

# Instantiate and move to device

generator = Generator().to(device)
discriminator = Discriminator().to(device)

criterion = nn.BCELoss()
opt_G = optim.Adam(generator.parameters(), lr=2e-4)
opt_D = optim.Adam(discriminator.parameters(), lr=2e-4)

Training Loop for PyTorch GAN

Note the explicit gradient handling with zero_grad(), backward(), and step():

# Data loading

transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize([0.5], [0.5])
])

dataloader = torch.utils.data.DataLoader(
    datasets.MNIST('.', train=True, download=True, transform=transform),
    batch_size=64, shuffle=True
)

for epoch in range(10000):
    for real_imgs, _ in dataloader:
        real_imgs = real_imgs.to(device)
        batch_size = real_imgs.size(0)
        
        real_labels = torch.ones(batch_size, 1, device=device)
        fake_labels = torch.zeros(batch_size, 1, device=device)
        
        # ----- Train Discriminator -----

        opt_D.zero_grad()
        
        # Loss on real images

        outputs_real = discriminator(real_imgs)
        d_loss_real = criterion(outputs_real, real_labels)
        
        # Loss on fake images (detach prevents generator gradients)

        z = torch.randn(batch_size, latent_dim, device=device)
        fake_imgs = generator(z)
        outputs_fake = discriminator(fake_imgs.detach())
        d_loss_fake = criterion(outputs_fake, fake_labels)
        
        d_loss = (d_loss_real + d_loss_fake) / 2
        d_loss.backward()
        opt_D.step()
        
        # ----- Train Generator -----

        opt_G.zero_grad()
        
        # Generator wants discriminator to classify fakes as real

        outputs_g = discriminator(fake_imgs)  # No detach() — train generator

        g_loss = criterion(outputs_g, real_labels)
        
        g_loss.backward()
        opt_G.step()
    
    if epoch % 1000 == 0:
        print(f'Epoch {epoch:5d} | D loss: {d_loss.item():.4f} | G loss: {g_loss.item():.4f}')

Critical Implementation Details

The Generator's Fooling Objective

Both frameworks implement the same mathematical objective: the Generator minimizes log(1 − D(G(z))) by training against labels of 1 (real). In practice, this is equivalent to maximizing log D(G(z)) — the probability that the discriminator assigns to generated images.

Discriminator Weight Freezing in TensorFlow

The line discriminator.trainable = False before compiling the combined GAN is essential. It ensures that when gan.train_on_batch() is called, gradients flow only through the generator's weights.

Detach() in PyTorch

The .detach() method creates a tensor without gradient history. This is applied to fake_imgs during discriminator training to prevent generator weights from receiving gradients in that phase.

Common Pitfalls in GAN Image Synthesis

Based on the AI-For-Beginners curriculum and established best practices, watch for these issues:

Problem	Diagnostic	Solution
Mode collapse	Generator outputs identical images regardless of input noise	Implement mini-batch discrimination or feature matching loss
Vanishing gradients	Discriminator loss drops to 0, generator loss plateaus	Switch to WGAN-GP loss with gradient penalty for more stable training dynamics
Training imbalance	One network dominates immediately	Use TTUR (Two Time-Scale Update Rule): set generator learning rate higher than discriminator
Checkerboard artifacts	Grid-like patterns in generated images	Use stride-2 convolutions with appropriate kernel sizes; avoid mismatching stride and kernel
High-resolution instability	Training diverges above 64×64 resolution	Apply progressive growing: start at low resolution, gradually add layers

Extending Your GAN Implementation

The foundation in GANTF.ipynb and GANPyTorch.ipynb supports several advanced directions:

• Conditional GANs: Concatenate one-hot class labels to the latent vector for controlled generation of specific digits
• CIFAR-10/Custom datasets: Modify img_shape to (32, 32, 3) and adjust generator/discriminator depths
• Style-based architectures: Replace the single latent vector with a learned constant and adaptive instance normalization (StyleGAN approach)
• Wasserstein loss: Replace BinaryCrossentropy with a critic-based loss and weight clipping or gradient penalty

The AI-For-Beginners GAN lesson also covers neural style transfer using pretrained VGG features for content and style loss computation — see the README for architectural details.

Summary

Implementing generative adversarial networks for image synthesis using the Microsoft AI-For-Beginners repository provides:

• Complete DCGAN architectures in both TensorFlow/Keras and PyTorch with matching layer configurations
• Explicit adversarial training loops that alternate discriminator and generator updates while managing gradient flow
• Practical MNIST examples that generate recognizable digits within thousands of iterations
• Clear pathways to advanced techniques including conditional generation, WGAN-GP, and progressive growing

The provided notebooks at lessons/4-ComputerVision/10-GANs/ contain the full implementations with additional visualization code for monitoring training progress.

Frequently Asked Questions

What is the difference between a standard GAN and a DCGAN?

A DCGAN (Deep Convolutional GAN) replaces the fully-connected layers in early GAN implementations with strided convolutions in the discriminator and transposed convolutions in the generator. As implemented in the AI-For-Beginners curriculum, this architectural change stabilizes training and produces higher-quality images by leveraging spatial structure through convolutional operations.

Why does the generator use tanh activation for output?

The tanh activation constrains pixel values to the range [-1, 1], matching the normalized input distribution. The training code applies (images - 127.5) / 127.5 to scale MNIST from [0, 255] to [-1, 1]. This symmetric range centers the data and improves gradient flow compared to sigmoid's [0, 1] range.

How long does it take to train a functional GAN on MNIST?

With the hyperparameters from GANTF.ipynb — batch size 64, learning rate 1e-4, on modern hardware — coherent digit generation typically emerges within 5-10 epochs (a few minutes on GPU). High-quality, diverse outputs require 50-100+ epochs. The curriculum recommends monitoring both losses: healthy training shows oscillating discriminator loss around 0.5 and gradually decreasing or stable generator loss.

Can these implementations scale to high-resolution image generation?

The basic DCGAN architecture struggles beyond 64×64 resolution due to training instability. For high-resolution GAN image synthesis, the Microsoft curriculum suggests progressive growing techniques where both networks start at 4×4 resolution and layers are incrementally added. This approach, combined with WGAN-GP loss modifications, enables generation of 1024×1024 images as demonstrated in contemporary research.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}