How GAN Generator and Discriminator Training Works: Architecture, Objectives, and Failure Modes

GAN generator and discriminator training operates as a two-player minimax game where the generator maps random noise to synthetic data while the discriminator classifies real versus fake samples, though this equilibrium often collapses into mode collapse, vanishing gradients, or oscillations without careful architectural remedies.

Generative Adversarial Networks (GANs) learn complex data distributions through adversarial learning defined in ch07_生成对抗网络(GAN)/ch7.md of the scutan90/DeepLearning-500-questions repository. Mastering GAN generator and discriminator training requires understanding their competing objectives, the alternating stochastic gradient updates that solve them, and the specific failure modes that destabilize the optimization process.

The Adversarial Minimax Objective

A GAN consists of two neural networks engaged in a zero-sum game. The Generator (G) maps a latent noise vector (z \sim p_z) (typically Gaussian) to synthetic samples (G(z)), while the Discriminator (D) outputs a scalar (D(\cdot) \in [0,1]) representing the probability that an input is real data versus generated data.

According to the original formulation in the repository, the optimization objective follows this minimax equation:

[ \min_{G}\max_{D} V(D,G)=\mathbb{E}{x\sim p{data}}[\log D(x)]+\mathbb{E}{z\sim p{z}}[\log (1-D(G(z)))] ]

The discriminator attempts to maximize this value by correctly classifying real samples as real and fake samples as fake. Simultaneously, the generator attempts to minimize the same objective by fooling the discriminator into classifying (G(z)) as real data. Theoretically, this process converges when the Jensen-Shannon divergence between the real data distribution (p_{data}) and the generator distribution (p_g) reaches its minimum.

Training Dynamics and Update Rules

In practice, the minimax game is solved through alternating stochastic gradient updates rather than simultaneous optimization:

1. Update Discriminator: Freeze (G) and train (D) using a mini-batch of real data and a mini-batch of generated data to maximize classification accuracy.
2. Update Generator: Freeze (D) and train (G) to minimize the probability that (D) correctly identifies generated samples as fake.

Balanced updates are critical for stability. Typical recipes train the discriminator for 1–5 steps for every single generator step to maintain competitive pressure without overpowering the generator.

The non-saturating generator loss (-\log D(G(z))) replaces the original (\log(1-D(G(z)))) term to avoid vanishing gradients when the discriminator confidently rejects early generator samples. This modification ensures stronger gradient flow back to the generator during initial training phases.

Common Failure Modes in GAN Training

The adversarial nature of GAN generator and discriminator training creates several characteristic instability patterns documented in ch07_生成对抗网络(GAN)/ch7.md.

Mode Collapse

Mode collapse occurs when the generator discovers and exploits a small subset of samples that consistently fool the discriminator, producing limited output variety. Rather than modeling the full data distribution, (G) collapses onto a few high-probability modes.

Remedies include Mini-batch discrimination (allowing (D) to compare samples within a batch), Unrolled GANs (which backpropagate through multiple discriminator update steps), and latent-space regularization techniques such as DRAGAN or MAD-GAN.

Non-Convergence and Oscillations

Unlike standard supervised learning, GAN losses do not necessarily decrease monotonically. Oscillations occur when the generator and discriminator enter an unstable equilibrium where losses bounce wildly rather than converging.

Stabilization techniques include reducing learning rates, applying spectral normalization to the discriminator to constrain Lipschitz constants, or adopting Wasserstein GAN (WGAN) with gradient penalty to replace the Jensen-Shannon divergence with the Earth Mover's distance.

Discriminator Overpowering

When the discriminator becomes significantly more capable than the generator, it achieves near-perfect classification. This causes the generator loss to approach (\log 1) (zero), creating vanishing gradients that stall training.

Solutions include training the generator more frequently than the discriminator, adding noise to discriminator inputs to smooth the decision boundary, or using label smoothing (e.g., setting real labels to 0.9 instead of 1.0) to prevent the discriminator from becoming overconfident.

Generator Overpowering

Conversely, if the generator outpaces the discriminator, (D) may output "real" predictions for all inputs, rendering its feedback useless. This occurs when the discriminator lacks capacity to identify subtle artifacts in generated samples.

Mitigation strategies include increasing discriminator capacity (more layers or filters), applying weight clipping (as in original WGAN implementations), or ensuring the discriminator has sufficient representational power relative to the generator.

Gradient Vanishing

When using sigmoid outputs with the original minimax loss, gradients can vanish when the discriminator easily distinguishes real from fake samples. This is particularly problematic early in training when generated samples are obviously synthetic.

The non-saturating loss (-\log D(G(z))) addresses this by flipping the generator's objective from minimizing the log-probability of detection to maximizing the log-probability of being classified as real. Alternatively, WGAN architectures replace the sigmoid output with a linear activation to provide unbounded gradients.

Practical Implementation: PyTorch Training Loop

The following implementation demonstrates the alternating update strategy with label smoothing and non-saturating loss, following the architectural patterns from the repository:

import torch, torch.nn as nn, torch.optim as optim
from torchvision import datasets, transforms

# -------------------------------------------------

# 1️⃣  Model definitions (very small for illustration)

# -------------------------------------------------

class Generator(nn.Module):
    def __init__(self, nz=100, ngf=64, nc=1):
        super().__init__()
        self.main = nn.Sequential(
            nn.ConvTranspose2d(nz, ngf*8, 4, 1, 0, bias=False),
            nn.BatchNorm2d(ngf*8), nn.ReLU(True),
            nn.ConvTranspose2d(ngf*8, ngf*4, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf*4), nn.ReLU(True),
            nn.ConvTranspose2d(ngf*4, ngf*2, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf*2), nn.ReLU(True),
            nn.ConvTranspose2d(ngf*2, nc, 4, 2, 1, bias=False),
            nn.Tanh()
        )
    def forward(self, z):
        return self.main(z)

class Discriminator(nn.Module):
    def __init__(self, ndf=64, nc=1):
        super().__init__()
        self.main = nn.Sequential(
            nn.Conv2d(nc, ndf, 4, 2, 1, bias=False),
            nn.LeakyReLU(0.2, inplace=True),
            nn.Conv2d(ndf, ndf*2, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ndf*2), nn.LeakyReLU(0.2, inplace=True),
            nn.Conv2d(ndf*2, ndf*4, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ndf*4), nn.LeakyReLU(0.2, inplace=True),
            nn.Conv2d(ndf*4, 1, 4, 1, 0, bias=False)
        )
    def forward(self, x):
        return self.main(x).view(-1)

# -------------------------------------------------

# 2️⃣  Hyper‑parameters & data loader

# -------------------------------------------------

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
nz = 100
lr = 2e-4
beta1 = 0.5
batch_size = 64

transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize([0.5], [0.5])
])
dset = datasets.MNIST(root='./data', download=True, transform=transform)
loader = torch.utils.data.DataLoader(dset, batch_size=batch_size, shuffle=True)

G = Generator(nz=nz).to(device)
D = Discriminator().to(device)

optimG = optim.Adam(G.parameters(), lr=lr, betas=(beta1, 0.999))
optimD = optim.Adam(D.parameters(), lr=lr, betas=(beta1, 0.999))
criterion = nn.BCEWithLogitsLoss()      # combines sigmoid + BCE

# -------------------------------------------------

# 3️⃣  Training loop (label smoothing + 1 D step / 1 G step)

# -------------------------------------------------

real_label = 0.9   # label smoothing

fake_label = 0.0

for epoch in range(30):
    for real_imgs, _ in loader:
        real_imgs = real_imgs.to(device)

        # ----- Train Discriminator -----

        D.zero_grad()
        # Real images

        output_real = D(real_imgs)
        loss_real = criterion(output_real, torch.full_like(output_real, real_label, device=device))
        loss_real.backward()

        # Fake images

        noise = torch.randn(real_imgs.size(0), nz, 1, 1, device=device)
        fake_imgs = G(noise).detach()
        output_fake = D(fake_imgs)
        loss_fake = criterion(output_fake, torch.full_like(output_fake, fake_label, device=device))
        loss_fake.backward()
        optimD.step()

        # ----- Train Generator -----

        G.zero_grad()
        noise = torch.randn(batch_size, nz, 1, 1, device=device)
        fake_imgs = G(noise)
        # Use non‑saturating loss: -log D(G(z))

        output = D(fake_imgs)
        loss_G = criterion(output, torch.full_like(output, real_label, device=device))  # want D to think fake is real

        loss_G.backward()
        optimG.step()
    print(f'Epoch {epoch:02d}  D_loss: {(loss_real+loss_fake).item():.4f}  G_loss: {loss_G.item():.4f}')

Key implementation details demonstrated include alternating updates (one discriminator step followed by one generator step), label smoothing (real_label = 0.9), and the non-saturating generator loss that prevents early training collapse.

Summary

• GAN generator and discriminator training follows a minimax game where (D) maximizes classification accuracy while (G) minimizes detection probability, theoretically minimizing the Jensen-Shannon divergence between distributions.
• Training requires alternating stochastic gradient updates, typically with 1–5 discriminator steps per generator step to maintain balance.
• Mode collapse, non-convergence, and gradient vanishing represent the most common failure modes, often caused by capacity imbalances between networks or the inherent instability of the Jensen-Shannon divergence when distribution supports are disjoint.
• Stabilization techniques include non-saturating losses, label smoothing, spectral normalization, and Wasserstein GAN architectures with gradient penalties.

Frequently Asked Questions

What is the minimax objective in GAN generator and discriminator training?

The minimax objective is defined as (\min_{G}\max_{D} V(D,G)=\mathbb{E}{x\sim p{data}}[\log D(x)]+\mathbb{E}{z\sim p{z}}[\log (1-D(G(z)))]). The discriminator (D) tries to maximize this value by correctly identifying real and fake samples, while the generator (G) tries to minimize it by producing samples that fool the discriminator into outputting high "real" probabilities.

Why does mode collapse occur during GAN training?

Mode collapse occurs when the generator discovers a small subset of samples that reliably fool the discriminator, causing it to produce limited variety and ignore other modes of the true data distribution. This happens because the generator optimizes for the specific weaknesses of the current discriminator rather than modeling the full data distribution, and is particularly common when the mapping from low-dimensional noise to high-dimensional images is highly non-linear.

How can I prevent the discriminator from becoming too strong during training?

Prevent discriminator overpowering by using label smoothing (setting real labels to 0.9 instead of 1.0), adding noise to discriminator inputs to regularize the decision boundary, or increasing the generator update frequency to 5:1 or higher relative to the discriminator. If gradients still vanish, consider switching to the Wasserstein GAN formulation which uses a linear output layer instead of sigmoid to provide unbounded gradients.

What is the non-saturating generator loss and why is it necessary?

The non-saturating loss replaces the original generator objective (\log(1-D(G(z)))) with (-\log D(G(z))), effectively flipping the generator's goal to maximizing the log-probability of being classified as real rather than minimizing the log-probability of being detected as fake. This is necessary because early in training, when the discriminator easily rejects generated samples, the original loss saturates and provides vanishingly small gradients, preventing the generator from learning.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}