# How GAN Generator and Discriminator Training Works: Architecture, Objectives, and Failure Modes > Understand GAN generator and discriminator training a complex minimax game Explore architectures objectives and failure modes like mode collapse and vanishing gradients to build stable models. - Repository: [scutan90/DeepLearning-500-questions](https://github.com/scutan90/DeepLearning-500-questions) - Tags: deep-dive - Published: 2026-03-06 --- **GAN generator and discriminator training operates as a two-player minimax game where the generator maps random noise to synthetic data while the discriminator classifies real versus fake samples, though this equilibrium often collapses into mode collapse, vanishing gradients, or oscillations without careful architectural remedies.** Generative Adversarial Networks (GANs) learn complex data distributions through adversarial learning defined in `ch07_生成对抗网络(GAN)/ch7.md` of the scutan90/DeepLearning-500-questions repository. Mastering GAN generator and discriminator training requires understanding their competing objectives, the alternating stochastic gradient updates that solve them, and the specific failure modes that destabilize the optimization process. ## The Adversarial Minimax Objective A GAN consists of two neural networks engaged in a zero-sum game. The **Generator** \(G\) maps a latent noise vector \(z \sim p_z\) (typically Gaussian) to synthetic samples \(G(z)\), while the **Discriminator** \(D\) outputs a scalar \(D(\cdot) \in [0,1]\) representing the probability that an input is real data versus generated data. According to the original formulation in the repository, the optimization objective follows this minimax equation: \[ \min_{G}\max_{D} V(D,G)=\mathbb{E}_{x\sim p_{data}}[\log D(x)]+\mathbb{E}_{z\sim p_{z}}[\log (1-D(G(z)))] \] The discriminator attempts to **maximize** this value by correctly classifying real samples as real and fake samples as fake. Simultaneously, the generator attempts to **minimize** the same objective by fooling the discriminator into classifying \(G(z)\) as real data. Theoretically, this process converges when the Jensen-Shannon divergence between the real data distribution \(p_{data}\) and the generator distribution \(p_g\) reaches its minimum. ## Training Dynamics and Update Rules In practice, the minimax game is solved through **alternating stochastic gradient updates** rather than simultaneous optimization: 1. **Update Discriminator**: Freeze \(G\) and train \(D\) using a mini-batch of real data and a mini-batch of generated data to maximize classification accuracy. 2. **Update Generator**: Freeze \(D\) and train \(G\) to minimize the probability that \(D\) correctly identifies generated samples as fake. **Balanced updates** are critical for stability. Typical recipes train the discriminator for 1–5 steps for every single generator step to maintain competitive pressure without overpowering the generator. The **non-saturating generator loss** \(-\log D(G(z))\) replaces the original \(\log(1-D(G(z)))\) term to avoid vanishing gradients when the discriminator confidently rejects early generator samples. This modification ensures stronger gradient flow back to the generator during initial training phases. ## Common Failure Modes in GAN Training The adversarial nature of GAN generator and discriminator training creates several characteristic instability patterns documented in `ch07_生成对抗网络(GAN)/ch7.md`. ### Mode Collapse **Mode collapse** occurs when the generator discovers and exploits a small subset of samples that consistently fool the discriminator, producing limited output variety. Rather than modeling the full data distribution, \(G\) collapses onto a few high-probability modes. **Remedies** include **Mini-batch discrimination** (allowing \(D\) to compare samples within a batch), **Unrolled GANs** (which backpropagate through multiple discriminator update steps), and latent-space regularization techniques such as **DRAGAN** or **MAD-GAN**. ### Non-Convergence and Oscillations Unlike standard supervised learning, GAN losses do not necessarily decrease monotonically. **Oscillations** occur when the generator and discriminator enter an unstable equilibrium where losses bounce wildly rather than converging. Stabilization techniques include reducing learning rates, applying **spectral normalization** to the discriminator to constrain Lipschitz constants, or adopting **Wasserstein GAN (WGAN)** with gradient penalty to replace the Jensen-Shannon divergence with the Earth Mover's distance. ### Discriminator Overpowering When the discriminator becomes significantly more capable than the generator, it achieves near-perfect classification. This causes the generator loss to approach \(\log 1\) (zero), creating **vanishing gradients** that stall training. Solutions include training the generator more frequently than the discriminator, adding noise to discriminator inputs to smooth the decision boundary, or using **label smoothing** (e.g., setting real labels to 0.9 instead of 1.0) to prevent the discriminator from becoming overconfident. ### Generator Overpowering Conversely, if the generator outpaces the discriminator, \(D\) may output "real" predictions for all inputs, rendering its feedback useless. This occurs when the discriminator lacks capacity to identify subtle artifacts in generated samples. Mitigation strategies include increasing discriminator capacity (more layers or filters), applying **weight clipping** (as in original WGAN implementations), or ensuring the discriminator has sufficient representational power relative to the generator. ### Gradient Vanishing When using sigmoid outputs with the original minimax loss, gradients can vanish when the discriminator easily distinguishes real from fake samples. This is particularly problematic early in training when generated samples are obviously synthetic. The **non-saturating loss** \(-\log D(G(z))\) addresses this by flipping the generator's objective from minimizing the log-probability of detection to maximizing the log-probability of being classified as real. Alternatively, **WGAN** architectures replace the sigmoid output with a linear activation to provide unbounded gradients. ## Practical Implementation: PyTorch Training Loop The following implementation demonstrates the alternating update strategy with label smoothing and non-saturating loss, following the architectural patterns from the repository: ```python import torch, torch.nn as nn, torch.optim as optim from torchvision import datasets, transforms # ------------------------------------------------- # 1️⃣ Model definitions (very small for illustration) # ------------------------------------------------- class Generator(nn.Module): def __init__(self, nz=100, ngf=64, nc=1): super().__init__() self.main = nn.Sequential( nn.ConvTranspose2d(nz, ngf*8, 4, 1, 0, bias=False), nn.BatchNorm2d(ngf*8), nn.ReLU(True), nn.ConvTranspose2d(ngf*8, ngf*4, 4, 2, 1, bias=False), nn.BatchNorm2d(ngf*4), nn.ReLU(True), nn.ConvTranspose2d(ngf*4, ngf*2, 4, 2, 1, bias=False), nn.BatchNorm2d(ngf*2), nn.ReLU(True), nn.ConvTranspose2d(ngf*2, nc, 4, 2, 1, bias=False), nn.Tanh() ) def forward(self, z): return self.main(z) class Discriminator(nn.Module): def __init__(self, ndf=64, nc=1): super().__init__() self.main = nn.Sequential( nn.Conv2d(nc, ndf, 4, 2, 1, bias=False), nn.LeakyReLU(0.2, inplace=True), nn.Conv2d(ndf, ndf*2, 4, 2, 1, bias=False), nn.BatchNorm2d(ndf*2), nn.LeakyReLU(0.2, inplace=True), nn.Conv2d(ndf*2, ndf*4, 4, 2, 1, bias=False), nn.BatchNorm2d(ndf*4), nn.LeakyReLU(0.2, inplace=True), nn.Conv2d(ndf*4, 1, 4, 1, 0, bias=False) ) def forward(self, x): return self.main(x).view(-1) # ------------------------------------------------- # 2️⃣ Hyper‑parameters & data loader # ------------------------------------------------- device = torch.device("cuda" if torch.cuda.is_available() else "cpu") nz = 100 lr = 2e-4 beta1 = 0.5 batch_size = 64 transform = transforms.Compose([ transforms.ToTensor(), transforms.Normalize([0.5], [0.5]) ]) dset = datasets.MNIST(root='./data', download=True, transform=transform) loader = torch.utils.data.DataLoader(dset, batch_size=batch_size, shuffle=True) G = Generator(nz=nz).to(device) D = Discriminator().to(device) optimG = optim.Adam(G.parameters(), lr=lr, betas=(beta1, 0.999)) optimD = optim.Adam(D.parameters(), lr=lr, betas=(beta1, 0.999)) criterion = nn.BCEWithLogitsLoss() # combines sigmoid + BCE # ------------------------------------------------- # 3️⃣ Training loop (label smoothing + 1 D step / 1 G step) # ------------------------------------------------- real_label = 0.9 # label smoothing fake_label = 0.0 for epoch in range(30): for real_imgs, _ in loader: real_imgs = real_imgs.to(device) # ----- Train Discriminator ----- D.zero_grad() # Real images output_real = D(real_imgs) loss_real = criterion(output_real, torch.full_like(output_real, real_label, device=device)) loss_real.backward() # Fake images noise = torch.randn(real_imgs.size(0), nz, 1, 1, device=device) fake_imgs = G(noise).detach() output_fake = D(fake_imgs) loss_fake = criterion(output_fake, torch.full_like(output_fake, fake_label, device=device)) loss_fake.backward() optimD.step() # ----- Train Generator ----- G.zero_grad() noise = torch.randn(batch_size, nz, 1, 1, device=device) fake_imgs = G(noise) # Use non‑saturating loss: -log D(G(z)) output = D(fake_imgs) loss_G = criterion(output, torch.full_like(output, real_label, device=device)) # want D to think fake is real loss_G.backward() optimG.step() print(f'Epoch {epoch:02d} D_loss: {(loss_real+loss_fake).item():.4f} G_loss: {loss_G.item():.4f}') ``` Key implementation details demonstrated include **alternating updates** (one discriminator step followed by one generator step), **label smoothing** (`real_label = 0.9`), and the **non-saturating generator loss** that prevents early training collapse. ## Summary - GAN generator and discriminator training follows a minimax game where \(D\) maximizes classification accuracy while \(G\) minimizes detection probability, theoretically minimizing the Jensen-Shannon divergence between distributions. - Training requires **alternating stochastic gradient updates**, typically with 1–5 discriminator steps per generator step to maintain balance. - **Mode collapse**, **non-convergence**, and **gradient vanishing** represent the most common failure modes, often caused by capacity imbalances between networks or the inherent instability of the Jensen-Shannon divergence when distribution supports are disjoint. - **Stabilization techniques** include non-saturating losses, label smoothing, spectral normalization, and Wasserstein GAN architectures with gradient penalties. ## Frequently Asked Questions ### What is the minimax objective in GAN generator and discriminator training? The minimax objective is defined as \(\min_{G}\max_{D} V(D,G)=\mathbb{E}_{x\sim p_{data}}[\log D(x)]+\mathbb{E}_{z\sim p_{z}}[\log (1-D(G(z)))]\). The discriminator \(D\) tries to maximize this value by correctly identifying real and fake samples, while the generator \(G\) tries to minimize it by producing samples that fool the discriminator into outputting high "real" probabilities. ### Why does mode collapse occur during GAN training? Mode collapse occurs when the generator discovers a small subset of samples that reliably fool the discriminator, causing it to produce limited variety and ignore other modes of the true data distribution. This happens because the generator optimizes for the specific weaknesses of the current discriminator rather than modeling the full data distribution, and is particularly common when the mapping from low-dimensional noise to high-dimensional images is highly non-linear. ### How can I prevent the discriminator from becoming too strong during training? Prevent discriminator overpowering by using **label smoothing** (setting real labels to 0.9 instead of 1.0), adding noise to discriminator inputs to regularize the decision boundary, or increasing the generator update frequency to 5:1 or higher relative to the discriminator. If gradients still vanish, consider switching to the **Wasserstein GAN** formulation which uses a linear output layer instead of sigmoid to provide unbounded gradients. ### What is the non-saturating generator loss and why is it necessary? The **non-saturating loss** replaces the original generator objective \(\log(1-D(G(z)))\) with \(-\log D(G(z))\), effectively flipping the generator's goal to maximizing the log-probability of being classified as real rather than minimizing the log-probability of being detected as fake. This is necessary because early in training, when the discriminator easily rejects generated samples, the original loss saturates and provides vanishingly small gradients, preventing the generator from learning.