What Is the GAN-Based Reward Component Based On in Learning to Paint?

The GAN-based reward component in the Learning to Paint system is based on a Wasserstein GAN (WGAN) discriminator that measures the learned Wasserstein distance between the generated canvas and the ground-truth image.

The hzwer/iccv2019-learningtopaint repository implements a reinforcement learning agent that learns to synthesize images stroke by stroke. At the core of its reward function lies a GAN-based reward component derived from a Wasserstein GAN critic, which provides dense feedback by evaluating how closely the current canvas resembles the target image according to the discriminator's learned metric.

Architecture of the GAN-Based Reward Component

The reward signal originates from a discriminator network defined in baseline_modelfree/DRL/wgan.py. This network implements a Wasserstein GAN critic that estimates the Earth Mover's distance between the distribution of generated canvases and real images.

Network Design and Input Processing

The Discriminator class processes image pairs concatenated along the channel dimension, creating a 6-channel input tensor (3 channels from the real/ground-truth image plus 3 channels from the generated/fake canvas). The architecture employs weight normalization (weightNorm) and custom TReLU activations for stable training:

class Discriminator(nn.Module):
    def __init__(self):
        super(Discriminator, self).__init__()
        self.conv0 = weightNorm(nn.Conv2d(6, 16, 5, 2, 2))
        self.conv1 = weightNorm(nn.Conv2d(16, 32, 5, 2, 2))
        self.conv2 = weightNorm(nn.Conv2d(32, 64, 5, 2, 2))
        self.conv3 = weightNorm(nn.Conv2d(64, 128, 5, 2, 2))
        self.conv4 = weightNorm(nn.Conv2d(128, 1, 1, 1, 0))
        self.relu0 = TReLU()
        self.relu1 = TReLU()
        self.relu2 = TReLU()
        self.relu3 = TReLU()

This convolutional stack outputs a scalar score representing the critic's evaluation of the image pair's similarity.

Calculating the GAN-Based Reward in DDPG

The GAN-based reward is computed as the difference in discriminator scores between consecutive canvas states, encouraging the agent to make progress toward the target image.

The cal_reward Helper Function

Located in baseline_modelfree/DRL/wgan.py, the cal_reward function utilizes a target network (target_netD)—a lagging copy of the main discriminator updated via soft updates—to generate stable reward signals:

def cal_reward(fake_data, real_data):
    return target_netD(torch.cat([real_data, fake_data], 1))

The function concatenates the ground-truth image (real_data) with the generated canvas (fake_data) along dimension 1, producing the 6-channel input expected by the discriminator.

Reward Difference Mechanism

In baseline_modelfree/DRL/ddpg.py, the DDPG agent calculates the GAN reward as the incremental improvement between the previous canvas (canvas0) and the new canvas (canvas1):

gan_reward = cal_reward(canvas1, gt) - cal_reward(canvas0, gt)

This difference formulation ensures the reinforcement learning agent receives positive feedback only when the new stroke improves visual similarity according to the WGAN critic's metric. A higher value indicates the new canvas lies closer to the ground-truth distribution in the Wasserstein space.

Training Procedure for the GAN-Based Reward Component

The discriminator learns to estimate Wasserstein distances through a gradient-penalty training procedure that maintains Lipschitz continuity.

WGAN-GP Update Loop

The update function in baseline_modelfree/DRL/wgan.py implements the standard WGAN-GP training algorithm. It computes the critic loss as the difference between fake and real scores, adds a gradient penalty term, and performs soft updates to synchronize the target network:

def update(fake_data, real_data):
    fake = torch.cat([real_data, fake_data], 1)
    real = torch.cat([real_data, real_data], 1)
    D_real = netD(real)
    D_fake = netD(fake)
    gradient_penalty = cal_gradient_penalty(netD, real, fake, real.shape[0])
    D_cost = D_fake.mean() - D_real.mean() + gradient_penalty
    D_cost.backward()
    optimizerD.step()
    soft_update(target_netD, netD, 0.001)
    return D_fake.mean(), D_real.mean(), gradient_penalty

The soft_update operation blends the target network parameters toward the live discriminator with a factor of 0.001, preventing sudden changes to the reward baseline that could destabilize the reinforcement learning process.

Summary

• The GAN-based reward component is founded on a Wasserstein GAN discriminator that estimates the distance between generated canvases and ground-truth images.
• The discriminator architecture in baseline_modelfree/DRL/wgan.py uses 6-channel concatenated inputs, weight normalization, and TReLU activations.
• Rewards are computed as differences in discriminator scores between consecutive states using the cal_reward function and a lagging target network.
• Training employs WGAN-GP with gradient penalty and soft updates to maintain stable, consistent reward signals for the DDPG agent in baseline_modelfree/DRL/ddpg.py.

Frequently Asked Questions

What type of GAN architecture supports the reward component?

The reward component relies on a Wasserstein GAN with Gradient Penalty (WGAN-GP). This architecture uses a critic network (discriminator) that estimates the Wasserstein distance between distributions rather than classifying real versus fake images, providing more stable gradients for the reinforcement learning agent.

How does the system prevent unstable reward signals during training?

The implementation utilizes a target network (target_netD) that is softly updated via soft_update with a factor of 0.001. This lagging copy of the discriminator generates the actual reward values, ensuring the reward signal changes gradually and does not destabilize the DDPG policy updates.

Why is the reward calculated as a difference between two canvas states?

The reward is defined as cal_reward(canvas1, gt) - cal_reward(canvas0, gt) to measure incremental improvement. This formulation ensures the agent receives positive reinforcement only when a new stroke moves the canvas closer to the ground-truth image according to the learned Wasserstein metric, rather than rewarding absolute similarity alone.

Which files contain the core implementation of the GAN-based reward?

The primary implementation resides in baseline_modelfree/DRL/wgan.py (discriminator definition, training, and cal_reward) and baseline_modelfree/DRL/ddpg.py (reward integration into the RL loop). Equivalent implementations exist in baseline/DRL/wgan.py and baseline/DRL/ddpg.py for the non-model-free variant.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}