# What Is the GAN-Based Reward Component Based On in Learning to Paint?

> Discover how the GAN-based reward component in Learning to Paint uses a WGAN discriminator to measure Wasserstein distance between generated and ground-truth images for AI art creation.

- Repository: [hzwer/iccv2019-learningtopaint](https://github.com/hzwer/iccv2019-learningtopaint)
- Tags: deep-dive
- Published: 2026-03-03

---

**The GAN-based reward component in the Learning to Paint system is based on a Wasserstein GAN (WGAN) discriminator that measures the learned Wasserstein distance between the generated canvas and the ground-truth image.**

The `hzwer/iccv2019-learningtopaint` repository implements a reinforcement learning agent that learns to synthesize images stroke by stroke. At the core of its reward function lies a **GAN-based reward component** derived from a Wasserstein GAN critic, which provides dense feedback by evaluating how closely the current canvas resembles the target image according to the discriminator's learned metric.

## Architecture of the GAN-Based Reward Component

The reward signal originates from a discriminator network defined in [`baseline_modelfree/DRL/wgan.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/DRL/wgan.py). This network implements a Wasserstein GAN critic that estimates the Earth Mover's distance between the distribution of generated canvases and real images.

### Network Design and Input Processing

The `Discriminator` class processes image pairs concatenated along the channel dimension, creating a 6-channel input tensor (3 channels from the real/ground-truth image plus 3 channels from the generated/fake canvas). The architecture employs **weight normalization** (`weightNorm`) and custom **TReLU** activations for stable training:

```python
class Discriminator(nn.Module):
    def __init__(self):
        super(Discriminator, self).__init__()
        self.conv0 = weightNorm(nn.Conv2d(6, 16, 5, 2, 2))
        self.conv1 = weightNorm(nn.Conv2d(16, 32, 5, 2, 2))
        self.conv2 = weightNorm(nn.Conv2d(32, 64, 5, 2, 2))
        self.conv3 = weightNorm(nn.Conv2d(64, 128, 5, 2, 2))
        self.conv4 = weightNorm(nn.Conv2d(128, 1, 1, 1, 0))
        self.relu0 = TReLU()
        self.relu1 = TReLU()
        self.relu2 = TReLU()
        self.relu3 = TReLU()

```

This convolutional stack outputs a scalar score representing the critic's evaluation of the image pair's similarity.

## Calculating the GAN-Based Reward in DDPG

The GAN-based reward is computed as the difference in discriminator scores between consecutive canvas states, encouraging the agent to make progress toward the target image.

### The `cal_reward` Helper Function

Located in [`baseline_modelfree/DRL/wgan.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/DRL/wgan.py), the `cal_reward` function utilizes a **target network** (`target_netD`)—a lagging copy of the main discriminator updated via soft updates—to generate stable reward signals:

```python
def cal_reward(fake_data, real_data):
    return target_netD(torch.cat([real_data, fake_data], 1))

```

The function concatenates the ground-truth image (`real_data`) with the generated canvas (`fake_data`) along dimension 1, producing the 6-channel input expected by the discriminator.

### Reward Difference Mechanism

In [`baseline_modelfree/DRL/ddpg.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/DRL/ddpg.py), the DDPG agent calculates the GAN reward as the incremental improvement between the previous canvas (`canvas0`) and the new canvas (`canvas1`):

```python
gan_reward = cal_reward(canvas1, gt) - cal_reward(canvas0, gt)

```

This difference formulation ensures the reinforcement learning agent receives positive feedback only when the new stroke improves visual similarity according to the WGAN critic's metric. A higher value indicates the new canvas lies closer to the ground-truth distribution in the Wasserstein space.

## Training Procedure for the GAN-Based Reward Component

The discriminator learns to estimate Wasserstein distances through a gradient-penalty training procedure that maintains Lipschitz continuity.

### WGAN-GP Update Loop

The `update` function in [`baseline_modelfree/DRL/wgan.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/DRL/wgan.py) implements the standard WGAN-GP training algorithm. It computes the critic loss as the difference between fake and real scores, adds a gradient penalty term, and performs soft updates to synchronize the target network:

```python
def update(fake_data, real_data):
    fake = torch.cat([real_data, fake_data], 1)
    real = torch.cat([real_data, real_data], 1)
    D_real = netD(real)
    D_fake = netD(fake)
    gradient_penalty = cal_gradient_penalty(netD, real, fake, real.shape[0])
    D_cost = D_fake.mean() - D_real.mean() + gradient_penalty
    D_cost.backward()
    optimizerD.step()
    soft_update(target_netD, netD, 0.001)
    return D_fake.mean(), D_real.mean(), gradient_penalty

```

The `soft_update` operation blends the target network parameters toward the live discriminator with a factor of 0.001, preventing sudden changes to the reward baseline that could destabilize the reinforcement learning process.

## Summary

- The **GAN-based reward component** is founded on a **Wasserstein GAN discriminator** that estimates the distance between generated canvases and ground-truth images.
- The discriminator architecture in [`baseline_modelfree/DRL/wgan.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/DRL/wgan.py) uses 6-channel concatenated inputs, weight normalization, and TReLU activations.
- Rewards are computed as differences in discriminator scores between consecutive states using the `cal_reward` function and a lagging **target network**.
- Training employs **WGAN-GP** with gradient penalty and soft updates to maintain stable, consistent reward signals for the DDPG agent in [`baseline_modelfree/DRL/ddpg.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/DRL/ddpg.py).

## Frequently Asked Questions

### What type of GAN architecture supports the reward component?

The reward component relies on a **Wasserstein GAN with Gradient Penalty (WGAN-GP)**. This architecture uses a critic network (discriminator) that estimates the Wasserstein distance between distributions rather than classifying real versus fake images, providing more stable gradients for the reinforcement learning agent.

### How does the system prevent unstable reward signals during training?

The implementation utilizes a **target network** (`target_netD`) that is softly updated via `soft_update` with a factor of 0.001. This lagging copy of the discriminator generates the actual reward values, ensuring the reward signal changes gradually and does not destabilize the DDPG policy updates.

### Why is the reward calculated as a difference between two canvas states?

The reward is defined as `cal_reward(canvas1, gt) - cal_reward(canvas0, gt)` to measure **incremental improvement**. This formulation ensures the agent receives positive reinforcement only when a new stroke moves the canvas closer to the ground-truth image according to the learned Wasserstein metric, rather than rewarding absolute similarity alone.

### Which files contain the core implementation of the GAN-based reward?

The primary implementation resides in [`baseline_modelfree/DRL/wgan.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/DRL/wgan.py) (discriminator definition, training, and `cal_reward`) and [`baseline_modelfree/DRL/ddpg.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/DRL/ddpg.py) (reward integration into the RL loop). Equivalent implementations exist in [`baseline/DRL/wgan.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline/DRL/wgan.py) and [`baseline/DRL/ddpg.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline/DRL/ddpg.py) for the non-model-free variant.