How L2 Reward Is Calculated in the ICCV 2019 Learning to Paint Training Process

The L2 reward is computed as the normalized reduction in pixel-wise mean-squared error between the current canvas and ground-truth image, calculated by dividing the distance improvement by the initial distance plus epsilon.

The hzwer/iccv2019-learningtopaint repository trains a reinforcement learning agent to synthesize images using brush strokes, where the L2 reward serves as the primary signal for reconstruction accuracy. This reward mechanism quantifies how effectively each action reduces the pixel-wise distance between the agent’s current canvas and the target ground-truth image, independent of the GAN-based adversarial reward added later in the DDPG update.

Computing the Current L2 Distance

The environment first calculates the raw reconstruction error using the cal_dis() method defined in baseline_modelfree/env.py (lines 100‑102). This function returns the average squared difference between the current canvas and the ground-truth image, normalized to the range [0,1] by dividing pixel values by 255.

def cal_dis(self):
    return (((self.canvas.float() - self.gt.float()) / 255) ** 2) \
             .mean(1).mean(1).mean(1)      # <-- mean over C, H, W

The normalization ensures that pixel values in [0, 255] are scaled to [0.0, 1.0] before computing the mean-squared error (MSE). The resulting tensor contains one scalar distance value per batch element, representing the current reconstruction quality.

Normalized Reward Calculation

The actual L2 reward is computed in cal_reward() (lines 103‑107 of baseline_modelfree/env.py) by measuring the improvement in distance relative to the initial error. The formula implements:

[ r_t = \frac{d_{t-1} - d_t}{\text{ini_dis} + \epsilon} ]

Where (d_t) is the current L2 distance, (d_{t-1}) is the distance from the previous step, and (\epsilon=1e{-8}) prevents division by zero.

def cal_reward(self):
    dis = self.cal_dis()
    reward = (self.lastdis - dis) / (self.ini_dis + 1e-8)
    self.lastdis = dis
    return to_numpy(reward)

The environment stores the initial distance (ini_dis) when the episode begins and maintains lastdis to track the previous step’s error. Positive rewards indicate the canvas moved closer to the target, while negative rewards indicate divergence.

Explicit L2 Formulation in DDPG

An alternative explicit formulation appears as a comment in baseline_modelfree/DRL/ddpg.py (line 103), showing the raw difference between pre-action and post-action MSE values:

# L2_reward = ((canvas0 - gt) ** 2).mean(1).mean(1).mean(1) \

#             - ((canvas1 - gt) ** 2).mean(1).mean(1).mean(1)

While this version omits the normalization by ini_dis, it demonstrates the underlying mathematical principle: the reward equals the reduction in pixel-wise squared error before and after applying a stroke action.

Practical Code Examples

Obtaining the L2 Reward from the Environment

The training loop retrieves the L2 reward through the environment’s step() method, which internally calls cal_reward():

# env = Paint(batch_size=32, max_step=40)   # defined in baseline_modelfree/env.py

obs = env.reset()                     # returns (canvas, gt, step-counter) tensor

for t in range(env.max_step):
    action = agent.select_action(obs)  # shape: (batch, 13) – stroke parameters

    obs, l2_reward, done, _ = env.step(action)  # l2_reward is a numpy array

    print(f"Step {t+1}: L2 reward = {l2_reward.mean():.5f}")
    if done:
        break

Re-creating the L2 Reward Manually

You can replicate the exact reward computation outside the environment using the same normalization and epsilon handling:

import torch

def l2_reward(canvas_prev, canvas_next, gt, ini_dis, eps=1e-8):
    # normalize to [0,1] as in cal_dis()

    def mse(a, b):
        return (((a.float() - b.float()) / 255) ** 2).mean(1).mean(1).mean(1)

    d_prev = mse(canvas_prev, gt)
    d_next = mse(canvas_next, gt)
    return ((d_prev - d_next) / (ini_dis + eps)).cpu().numpy()

Using the Explicit L2-Only Formula

For debugging or ablation studies, the unnormalized L2 difference can be computed directly:

def l2_reward_explicit(canvas0, canvas1, gt):
    # canvas0 – before action, canvas1 – after action

    loss0 = ((canvas0 - gt) ** 2).mean(1).mean(1).mean(1)
    loss1 = ((canvas1 - gt) ** 2).mean(1).mean(1).mean(1)
    return (loss0 - loss1).cpu().numpy()

Summary

• The L2 reward quantifies reconstruction improvement as the normalized reduction in MSE between canvas and ground-truth images.
• cal_dis() in baseline_modelfree/env.py computes the raw pixel-wise distance, normalizing pixel values by 255 to scale inputs to [0,1].
• cal_reward() divides the distance improvement by the initial distance plus 1e-8 to produce scale-invariant rewards that encourage consistent convergence regardless of image complexity.
• The commented explicit formula in ddpg.py reveals the underlying mathematical difference between pre-action and post-action reconstruction errors.

Frequently Asked Questions

Why is the L2 reward normalized by the initial distance?

Normalizing by ini_dis ensures that the reward magnitude remains consistent across different images regardless of their initial reconstruction difficulty. Without this scaling, high-complexity images with large initial errors would dominate the gradient updates, while simple images would produce vanishingly small rewards.

What is the purpose of the 1e-8 epsilon in the reward denominator?

The epsilon value prevents division-by-zero errors when the initial distance is extremely small or zero, which can occur with blank canvases or uniform target images. This numerical safeguard ensures stable training without affecting the reward scale for typical non-zero initial distances.

How does the L2 reward interact with the GAN-based adversarial reward?

According to the implementation in baseline_modelfree/DRL/ddpg.py, the L2 reward provides the primary reconstruction signal during the environment step, while the GAN reward is computed separately and added later during the DDPG update phase. This two-stage approach allows the agent to first learn basic stroke placement via reconstruction error before incorporating adversarial feedback.

Can the L2 reward be used without the initial distance normalization?

Yes, the commented code in ddpg.py shows that the raw difference between squared errors (loss0 - loss1) can serve as a valid reward signal. However, this unnormalized version lacks the scale invariance benefits of the default implementation and may cause training instability when the dataset contains images with widely varying complexity levels.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}