Paint Agent Reward Signal in Learning to Paint: How Relative MSE Improvement Drives Training

The paint agent in the hzwer/iccv2019-learningtopaint repository is trained with a relative improvement reward signal calculated as the normalized decrease in pixel-wise mean-squared error (MSE) between the current canvas and the ground-truth image.

The hzwer/iccv2019-learningtopaint project implements a deep reinforcement learning (DRL) system that teaches an agent to reconstruct images using sequential brush strokes. At the core of this learning process is a dense reward signal that provides immediate feedback after every stroke. This signal incentivizes the agent to minimize reconstruction error efficiently, driving the policy toward strokes that yield maximum visual improvement per step.

How the Reward Signal Is Computed in baseline/env.py

The reward logic resides in the environment implementation and relies on two coordinated methods: cal_dis() for measuring current canvas error, and cal_reward() for quantifying step-wise progress.

Measuring Canvas Distance with cal_dis()

The environment first computes the normalized pixel-wise MSE between the current canvas state and the target ground-truth image. In baseline/env.py (lines 100–101), the cal_dis() method implements this calculation:

def cal_dis(self):
    return (((self.canvas.float() - self.gt.float()) / 255) ** 2) \
             .mean(1).mean(1).mean(1)

This function divides the pixel difference by 255 to normalize the 8-bit color range, squares the errors, and averages across all spatial dimensions and channels to produce a scalar distance metric for each item in the batch.

Calculating Relative Improvement with cal_reward()

The actual reward signal represents the relative reduction in distance from the previous step. Implemented in baseline/env.py (lines 103–107), cal_reward() computes:

def cal_reward(self):
    dis = self.cal_dis()
    reward = (self.lastdis - dis) / (self.ini_dis + 1e-8)
    self.lastdis = dis
    return to_numpy(reward)

Here, self.lastdis stores the MSE from the previous timestep, while self.ini_dis represents the initial distance at episode start. The formula (self.lastdis - dis) / (self.ini_dis + 1e-8) yields a normalized reward where positive values indicate the canvas moved closer to the target, and negative values indicate divergence. The small epsilon (1e-8) prevents division by zero.

Integrating the Reward Signal into DRL Training

The computed reward flows directly into the DRL algorithm through the environment's step interface. When env.step(action) is called in the training loops located in baseline/train.py or baseline_modelfree/train.py, it internally invokes cal_reward() and returns the scalar reward to the agent.

This design provides dense supervision: every brush stroke receives immediate feedback proportional to its contribution toward reducing the overall reconstruction error. The DDPG agent (implemented in baseline/DRL/ddpg.py) uses these rewards to update its policy, favoring actions that produce the largest relative gains in canvas accuracy.

Practical Code Examples

The following examples demonstrate how to access and utilize the reward signal in custom training scripts.

Running a single environment step:

from baseline.env import Paint

# Initialize environment for 4 parallel episodes with 10 steps maximum

env = Paint(batch_size=4, max_step=10)
env.load_data()
obs = env.reset(test=False)                 # Reset to initial canvas state

action = env.action_space * [0]             # Replace with policy network output

obs, reward, done, _ = env.step(action)     # Returns relative MSE improvement

print("Step reward:", reward)              # Positive = canvas improved

Integrating into a DDPG training loop:

from baseline.env import Paint
from baseline.DRL.ddpg import Agent

env = Paint(batch_size=8, max_step=20)
env.load_data()
agent = Agent(state_dim=env.observation_space,
              action_dim=env.action_space,
              max_action=1.0)

state = env.reset(test=False)
for step in range(env.max_step):
    action = agent.select_action(state)
    next_state, reward, done, _ = env.step(action)
    
    # Store transition with relative improvement reward

    agent.store(state, action, reward, next_state, done)
    agent.update()                         # Learns from reward signal

    
    state = next_state
    if done.all():
        break

Summary

• The paint agent receives a relative improvement reward signal based on normalized MSE reduction, not absolute error values.
• cal_dis() in baseline/env.py computes pixel-wise MSE between canvas and ground truth, normalized by 255.
• cal_reward() converts distance changes into a scaled reward using (lastdis - dis) / (ini_dis + 1e-8).
• The reward signal is returned by env.step() and consumed by DRL algorithms like DDPG in baseline/train.py.
• Positive rewards indicate the canvas improved relative to the previous step; negative rewards indicate degradation.

Frequently Asked Questions

What is the mathematical formula for the paint agent's reward signal?

The reward equals the relative decrease in normalized MSE: (previous_distance - current_distance) / (initial_distance + 1e-8). This formula scales the improvement by the initial error magnitude, ensuring rewards are comparable across different images regardless of their complexity or initial canvas state.

Why does the reward use normalized MSE instead of absolute pixel differences?

The code normalizes pixel values by dividing by 255 before squaring (/ 255), which scales the 8-bit RGB range to [0, 1]. This normalization stabilizes training by preventing large squared errors from dominating the gradient updates and ensures the reward magnitude remains consistent across different image resolutions and color distributions.

Where is the reward signal calculated in the source code?

The reward computation resides in baseline/env.py within the cal_reward() method (lines 103–107). The same logic appears in baseline_modelfree/env.py for the model-free variant. Both files define the Paint environment class that exposes this signal through the step() method used by training scripts in baseline/train.py.

How does the reward signal handle the first step of an episode?

During initialization, the environment stores the initial distance in self.ini_dis and sets self.lastdis to this value. At the first step, cal_reward() computes the current distance and calculates reward as (ini_dis - current_dis) / (ini_dis + 1e-8), giving the agent immediate feedback on how much the first brush stroke improved the blank canvas relative to the target.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}