How the Paint Agent Is Trained in the Learning-to-Paint Repository: DDPG Pipeline Explained

The paint agent is trained using a Deep Deterministic Policy Gradient (DDPG) algorithm that learns to predict brush-stroke parameters through interaction with a differentiable painting environment, utilizing an actor-critic architecture with ResNet backbones and a GAN-based auxiliary reward system.

The hzwer/iccv2019-learningtopaint repository implements a reinforcement learning approach to neural painting where an agent learns to reproduce target images through sequential brush strokes. This article explains exactly how the paint agent is trained by examining the DDPG implementation, network architectures, and training loop defined in the baseline_modelfree directory.

The DDPG Training Architecture

The training system consists of four integrated components that implement the Deep Deterministic Policy Gradient algorithm:

• Environment (baseline_modelfree/env.py): The Paint class provides a differentiable canvas, target image loading, and pixel-wise L₂ error rewards.
• Replay Memory (baseline_modelfree/DRL/rpm.py): Stores transition tuples (state, action, reward, next_state, done) for off-policy learning.
• DDPG Agent (baseline_modelfree/DRL/ddpg.py): Contains the actor network that predicts stroke parameters and the critic network that evaluates Q-values, plus a GAN discriminator for auxiliary rewards.
• Training Orchestrator (baseline_modelfree/train.py): Manages the interaction loop, validation intervals, and model checkpointing.

The Training Pipeline

Initialization and Environment Setup

The training process begins in baseline_modelfree/train.py with hyperparameter parsing and object instantiation:

parser = argparse.ArgumentParser(description='Learning to Paint')

# Hyperparameters: batch_size, env_batch, max_step, tau, discount, rmsize

args = parser.parse_args()

from DRL.ddpg import DDPG
from DRL.multi import fastenv

fenv = fastenv(args.max_step, args.env_batch, writer)
agent = DDPG(args.batch_size, args.env_batch, args.max_step,
             args.tau, args.discount, args.rmsize,
             writer, args.resume, args.output)

The fastenv wrapper vectorizes the Paint environment for parallel processing, while DDPG initializes the actor and critic networks along with the replay buffer (rpm). If --resume is specified, the agent loads previous weights from the output directory.

Interaction and Experience Collection

The main training loop follows the standard reinforcement learning interaction pattern defined in baseline_modelfree/train.py:

while step <= train_times:
    step += 1
    if observation is None:
        observation = env.reset()
        agent.reset(observation, noise_factor)
    
    action = agent.select_action(observation, noise_factor=noise_factor)
    observation, reward, done, _ = env.step(action)
    agent.observe(reward, observation, done, step)

In baseline_modelfree/DRL/ddpg.py, the select_action method runs a forward pass through the actor network (self.play) and adds parameter-space noise for exploration when noise_factor > 0. The observe method appends transitions to the replay memory (rpm.append), storing states, actions, rewards, and termination flags for later learning.

Policy and Value Updates

After each episode completes (when reaching max_step or done=True), the script performs learning updates if past the warmup phase:

if step > args.warmup:
    lr = (3e-4, 1e-3)  # Actor and critic learning rates

    for i in range(episode_train_times):
        Q, value_loss = agent.update_policy(lr)

The update_policy method in baseline_modelfree/DRL/ddpg.py implements the core DDPG algorithm through the following steps:

1. Sample a minibatch from self.memory.sample_batch containing stored transitions.
2. Compute target actions using the target actor network, then evaluate target Q-values using the target critic (self.evaluate(next_state, next_action, True)).
3. Calculate TD target: target_q = discount * (1 - terminal) * target_q + step_reward.
4. Update critic by minimizing MSE between current Q-values and the TD target (value_loss.backward()).
5. Update actor by maximizing the critic's Q-value estimate via policy_loss = -pre_q.mean().
6. Soft-update target networks using soft_update(self.actor_target, self.actor, self.tau) with default tau=0.001.

Additionally, each update calls self.update_gan(next_state) to train the auxiliary discriminator defined in baseline_modelfree/DRL/wgan.py, providing supplemental rewards beyond the pixel-wise L₂ loss.

Validation and Checkpointing

Every validate_interval episodes, the script runs evaluation without exploration noise:

reward, dist = evaluate(env, agent.select_action, debug=debug)
agent.save_model(output)

The Evaluator computes mean reward and distance metrics, while DDPG.save_model persists the actor, critic, and GAN weights to the output directory for later resumption or inference.

Network Architectures

The actor and critic use ResNet backbones defined in baseline_modelfree/DRL/actor.py and baseline_modelfree/DRL/critic.py:

• Actor: ResNet(9, 18, 65) — Takes 9-channel input (canvas + target + previous action) and outputs 65-dimensional stroke parameters representing brush location, size, rotation, and color.
• Critic: ResNet_wobn(9, 18, 1) — Uses weight normalization (instead of batch normalization) to estimate Q-values for state-action pairs, producing a scalar value prediction.

The stroke parameters generated by the actor are rendered into actual brush strokes via the decoder network in baseline_modelfree/Renderer/model.py.

Practical Training Examples

Starting Training from Command Line

Launch the training process with specified hyperparameters:

python baseline_modelfree/train.py \
    --batch_size 96 \
    --env_batch 96 \
    --max_step 40 \
    --train_times 2000000 \
    --warmup 400 \
    --tau 0.001 \
    --discount 0.95 \
    --noise_factor 0.05 \
    --validate_interval 50 \
    --output ./model

Loading a Trained Agent for Inference

Use the saved checkpoint to generate paintings without exploration noise:

from baseline_modelfree.DRL.ddpg import DDPG
from baseline_modelfree.DRL.multi import fastenv

agent = DDPG(batch_size=1, env_batch=1, max_step=40,
             tau=0.001, discount=0.95, rmsize=800,
             writer=None, resume='./model', output='./model')
env = fastenv(max_step=40, env_batch=1, writer=None)

obs = env.reset()
for t in range(40):
    action = agent.select_action(obs)  # No noise_factor during inference

    obs, reward, done, _ = env.step(action)
    if done:
        break

# Final canvas available in env.env.canvas[0]

Inspecting Network Architectures

Verify the loaded model structures:

print(agent.actor)   # ResNet(9, depth=18, num_outputs=65)

print(agent.critic)  # ResNet_wobn(9, depth=18, num_outputs=1)

Summary

• The paint agent is trained using DDPG (Deep Deterministic Policy Gradient) with an actor-critic architecture implemented in baseline_modelfree/DRL/ddpg.py.
• The actor (ResNet(9,18,65)) predicts 65-dimensional brush strokes while the critic (ResNet_wobn(9,18,1)) evaluates Q-values for state-action pairs.
• A GAN-based auxiliary reward from baseline_modelfree/DRL/wgan.py supplements the pixel-wise L₂ loss from the Paint environment.
• Soft updates with parameter tau (default 0.001) gradually synchronize target networks with online networks.
• The replay buffer stores transitions for off-policy learning, with a warmup phase (default 400 steps) before learning updates begin.
• Training is orchestrated by baseline_modelfree/train.py, which handles the interaction loop, periodic validation, and model checkpointing every validate_interval episodes.

Frequently Asked Questions

What reinforcement learning algorithm is used to train the paint agent?

The paint agent is trained using the Deep Deterministic Policy Gradient (DDPG) algorithm, as implemented in baseline_modelfree/DRL/ddpg.py. DDPG is an off-policy actor-critic method suitable for continuous action spaces, where the actor network predicts brush-stroke parameters and the critic network estimates the expected return for those actions.

How does the environment calculate rewards during training?

The Paint class in baseline_modelfree/env.py computes rewards based on the pixel-wise L₂ error between the current canvas state and the target image. Additionally, the training pipeline incorporates a GAN-based auxiliary reward from baseline_modelfree/DRL/wgan.py via the update_gan() method, providing learned perceptual feedback to improve stroke quality beyond pixel-level similarity.

What are the critical hyperparameters for training convergence?

Key hyperparameters include tau (0.001 for soft target network updates), discount (0.95 for future reward discounting), noise_factor (0.05 for Ornstein-Uhlenbeck exploration noise), and warmup (400 initial steps with random actions). The actor and critic use learning rates of 3e-4 and 1e-3 respectively during early training, as specified in the update_policy call within baseline_modelfree/train.py.

How do I resume training from a previous checkpoint?

Pass the --resume flag when executing baseline_modelfree/train.py, or set resume=True when instantiating the DDPG class in Python. The constructor automatically loads actor, critic, and GAN weights from the specified output directory, allowing training to continue from the saved state without losing learned parameters.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}