# How the Paint Agent Is Trained in the Learning-to-Paint Repository: DDPG Pipeline Explained

> Learn how the paint agent is trained in the hzwer/iccv2019-learningtopaint repository. Discover the DDPG pipeline and actor-critic architecture behind this innovative approach.

- Repository: [hzwer/iccv2019-learningtopaint](https://github.com/hzwer/iccv2019-learningtopaint)
- Tags: deep-dive
- Published: 2026-03-03

---

**The paint agent is trained using a Deep Deterministic Policy Gradient (DDPG) algorithm that learns to predict brush-stroke parameters through interaction with a differentiable painting environment, utilizing an actor-critic architecture with ResNet backbones and a GAN-based auxiliary reward system.**

The [hzwer/iccv2019-learningtopaint](https://github.com/hzwer/iccv2019-learningtopaint) repository implements a reinforcement learning approach to neural painting where an agent learns to reproduce target images through sequential brush strokes. This article explains exactly how the paint agent is trained by examining the DDPG implementation, network architectures, and training loop defined in the `baseline_modelfree` directory.

## The DDPG Training Architecture

The training system consists of four integrated components that implement the Deep Deterministic Policy Gradient algorithm:

- **Environment** ([`baseline_modelfree/env.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/env.py)): The `Paint` class provides a differentiable canvas, target image loading, and pixel-wise L₂ error rewards.
- **Replay Memory** ([`baseline_modelfree/DRL/rpm.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/DRL/rpm.py)): Stores transition tuples *(state, action, reward, next_state, done)* for off-policy learning.
- **DDPG Agent** ([`baseline_modelfree/DRL/ddpg.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/DRL/ddpg.py)): Contains the **actor** network that predicts stroke parameters and the **critic** network that evaluates Q-values, plus a GAN discriminator for auxiliary rewards.
- **Training Orchestrator** ([`baseline_modelfree/train.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/train.py)): Manages the interaction loop, validation intervals, and model checkpointing.

## The Training Pipeline

### Initialization and Environment Setup

The training process begins in [`baseline_modelfree/train.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/train.py) with hyperparameter parsing and object instantiation:

```python
parser = argparse.ArgumentParser(description='Learning to Paint')

# Hyperparameters: batch_size, env_batch, max_step, tau, discount, rmsize

args = parser.parse_args()

from DRL.ddpg import DDPG
from DRL.multi import fastenv

fenv = fastenv(args.max_step, args.env_batch, writer)
agent = DDPG(args.batch_size, args.env_batch, args.max_step,
             args.tau, args.discount, args.rmsize,
             writer, args.resume, args.output)

```

The `fastenv` wrapper vectorizes the `Paint` environment for parallel processing, while `DDPG` initializes the actor and critic networks along with the replay buffer (`rpm`). If `--resume` is specified, the agent loads previous weights from the output directory.

### Interaction and Experience Collection

The main training loop follows the standard reinforcement learning interaction pattern defined in [`baseline_modelfree/train.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/train.py):

```python
while step <= train_times:
    step += 1
    if observation is None:
        observation = env.reset()
        agent.reset(observation, noise_factor)
    
    action = agent.select_action(observation, noise_factor=noise_factor)
    observation, reward, done, _ = env.step(action)
    agent.observe(reward, observation, done, step)

```

In [`baseline_modelfree/DRL/ddpg.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/DRL/ddpg.py), the `select_action` method runs a forward pass through the actor network (`self.play`) and adds parameter-space noise for exploration when `noise_factor > 0`. The `observe` method appends transitions to the replay memory (`rpm.append`), storing states, actions, rewards, and termination flags for later learning.

### Policy and Value Updates

After each episode completes (when reaching `max_step` or `done=True`), the script performs learning updates if past the warmup phase:

```python
if step > args.warmup:
    lr = (3e-4, 1e-3)  # Actor and critic learning rates

    for i in range(episode_train_times):
        Q, value_loss = agent.update_policy(lr)

```

The `update_policy` method in [`baseline_modelfree/DRL/ddpg.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/DRL/ddpg.py) implements the core DDPG algorithm through the following steps:

1. **Sample** a minibatch from `self.memory.sample_batch` containing stored transitions.
2. **Compute target actions** using the target actor network, then evaluate target Q-values using the target critic (`self.evaluate(next_state, next_action, True)`).
3. **Calculate TD target**: `target_q = discount * (1 - terminal) * target_q + step_reward`.
4. **Update critic** by minimizing MSE between current Q-values and the TD target (`value_loss.backward()`).
5. **Update actor** by maximizing the critic's Q-value estimate via `policy_loss = -pre_q.mean()`.
6. **Soft-update** target networks using `soft_update(self.actor_target, self.actor, self.tau)` with default `tau=0.001`.

Additionally, each update calls `self.update_gan(next_state)` to train the auxiliary discriminator defined in [`baseline_modelfree/DRL/wgan.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/DRL/wgan.py), providing supplemental rewards beyond the pixel-wise L₂ loss.

### Validation and Checkpointing

Every `validate_interval` episodes, the script runs evaluation without exploration noise:

```python
reward, dist = evaluate(env, agent.select_action, debug=debug)
agent.save_model(output)

```

The `Evaluator` computes mean reward and distance metrics, while `DDPG.save_model` persists the actor, critic, and GAN weights to the output directory for later resumption or inference.

## Network Architectures

The actor and critic use ResNet backbones defined in [`baseline_modelfree/DRL/actor.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/DRL/actor.py) and [`baseline_modelfree/DRL/critic.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/DRL/critic.py):

- **Actor**: `ResNet(9, 18, 65)` — Takes 9-channel input (canvas + target + previous action) and outputs 65-dimensional stroke parameters representing brush location, size, rotation, and color.
- **Critic**: `ResNet_wobn(9, 18, 1)` — Uses weight normalization (instead of batch normalization) to estimate Q-values for state-action pairs, producing a scalar value prediction.

The stroke parameters generated by the actor are rendered into actual brush strokes via the decoder network in [`baseline_modelfree/Renderer/model.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/Renderer/model.py).

## Practical Training Examples

### Starting Training from Command Line

Launch the training process with specified hyperparameters:

```bash
python baseline_modelfree/train.py \
    --batch_size 96 \
    --env_batch 96 \
    --max_step 40 \
    --train_times 2000000 \
    --warmup 400 \
    --tau 0.001 \
    --discount 0.95 \
    --noise_factor 0.05 \
    --validate_interval 50 \
    --output ./model

```

### Loading a Trained Agent for Inference

Use the saved checkpoint to generate paintings without exploration noise:

```python
from baseline_modelfree.DRL.ddpg import DDPG
from baseline_modelfree.DRL.multi import fastenv

agent = DDPG(batch_size=1, env_batch=1, max_step=40,
             tau=0.001, discount=0.95, rmsize=800,
             writer=None, resume='./model', output='./model')
env = fastenv(max_step=40, env_batch=1, writer=None)

obs = env.reset()
for t in range(40):
    action = agent.select_action(obs)  # No noise_factor during inference

    obs, reward, done, _ = env.step(action)
    if done:
        break

# Final canvas available in env.env.canvas[0]

```

### Inspecting Network Architectures

Verify the loaded model structures:

```python
print(agent.actor)   # ResNet(9, depth=18, num_outputs=65)

print(agent.critic)  # ResNet_wobn(9, depth=18, num_outputs=1)

```

## Summary

- The paint agent is trained using **DDPG** (Deep Deterministic Policy Gradient) with an actor-critic architecture implemented in [`baseline_modelfree/DRL/ddpg.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/DRL/ddpg.py).
- The **actor** (`ResNet(9,18,65)`) predicts 65-dimensional brush strokes while the **critic** (`ResNet_wobn(9,18,1)`) evaluates Q-values for state-action pairs.
- A **GAN-based auxiliary reward** from [`baseline_modelfree/DRL/wgan.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/DRL/wgan.py) supplements the pixel-wise L₂ loss from the `Paint` environment.
- **Soft updates** with parameter `tau` (default 0.001) gradually synchronize target networks with online networks.
- The replay buffer stores transitions for off-policy learning, with a `warmup` phase (default 400 steps) before learning updates begin.
- Training is orchestrated by [`baseline_modelfree/train.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/train.py), which handles the interaction loop, periodic validation, and model checkpointing every `validate_interval` episodes.

## Frequently Asked Questions

### What reinforcement learning algorithm is used to train the paint agent?

The paint agent is trained using the **Deep Deterministic Policy Gradient (DDPG)** algorithm, as implemented in [`baseline_modelfree/DRL/ddpg.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/DRL/ddpg.py). DDPG is an off-policy actor-critic method suitable for continuous action spaces, where the actor network predicts brush-stroke parameters and the critic network estimates the expected return for those actions.

### How does the environment calculate rewards during training?

The `Paint` class in [`baseline_modelfree/env.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/env.py) computes rewards based on the **pixel-wise L₂ error** between the current canvas state and the target image. Additionally, the training pipeline incorporates a **GAN-based auxiliary reward** from [`baseline_modelfree/DRL/wgan.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/DRL/wgan.py) via the `update_gan()` method, providing learned perceptual feedback to improve stroke quality beyond pixel-level similarity.

### What are the critical hyperparameters for training convergence?

Key hyperparameters include `tau` (0.001 for soft target network updates), `discount` (0.95 for future reward discounting), `noise_factor` (0.05 for Ornstein-Uhlenbeck exploration noise), and `warmup` (400 initial steps with random actions). The actor and critic use learning rates of 3e-4 and 1e-3 respectively during early training, as specified in the `update_policy` call within [`baseline_modelfree/train.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/train.py).

### How do I resume training from a previous checkpoint?

Pass the `--resume` flag when executing [`baseline_modelfree/train.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/train.py), or set `resume=True` when instantiating the `DDPG` class in Python. The constructor automatically loads actor, critic, and GAN weights from the specified output directory, allowing training to continue from the saved state without losing learned parameters.