Learning to Paint ICCV 2019 Repository: Neural Agent for Stroke-Based Image Synthesis

The Learning to Paint ICCV 2019 repository implements a model-based deep reinforcement learning system that trains a neural agent to reproduce images using sequential brush strokes, combining a differentiable neural renderer with a DDPG agent to mimic human painting techniques.

The hzwer/iccv2019-learningtopaint repository provides the official implementation of the ICCV 2019 paper “Learning to paint with model-based deep reinforcement learning” by Huang et al. This codebase demonstrates how a neural agent can learn to synthesize photorealistic images through hundreds of iterative stroke decisions, treating painting as a sequential decision-making problem rather than pixel-wise regression.

Core Architecture Components

The Learning to Paint repository consists of three tightly coupled subsystems that work together to enable differentiable stroke-based rendering.

Neural Renderer

The neural renderer provides a fully-convolutional network (FCN) that serves as a differentiable painting environment. Located in baseline/Renderer/model.py, this FCN maps a 10-dimensional stroke parameter vector to a rasterized 128×128 stroke image. This differentiability allows gradients to flow from the final canvas back to stroke parameters during reinforcement learning. The renderer is initially trained using supervised learning on synthetic data generated by the analytical draw() function in baseline/Renderer/stroke_gen.py.

Stroke Generation Pre-training

Before the agent trains, the neural renderer undergoes supervised pre-training to learn accurate stroke appearance. The script baseline/train_renderer.py trains the FCN using MSE loss between predicted strokes and ground-truth renderings from the procedural draw() function. Each stroke is parameterized as f = (x0, y0, x1, y1, x2, y2, z0, z2, w0, w2), representing position, control points, radius, and opacity. The trained weights are saved as renderer.pkl for use during agent training.

Model-Based DRL Agent

The painting agent implements Deep Deterministic Policy Gradient (DDPG) with separate ResNet-based networks. The actor (baseline/DRL/actor.py) receives the current canvas, target image, step counter, and coordinate channels, outputting a 65-dimensional action representing the next 5 strokes (10 parameters each) plus 3 color channels. The critic (baseline/DRL/critic.py) evaluates state-action pairs using a combination of L2 improvement and GAN-based perceptual rewards calculated in baseline/DDPG/ddpg.py.

How the Painting System Works

The complete pipeline follows a structured iterative process that transforms a blank canvas into a painted image through hundreds of discrete steps.

1. State Representation – The agent observes a state tensor concatenating the current canvas, target image, normalized step counter, and coordinate convolution channels.

2. Action Selection – The ResNet actor predicts stroke parameters that maximize expected canvas improvement.

3. Stroke Rendering – The decode() function in baseline/DDPG/ddpg.py feeds the 10-dimensional stroke vectors through the neural renderer to produce differentiable stroke images.

4. Reward Computation – The environment calculates rewards using both pixel-wise L2 distance and GAN-based perceptual loss through cal_reward.

5. Policy Update – The DDPG algorithm stores transitions in a replay buffer and updates actor-critic weights via gradient descent in baseline/train.py.

Key Implementation Files

• baseline/Renderer/model.py – FCN architecture for the differentiable neural renderer.

• baseline/Renderer/stroke_gen.py – Procedural stroke generator using the analytical draw() function.

• baseline/train_renderer.py – Supervised training script for the renderer network.

• baseline/DRL/actor.py – ResNet policy network that predicts stroke parameters.

• baseline/DRL/critic.py – ResNet value network for Q-value estimation.

• baseline/DDPG/ddpg.py – Core DDPG implementation integrating actor, critic, renderer, and reward computation.

• baseline/train.py – Main training loop orchestrating the painting agent's interaction with the environment.

• baseline/utils/util.py – Helper functions for image loading, saving, and coordinate channel generation.

Practical Usage Examples

Pre-training the Neural Renderer

Run the supervised training script to teach the FCN to render parametric strokes accurately:

# From baseline/train_renderer.py

from Renderer.model import FCN
from Renderer.stroke_gen import draw
import torch
import torch.nn as nn
import torch.optim as optim

net = FCN()
optimizer = optim.Adam(net.parameters(), lr=3e-6)
criterion = nn.MSELoss()

for step in range(500000):
    batch_f = torch.rand(64, 10)  # Random stroke parameters

    batch_img = torch.tensor([draw(f.numpy()) for f in batch_f])
    pred = net(batch_f)
    loss = criterion(pred, batch_img)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    if step % 1000 == 0:
        torch.save(net.state_dict(), "../renderer.pkl")

Training the Painting Agent

Execute the DDPG training loop with specified hyperparameters:

python3 baseline/train.py \
    --max_step 40 \
    --train_times 2000000 \
    --env_batch 96 \
    --batch_size 96 \
    --output ./model

This initializes the actor-critic networks and begins the model-based RL training process, periodically saving checkpoints to ./model.

Generating Paintings with the Trained Agent

Use the trained DDPG agent to paint a target image through iterative stroke application:

import torch
from DRL.ddpg import DDPG
from utils.util import load_image, save_image

# Initialize agent with trained weights

agent = DDPG(batch_size=1, env_batch=1, max_step=40,
             resume='./model', output_path='./model')

target = load_image('image/test.png')  # 128x128 RGB input

canvas = torch.zeros_like(target)      # Blank canvas

coord = torch.zeros((1, 2, 128, 128))  # Coordinate channels

for step in range(40):
    # Construct state tensor

    step_tensor = torch.full((1, 1, 128, 128), step / 40)
    state = torch.cat([canvas, target, step_tensor, coord], dim=1)
    
    action = agent.select_action(state)  # 65-dim vector: 5 strokes + colors

    canvas = agent.decode(action, canvas)  # Apply strokes via neural renderer

save_image(canvas, 'result.png')

Summary

• The Learning to Paint ICCV 2019 repository implements a complete model-based RL pipeline for stroke-based image synthesis.
• The system uses a differentiable neural renderer (baseline/Renderer/model.py) to enable gradient flow through the painting environment.
• A DDPG agent with ResNet actor-critic architectures learns optimal stroke sequences through millions of training steps.
• The 10-dimensional stroke parameterization controls position, curvature, width, and opacity for each brush mark.
• The codebase includes complete training scripts, utilities for TensorBoard logging, and a Colab notebook for browser-based execution.

Frequently Asked Questions

What is the purpose of the Learning to Paint ICCV 2019 repository?

The repository implements the research system described in the ICCV 2019 paper "Learning to paint with model-based deep reinforcement learning." Its purpose is to demonstrate how a neural agent can learn to reproduce images using a sequence of brush strokes rather than direct pixel generation, treating painting as a sequential decision-making task solved through deep reinforcement learning.

How does the neural renderer work in the Learning to Paint system?

The neural renderer is a fully-convolutional network defined in baseline/Renderer/model.py that converts 10-dimensional stroke parameters into rasterized 128×128 stroke images. It provides a differentiable approximation of the painting process, allowing gradients to propagate from the final canvas back to stroke decisions during DDPG training. The renderer is first pre-trained on synthetic data generated by the analytical draw() function in baseline/Renderer/stroke_gen.py using MSE loss.

What reinforcement learning algorithm does the Learning to Paint repository use?

The repository implements Deep Deterministic Policy Gradient (DDPG), a continuous-control algorithm suitable for the high-dimensional action space of stroke parameters. The implementation in baseline/DDPG/ddpg.py features a ResNet-based actor that predicts stroke parameters and a ResNet-based critic that evaluates state-action pairs using combined L2 and GAN-based perceptual rewards.

What are the main files in the Learning to Paint ICCV 2019 codebase?

The essential files include baseline/Renderer/model.py (neural renderer FCN), baseline/DRL/actor.py and critic.py (policy and value networks), baseline/DDPG/ddpg.py (core RL algorithm), and baseline/train.py (main training orchestration). Utility functions reside in baseline/utils/, while baseline/train_renderer.py handles the supervised pre-training phase required before agent training begins.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}