How the Neural Renderer Is Trained in the Learning-to-Paint Framework

The neural renderer is trained via supervised learning on pairs of random 10-dimensional stroke parameters and their procedurally generated 128×128 ground-truth images, minimizing pixel-wise mean squared error using the Adam optimizer with a manually decayed learning rate schedule.

The hzwer/iccv2019-learningtopaint repository implements a stroke-based painting agent that relies on a differentiable neural renderer to convert parametric brush strokes into pixel images. Unlike traditional rasterizers, this neural renderer is a fully-convolutional network (FCN) learned end-to-end from synthetic data, enabling gradient flow through the rendering process for reinforcement learning agents. Understanding how this component is trained reveals the foundation of the entire learning-to-paint pipeline.

Neural Renderer Architecture

The renderer architecture is defined in [baseline/Renderer/model.py](https://github.com/hzwer/iccv2019-learningtopaint/blob/master/baseline/Renderer/model.py) as the FCN class. This network learns to map a low-dimensional parameter vector to a high-resolution stroke image through a series of dense expansions and transposed convolutions.

Input and Feature Expansion

The network accepts a 10-dimensional vector f sampled from a uniform distribution, encoding brush position, control points, radius, and color parameters. Four fully-connected layers expand this vector to 4096 features, which are reshaped into a 16×16×16 spatial tensor.

Upsampling Cascade

A sequence of 2-D convolutional layers with PixelShuffle upsampling progressively doubles the spatial resolution through three stages: 16→32, 32→64, and 64→128 pixels. This generates the final 128×128 output.

Output Activation

A sigmoid activation produces pixel values in [0, 1], followed by a flipping operation (1 - x) to match the color space of the ground-truth stroke images generated by the deterministic renderer.

Ground-Truth Generation

Training targets are created on-the-fly by the procedural function draw in [baseline/Renderer/stroke_gen.py](https://github.com/hzwer/iccv2019-learningtopaint/blob/master/baseline/Renderer/stroke_gen.py). This deterministic function takes the same 10-dimensional parameter vector f used as network input and returns a rasterized 128×128 grayscale image representing the brush stroke.

By generating targets procedurally rather than using static datasets, the training loop exposes the neural renderer to an unlimited diversity of stroke shapes, positions, and thicknesses, preventing overfitting to a limited corpus.

Training Loop and Optimization

The complete training procedure is orchestrated in [baseline/train_renderer.py](https://github.com/hzwer/iccv2019-learningtopaint/blob/master/baseline/train_renderer.py). The script implements a standard supervised regression pipeline with specific optimizations for the stroke generation task.

Data Sampling and Target Creation

For each training step, the script generates a fresh batch of 64 random vectors using np.random.uniform(0, 1, (64, 10)). Corresponding ground-truth images are computed immediately via the draw function:

f = np.random.uniform(0, 1, 10)
ground_truth = draw(f)

This ensures the network never sees identical training examples, as each batch contains unique synthetic strokes.

Loss Function and Optimizer

The training minimizes mean squared error (MSE) between the neural renderer's output and the procedural ground truth using nn.MSELoss. Optimization is performed with Adam (optim.Adam), configured with an initial learning rate that follows a manual decay schedule.

Learning Rate Schedule

The learning rate is adjusted based on training steps:

• 1e-4 for steps < 200,000
• 1e-5 for steps 200,000–400,000
• 1e-6 for steps > 400,000

This progressive decay stabilizes training as the network approaches convergence on the fine details of stroke textures.

Logging and Checkpointing

The training script logs metrics to TensorBoard every 100 steps, including scalar loss values and visual samples of generated strokes versus ground truth. Model checkpoints are saved to renderer.pkl every 1,000 steps, ensuring recoverability from interruptions.

Step-by-Step Training Procedure

The following sequence from baseline/train_renderer.py illustrates the core training iteration:

1. Sample parameters: Generate random vector f ∈ ℝ¹⁰ uniformly distributed [0, 1]
2. Create targets: Compute draw(f) to obtain 128×128 ground-truth images
3. Tensor conversion: Move batches to GPU if available
4. Forward pass: Generate predictions via net(train_batch)
5. Compute loss: Calculate MSE between generated and target images
6. Backpropagation: Execute loss.backward() and optimizer step
7. Adjust learning rate: Manually decay based on step count thresholds
8. Log metrics: Write loss and sample images to TensorBoard
9. Save checkpoint: Persist model state every 1,000 iterations

Through millions of iterations over unique random strokes, the FCN learns a differentiable mapping from parameter space to image space, effectively becoming a neural surrogate for the procedural draw function.

Practical Usage Examples

To train the neural renderer from the repository root:

# Run the training script

python baseline/train_renderer.py

# Monitor training progress

tensorboard --logdir ../train_log/

To use a trained renderer for inference:

import torch
import numpy as np
from baseline.Renderer.model import FCN
from baseline.Renderer.stroke_gen import draw

# Load trained weights

net = FCN()
net.load_state_dict(torch.load('renderer.pkl', map_location='cpu'))
net.eval()

# Generate random stroke parameters

f = np.random.uniform(0, 1, 10).astype('float32')
f_tensor = torch.from_numpy(f).unsqueeze(0)

# Neural rendering

with torch.no_grad():
    prediction = net(f_tensor).squeeze(0).numpy()

# Compare with procedural ground truth

ground_truth = draw(f)

Summary

• Architecture: The neural renderer is a fully-convolutional network with dense feature expansion and PixelShuffle upsampling, defined in baseline/Renderer/model.py.
• Supervision: Training uses procedurally generated targets from baseline/Renderer/stroke_gen.py, creating infinite diverse stroke examples on-the-fly.
• Optimization: The Adam optimizer minimizes MSE loss with a manually scheduled learning rate decay (1e-4 → 1e-5 → 1e-6) over 400k+ steps.
• Differentiability: The trained model provides a differentiable alternative to the deterministic draw function, enabling gradient-based optimization of stroke parameters in downstream reinforcement learning agents.

Frequently Asked Questions

What loss function is used to train the neural renderer?

The training process uses mean squared error (MSE) loss implemented via nn.MSELoss in PyTorch. This pixel-wise loss measures the difference between the 128×128 output of the neural network and the corresponding ground-truth image generated by the procedural draw function.

How is the ground-truth data created for training?

Ground-truth images are generated procedurally on-the-fly by the draw function in baseline/Renderer/stroke_gen.py. For each training batch, random 10-dimensional parameter vectors are sampled, and the deterministic renderer produces the target stroke images immediately before the forward pass. This eliminates the need for a static dataset and ensures unlimited training diversity.

What is the learning rate schedule for training the renderer?

The learning rate follows a manual step-decay schedule implemented in the training loop: 1e-4 for the first 200,000 steps, 1e-5 for steps 200,000 to 400,000, and 1e-6 for all subsequent steps. This schedule is hardcoded in baseline/train_renderer.py and applied after each optimizer step based on the current global step count.

Why is the neural renderer necessary instead of using the procedural draw function directly?

The procedural draw function is non-differentiable, preventing gradient flow through the rendering process. The neural renderer approximates this function with a fully-differentiable neural network, enabling reinforcement learning agents to optimize stroke parameters via backpropagation through the rendering layer during the painting episode training.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}