How the Neural Renderer in Learning to Paint Works: Architecture and Training Explained

The neural renderer in Learning to Paint is a compact fully-convolutional network (FCN) that learns to rasterize 10-dimensional brush stroke parameters into 128×128 pixel images, trained via supervised learning to mimic an analytical Bézier curve rasterizer.

The Learning to Paint project by hzwer implements a differentiable neural renderer that converts parametric stroke descriptions into pixel outputs. This network enables reinforcement learning agents to predict painting outcomes through fast GPU inference rather than expensive CPU-based simulation. Understanding this component is essential for anyone extending the painting agent or adapting the renderer for other stroke-based graphics applications.

Input Representation and Stroke Parameterization

The renderer accepts a 10-dimensional vector f that fully describes a single brush stroke. In baseline/Renderer/stroke_gen.py, this vector is defined as:

f = (x0, y0, x1, y1, x2, y2, z0, z2, w0, w2)

Each parameter controls specific geometric and optical properties:

• x0, y0, x2, y2 – Normalized start and end coordinates in the range [0, 1]
• x1, y1 – Relative control point offsets defining a quadratic Bézier curve between start and end
• z0, z2 – Brush radius at the start and end points
• w0, w2 – Opacity (intensity) values at the start and end points

This compact representation allows the network to learn continuous stroke interpolation, where radius and opacity vary smoothly along the curve trajectory.

Ground-Truth Rasterization Pipeline

Supervision for the neural renderer comes from an analytical rasterizer implemented in baseline/Renderer/stroke_gen.py. The function draw(f, width=128) generates ground-truth images through the following process:

1. High-resolution rendering – The stroke is drawn on a 256×256 canvas
2. Curve sampling – 100 points are sampled along the quadratic Bézier curve defined by (x0, y0), (x1, y1), (x2, y2)
3. Circle stamping – At each point, circles are painted with radii and intensities interpolated between z0/z2 and w0/w2
4. Downsampling – The canvas is resized to 128×128 using area averaging
5. Inversion – The final image is computed as 1 - canvas so that white represents the painted region

This produces the target tensor used during training.

Neural Renderer Architecture in model.py

The renderer is implemented as class FCN in baseline/Renderer/model.py. The architecture follows an encoder-decoder pattern with progressive upsampling:

Fully-connected expansion – Four linear layers expand the 10-dimensional input to a 4096-dimensional feature vector.

Spatial reshaping – The vector is reshaped to a 16×16×16 feature map using x.view(-1, 16, 16, 16).

Progressive upsampling – A series of 2D convolutional layers with pixel-shuffle upsampling (scale factor 2) increase spatial resolution while reducing channel depth:

• 16×16 → 32×32
• 32×32 → 64×64
• 64×64 → 128×128

Output activation – A final sigmoid activation constrains values to [0, 1], followed by inversion (1 - x) to match the ground-truth convention where white pixels represent paint.

The network contains approximately 1.5M parameters and performs inference in a single forward pass.

Training the Neural Renderer

The training script baseline/train_renderer.py implements an end-to-end supervised learning pipeline:

• Data generation – Random stroke parameters are sampled uniformly for each training step, ensuring unlimited diversity without dataset storage requirements
• Target generation – The draw(f) function produces ground-truth 128×128 targets on-the-fly
• Loss function – Mean squared error (nn.MSELoss) between the predicted and target images
• Optimization – Adam optimizer with manually annealed learning rate over 500,000 training steps

This approach eliminates the need for a pre-rendered dataset while ensuring the network generalizes across the full distribution of possible strokes.

Inference and Usage Example

After training, the renderer can be loaded from renderer.pkl and used for fast stroke rasterization:

import torch
from baseline.Renderer.model import FCN
from baseline.Renderer.stroke_gen import draw
import matplotlib.pyplot as plt

# Load pretrained neural renderer

net = FCN()
net.load_state_dict(torch.load('renderer.pkl'))
net.eval()

# Define stroke parameters (x0, y0, x1, y1, x2, y2, z0, z2, w0, w2)

stroke_params = [0.2, 0.3, 0.5, 0.4, 0.1, 0.6, 0.05, 0.12, 0.8, 0.6]
f = torch.tensor(stroke_params).unsqueeze(0).float()

# Neural rendering (differentiable)

with torch.no_grad():
    neural_img = net(f).squeeze().cpu().numpy()

# Ground-truth rasterization (non-differentiable)

gt_img = draw(f.squeeze().numpy())

# Visualize comparison

plt.figure(figsize=(8, 4))
plt.subplot(1, 2, 1)
plt.title('Neural Renderer Output')
plt.imshow(neural_img, cmap='gray')
plt.subplot(1, 2, 2)
plt.title('Analytical Ground Truth')
plt.imshow(gt_img, cmap='gray')
plt.show()

The neural output closely approximates the analytical rasterizer while remaining fully differentiable, enabling gradient-based optimization through the rendering step.

Summary

• The neural renderer is a fully-convolutional network defined in baseline/Renderer/model.py that maps 10-D stroke parameters to 128×128 images
• Input strokes are parameterized as quadratic Bézier curves with variable radius and opacity
• Ground-truth supervision comes from draw() in baseline/Renderer/stroke_gen.py, which analytically rasterizes strokes at 256×256 before downsampling
• The architecture uses pixel-shuffle upsampling to progressively expand 16×16 features to full resolution
• Training occurs via MSE loss against on-the-fly rendered targets over 500k steps using Adam
• The trained renderer enables differentiable, GPU-accelerated stroke rendering for reinforcement learning agents

Frequently Asked Questions

What is the output resolution of the neural renderer?

The network outputs fixed 128×128 grayscale images regardless of input stroke complexity. The architecture hardcodes this resolution through the specific progression of upsampling layers (16→32→64→128) in model.py. For different output sizes, the network architecture would require modification of the initial reshape dimensions or the number of upsampling stages.

Why does the network invert the output with 1-x?

Both the analytical rasterizer in stroke_gen.py and the neural network apply inversion (1 - canvas) so that painted regions appear white (value 1.0) and the background appears black (value 0.0). This convention aligns with the painting simulation where white pixels represent accumulated paint on a black canvas, making it easier for the reinforcement learning agent to compute state differences.

How does pixel-shuffle upsampling work in this architecture?

Pixel-shuffle (also called sub-pixel convolution) rearranges elements from the channel dimension into spatial dimensions. In baseline/Renderer/model.py, each upsampling stage uses convolution to expand channels by a factor of 4 (for 2× upsampling), then applies pixel_shuffle(2) to reshape those channels into higher-resolution feature maps. This approach reduces checkerboard artifacts compared to transposed convolutions while maintaining computational efficiency.

Can the neural renderer handle different brush sizes?

Yes, the network learns continuous brush radius interpolation through the z0 and z2 input parameters. Because training samples radii uniformly across the valid range, the network generalizes to arbitrary brush sizes within the distribution seen during training. However, extreme values outside the training distribution (e.g., radii approaching zero or exceeding canvas bounds) may produce artifacts since the network learns an approximation rather than a perfect physical simulation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}