Neural Renderer Network Definition in hzwer/iccv2019-learningtopaint: FCN Architecture and File Location

The neural renderer network definition is implemented as the FCN class in baseline/Renderer/model.py (and identically in baseline_modelfree/Renderer/model.py), defining a fully-convolutional network that maps a 10-dimensional stroke vector to a 128×128 RGB image.

The hzwer/iccv2019-learningtopaint repository contains the official PyTorch implementation of the ICCV 2019 paper "Learning to Paint". At the core of this stroke-based rendering system lies the neural renderer network definition, which serves as the differentiable brush engine that converts low-dimensional stroke parameters into pixel-level outputs. This article identifies the exact file locations, examines the FCN class architecture, and provides practical code examples for integrating the renderer into training or inference pipelines.

Neural Renderer Architecture Overview

The neural renderer is implemented as the FCN class, a fully-convolutional network that progressively expands a low-dimensional latent vector into a full-resolution image. According to the source code analysis, the network accepts a 10-dimensional input representing brush stroke parameters and outputs a 128×128 image with pixel values in the range [0, 1].

The architecture consists of two distinct stages:

1. Fully-connected tower: Expands the 10-D vector through dense layers to a high-dimensional feature representation
2. Convolutional upsampling pipeline: Uses strided convolutions combined with nn.PixelShuffle to progressively increase spatial resolution from 16×16 to 128×128

FCN (nn.Module)
└─ Fully‑connected tower
   ├─ Linear(10 → 512)
   ├─ Linear(512 → 1024)
   ├─ Linear(1024 → 2048)
   └─ Linear(2048 → 4096)
   → reshaped to (batch, 16, 16, 16)

└─ Convolution‑up‑sampling pipeline
   ├─ Conv2d(16 → 32)  + ReLU
   ├─ Conv2d(32 → 32)  → PixelShuffle(2)   (↑×2)
   ├─ Conv2d(8  → 16)  + ReLU
   ├─ Conv2d(16 → 16)  → PixelShuffle(2)   (↑×2)
   ├─ Conv2d(4  → 8)   + ReLU
   ├─ Conv2d(8  → 4)   → PixelShuffle(2)   (↑×2)
   └─ Sigmoid → 1‑x → reshape to (batch, 128, 128)

File Location of the Neural Renderer Network Definition

The FCN class definition is duplicated across two baseline variants in the repository. Both implementations are identical and are imported throughout the codebase in files such as predict.py, train_renderer.py, test.py, and various DRL training scripts.

Baseline Implementation Path

The primary neural renderer network definition resides in:

• baseline/Renderer/model.py

This file contains the complete FCN class definition used by the standard supervised learning pipeline.

Model-Free Variant Path

An identical copy exists for the model-free reinforcement learning variant:

• baseline_modelfree/Renderer/model.py

Both files define the same network architecture and can be imported interchangeably depending on which training paradigm you are using.

FCN Class Implementation Details

The FCN class inherits from nn.Module and implements a custom forward pass that handles the dimensional transformations between the fully-connected and convolutional stages.

Fully-Connected Tower

The first stage processes the input stroke vector through a deep MLP:

• Input: Tensor of shape (batch_size, 10)
• Hidden layers: 512, 1024, 2048 units with ReLU activations
• Output: 4096 units, reshaped to (batch, 16, 16, 16) to serve as the initial feature map

This expansion creates a rich, high-dimensional representation that contains the structural information needed to reconstruct the brush stroke.

Convolutional Upsampling Pipeline

The second stage uses PixelShuffle (sub-pixel convolution) to upsample the feature map through three successive doubling operations (16→32→64→128):

1. Conv2d(16→32) → ReLU → Conv2d(32→32) → PixelShuffle(2)
2. Conv2d(8→16) → ReLU → Conv2d(16→16) → PixelShuffle(2)
3. Conv2d(4→8) → ReLU → Conv2d(8→4) → PixelShuffle(2)

Finally, a Sigmoid activation followed by the operation 1 - x produces the final output image with values inverted to the [0, 1] range suitable for alpha compositing onto a canvas.

Using the Neural Renderer in Code

The following examples demonstrate how to instantiate, train, and persist the neural renderer based on the actual usage patterns found in the repository.

Instantiating and Forward Pass

Import the FCN class from the appropriate baseline directory and run inference:

import torch
from baseline.Renderer.model import FCN   # or baseline_modelfree.Renderer.model

# Create the renderer

renderer = FCN()

# Dummy latent vector (batch size = 1, dim = 10)

z = torch.randn(1, 10)

# Generate an image

img = renderer(z)          # shape: (1, 128, 128)

print(img.shape)           # torch.Size([1, 128, 128])

Training Loop Integration

When training the renderer (as done in baseline/train_renderer.py), use standard PyTorch optimization:

optimizer = torch.optim.Adam(renderer.parameters(), lr=1e-4)

for step in range(num_steps):
    # Sample latent vectors and ground‑truth images

    z, target = get_batch()               # z: (B,10), target: (B,128,128)

    pred = renderer(z)                     # forward

    loss = torch.nn.functional.mse_loss(pred, target)

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

Saving and Loading Checkpoints

Persist trained weights using PyTorch's state dict mechanism:

# Save

torch.save(renderer.state_dict(), "renderer_fcn.pth")

# Load

renderer = FCN()
renderer.load_state_dict(torch.load("renderer_fcn.pth"))
renderer.eval()

Related Files and Integration Points

The neural renderer network definition is utilized throughout the codebase by several key modules:

File	Purpose
baseline/Renderer/stroke_gen.py	Generates the 10-dimensional stroke representation fed into FCN
predict.py	Entry point for inference that imports FCN and renders user-provided stroke sequences
baseline/train_renderer.py	Orchestrates data loading and optimization for the renderer
baseline_modelfree/train_renderer.py	Training pipeline for the model-free reinforcement learning variant
test.py	Evaluation scripts that use the trained FCN for metric calculation

These files together constitute the complete workflow for stroke generation, neural rendering, and canvas compositing.

Summary

• The neural renderer network definition is located in baseline/Renderer/model.py and baseline_modelfree/Renderer/model.py as the FCN class.
• The network accepts a 10-dimensional stroke vector and outputs a 128×128 RGB image through a combination of fully-connected expansion and PixelShuffle upsampling.
• The architecture uses a 4096-unit bottleneck reshaped to 16×16×16, followed by three 2× upsampling stages to reach full resolution.
• The class is imported across training, testing, and inference scripts including predict.py and train_renderer.py.
• Both baseline variants share identical network definitions, ensuring consistent stroke rendering across supervised and reinforcement learning pipelines.

Frequently Asked Questions

What is the input dimension of the neural renderer?

The FCN class accepts a 10-dimensional tensor representing brush stroke parameters. These parameters typically encode stroke location, rotation, size, and color information generated by stroke_gen.py or predicted by the agent network.

Where is the neural renderer trained in the codebase?

Training occurs in baseline/train_renderer.py for the supervised baseline and baseline_modelfree/train_renderer.py for the model-free variant. Both scripts import the FCN class from their respective Renderer/model.py files and optimize the network using MSE loss between predicted and ground-truth brush strokes.

What is the difference between the baseline and model-free renderer?

There is no architectural difference between the two renderers. Both the baseline (baseline/Renderer/model.py) and model-free (baseline_modelfree/Renderer/model.py) implementations contain identical FCN class definitions. The distinction lies in how the 10-dimensional stroke vectors are generated—either from a learned model or from the reinforcement learning agent—while the neural renderer network definition remains constant.

How does the FCN class generate high-resolution images from low-dimensional inputs?

The FCN class uses a two-stage pipeline: first, a fully-connected tower expands the 10-D vector to 4096 features, which are reshaped into a 16×16 spatial grid with 16 channels. Then, a series of convolutional layers with PixelShuffle operations progressively doubles the spatial resolution three times (16→32→64→128) while reducing channels, ultimately producing the final 128×128 output through a sigmoid activation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}