Neural Renderer FCN Architecture in Learning to Paint: Technical Deep Dive

The neural renderer FCN architecture is a fully-convolutional network that maps a 10‑dimensional latent vector to a 128×128 pixel brush stroke using fully‑connected expansion followed by pixel‑shuffle upsampling, implemented in the baseline/Renderer/model.py file of the hzwer/iccv2019-learningtopaint repository.

The hzwer/iccv2019-learningtopaint repository provides a PyTorch implementation of a stroke‑based painting agent introduced at ICCV 2019. Central to this system is the neural renderer FCN architecture, which generates realistic brush textures from compact style vectors without relying on transposed convolutions or interpolation.

Neural Renderer FCN Architecture Overview

The FCN class defined in baseline/Renderer/model.py inherits from torch.nn.Module and implements a two‑stage pipeline: latent expansion via fully‑connected layers and progressive upsampling via convolutional blocks with pixel‑shuffle operations. This design transforms a low‑dimensional control signal into a high‑resolution spatial image while preserving fine‑grained details critical for stroke rendering.

Input Specifications

The network accepts a 10‑dimensional latent vector representing style or brush parameters. This input is typically sampled from a distribution or predicted by an upstream policy network in the full painting pipeline.

Output Specifications

The renderer produces a single‑channel 128×128 grayscale image with pixel values normalized to the range [0, 1]. The final output undergoes a sigmoid activation followed by an inversion (1 - x) to match the dataset’s brush‑stroke representation.

Layer‑by‑Layer Architecture Breakdown

According to the source code in baseline/Renderer/model.py (lines 6‑34), the FCN forward pass executes the following transformations:

1. Latent Expansion (Fully‑Connected Stack)

   • nn.Linear(10, 512) – Projects the input into a 512‑dimensional space.
   • nn.Linear(512, 1024) – Expands to 1024 dimensions for richer expressiveness.
   • nn.Linear(1024, 2048) – Further deepens the representation.
   • nn.Linear(2048, 4096) – Generates a flat vector sufficient to populate a 4‑D tensor.

2. Spatial Reshaping

   • view(-1, 16, 16, 16) – Reshapes the 4096‑element vector into a tensor of shape (B, 16, 16, 16), creating a 16‑channel feature map with 16×16 spatial resolution.

3. Progressive Upsampling (Conv + Pixel‑Shuffle)

   • First Upsampling Block:
     • nn.Conv2d(16, 32, 3, stride=1, padding=1) – Increases channel depth to 32 while maintaining 16×16 spatial dimensions.
     • nn.PixelShuffle(2) – Rearranges elements to upscale spatial resolution to 32×32, reducing channels from 32 to 8.
   • Second Upsampling Block:
     • nn.Conv2d(8, 16, 3, stride=1, padding=1) – Refines the 8‑channel 32×32 feature map.
     • nn.PixelShuffle(2) – Upsamples to 64×64, reducing channels from 16 to 4.
   • Third Upsampling Block:
     • nn.Conv2d(4, 8, 3, stride=1, padding=1) – Processes the 4‑channel 64×64 representation.
     • nn.PixelShuffle(2) – Final upscaling to target resolution 128×128, reducing channels from 8 to 2.

4. Final Activation

   • torch.sigmoid – Squashes values to [0, 1].
   • 1 - x – Inverts the output to align with the expected brush stroke format, yielding a single‑channel image after dimensionality reduction.

Implementation Details in PyTorch

The architecture explicitly avoids transposed convolutions, which can introduce checkerboard artifacts, in favor of pixel‑shuffle (nn.PixelShuffle) for learned upsampling. Each convolutional layer uses a 3×3 kernel, stride 1, and padding 1, ensuring spatial dimensions remain constant before the upsampling operation doubles the resolution.

Key Architectural Features

• Parameter Efficiency: The network uses only four fully‑connected layers before transitioning to convolutions, minimizing the parameter count for the latent mapping while delegating spatial generation to the convolutional stack.
• Multi‑Stage Upsampling: Three successive PixelShuffle(2) operations provide an 8× total spatial upsampling factor (2³), transforming the 16×16 intermediate tensor to the final 128×128 output.
• Deterministic Output: The sigmoid activation guarantees stable gradient behavior during training and ensures generated strokes remain within valid pixel ranges.

Practical Code Examples

Instantiating the Renderer and Generating Strokes

The following snippet demonstrates how to load the FCN class and produce brush strokes from random latent vectors:

import torch
from baseline.Renderer.model import FCN

# Initialize the neural renderer

renderer = FCN()

# Dummy latent vectors: batch size 4, 10‑dimensional style codes

z = torch.randn(4, 10)

# Forward pass generates strokes of shape (B, 1, 128, 128)

stroke = renderer(z)

print(stroke.shape)           # torch.Size([4, 1, 128, 128])

print(stroke.min(), stroke.max())  # Values constrained to [0, 1]

Training the Neural Renderer

This example shows a standard training loop using the Adam optimizer and MSE loss, as implemented in the repository’s training scripts:

import torch
import torch.optim as optim
from baseline.Renderer.model import FCN

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = FCN().to(device)
optimizer = optim.Adam(model.parameters(), lr=1e-4)
criterion = torch.nn.MSELoss()

# Hypothetical data loader providing (style_vector, target_image) pairs

for epoch in range(num_epochs):
    for style_vec, target_img in train_loader:
        style_vec = style_vec.to(device)      # Shape: (B, 10)

        target_img = target_img.to(device)      # Shape: (B, 1, 128, 128)

        optimizer.zero_grad()
        output = model(style_vec)               # Shape: (B, 1, 128, 128)

        loss = criterion(output, target_img)
        loss.backward()
        optimizer.step()

Visualizing Generated Strokes

To inspect the output of the neural renderer FCN architecture during inference or debugging:

import matplotlib.pyplot as plt
import numpy as np

renderer = FCN()
z = torch.randn(1, 10)
stroke = renderer(z).detach().cpu().numpy().squeeze()  # Shape: (128, 128)

plt.imshow(stroke, cmap='gray')
plt.title('Generated Brush Stroke')
plt.axis('off')
plt.show()

Repository File Structure

The neural renderer FCN architecture integrates with the broader Learning to Paint system through the following key files:

• baseline/Renderer/model.py – Contains the FCN class definition and forward pass implementation (lines 6‑34).
• baseline/Renderer/stroke_gen.py – Utilities for converting FCN outputs into stroke sequences compatible with the painting environment.
• baseline/train_renderer.py – Training script that optimizes the FCN parameters using stroke‑image pairs.
• baseline/utils/util.py – Helper functions for data loading, preprocessing, and tensor transformations.
• predict.py – Inference entry point demonstrating how to load a trained FCN checkpoint and generate strokes.

Summary

• The neural renderer FCN architecture accepts a 10‑dimensional latent vector and outputs a 128×128 single‑channel image.
• The network expands the latent code through four fully‑connected layers (10 → 4096 parameters) before reshaping into a spatial tensor.
• Three pixel‑shuffle upsampling stages progressively increase resolution from 16×16 to 128×128 while reducing channel depth, avoiding transposed convolution artifacts.
• All convolutional operations use 3×3 kernels with stride 1 and padding 1, maintaining spatial fidelity throughout the upsampling pipeline.
• The implementation resides in baseline/Renderer/model.py within the hzwer/iccv2019-learningtopaint repository.

Frequently Asked Questions

What is the input dimension of the FCN neural renderer?

The FCN class expects a 10‑dimensional input vector representing the style or parameters of a brush stroke. This vector is processed by the fully‑connected layers to generate the initial 4096‑element feature representation that is reshaped into a 16×16 spatial grid with 16 channels.

Why does the architecture use pixel‑shuffle instead of transposed convolutions?

The neural renderer FCN architecture employs nn.PixelShuffle(2) for upsampling because it reduces checkerboard artifacts commonly associated with transposed convolutions. Pixel‑shuffle rearranges channel elements into spatial dimensions, providing a learned upsampling mechanism that preserves high‑frequency details critical for realistic brush texture generation while maintaining stable gradients during backpropagation.

What is the final image size produced by the neural renderer?

The network outputs a single‑channel 128×128 pixel image. This resolution is achieved through three successive doubling operations (16→32→64→128) performed by the pixel‑shuffle layers, matching the stroke size used in the ICCV 2019 Learning to Paint paper’s experimental setup.

How is the FCN class implemented in the Learning to Paint repository?

In baseline/Renderer/model.py, the FCN class inherits from torch.nn.Module and defines the forward pass as a sequential pipeline: four linear layers, a view operation to (B, 16, 16, 16), three convolutional blocks each followed by PixelShuffle(2), and a final sigmoid activation with inversion. This compact implementation spans approximately 28 lines of code (lines 6‑34) and uses standard PyTorch operations for full differentiability.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}