# Neural Renderer FCN Architecture in Learning to Paint: Technical Deep Dive

> Explore the Neural Renderer FCN architecture a fully convolutional network mapping latent vectors to brush strokes implemented in hzwericcv2019learningtopaint repository.

- Repository: [hzwer/iccv2019-learningtopaint](https://github.com/hzwer/iccv2019-learningtopaint)
- Tags: technical-deep-dive
- Published: 2026-03-03

---

**The neural renderer FCN architecture is a fully-convolutional network that maps a 10‑dimensional latent vector to a 128×128 pixel brush stroke using fully‑connected expansion followed by pixel‑shuffle upsampling, implemented in the [`baseline/Renderer/model.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline/Renderer/model.py) file of the hzwer/iccv2019-learningtopaint repository.**

The hzwer/iccv2019-learningtopaint repository provides a PyTorch implementation of a stroke‑based painting agent introduced at ICCV 2019. Central to this system is the **neural renderer FCN architecture**, which generates realistic brush textures from compact style vectors without relying on transposed convolutions or interpolation.

## Neural Renderer FCN Architecture Overview

The `FCN` class defined in [`baseline/Renderer/model.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline/Renderer/model.py) inherits from `torch.nn.Module` and implements a two‑stage pipeline: **latent expansion** via fully‑connected layers and **progressive upsampling** via convolutional blocks with pixel‑shuffle operations. This design transforms a low‑dimensional control signal into a high‑resolution spatial image while preserving fine‑grained details critical for stroke rendering.

### Input Specifications

The network accepts a **10‑dimensional** latent vector representing style or brush parameters. This input is typically sampled from a distribution or predicted by an upstream policy network in the full painting pipeline.

### Output Specifications

The renderer produces a **single‑channel 128×128** grayscale image with pixel values normalized to the range **[0, 1]**. The final output undergoes a `sigmoid` activation followed by an inversion (`1 - x`) to match the dataset’s brush‑stroke representation.

## Layer‑by‑Layer Architecture Breakdown

According to the source code in [`baseline/Renderer/model.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline/Renderer/model.py) (lines 6‑34), the `FCN` forward pass executes the following transformations:

1.  **Latent Expansion (Fully‑Connected Stack)**
    - `nn.Linear(10, 512)` – Projects the input into a 512‑dimensional space.
    - `nn.Linear(512, 1024)` – Expands to 1024 dimensions for richer expressiveness.
    - `nn.Linear(1024, 2048)` – Further deepens the representation.
    - `nn.Linear(2048, 4096)` – Generates a flat vector sufficient to populate a 4‑D tensor.

2.  **Spatial Reshaping**
    - `view(-1, 16, 16, 16)` – Reshapes the 4096‑element vector into a tensor of shape **(B, 16, 16, 16)**, creating a 16‑channel feature map with 16×16 spatial resolution.

3.  **Progressive Upsampling (Conv + Pixel‑Shuffle)**
    - **First Upsampling Block:**
        - `nn.Conv2d(16, 32, 3, stride=1, padding=1)` – Increases channel depth to 32 while maintaining 16×16 spatial dimensions.
        - `nn.PixelShuffle(2)` – Rearranges elements to upscale spatial resolution to **32×32**, reducing channels from 32 to **8**.
    - **Second Upsampling Block:**
        - `nn.Conv2d(8, 16, 3, stride=1, padding=1)` – Refines the 8‑channel 32×32 feature map.
        - `nn.PixelShuffle(2)` – Upsamples to **64×64**, reducing channels from 16 to **4**.
    - **Third Upsampling Block:**
        - `nn.Conv2d(4, 8, 3, stride=1, padding=1)` – Processes the 4‑channel 64×64 representation.
        - `nn.PixelShuffle(2)` – Final upscaling to target resolution **128×128**, reducing channels from 8 to **2**.

4.  **Final Activation**
    - `torch.sigmoid` – Squashes values to [0, 1].
    - `1 - x` – Inverts the output to align with the expected brush stroke format, yielding a single‑channel image after dimensionality reduction.

## Implementation Details in PyTorch

The architecture explicitly avoids transposed convolutions, which can introduce checkerboard artifacts, in favor of **pixel‑shuffle** (`nn.PixelShuffle`) for learned upsampling. Each convolutional layer uses a **3×3 kernel**, stride 1, and padding 1, ensuring spatial dimensions remain constant before the upsampling operation doubles the resolution.

### Key Architectural Features

- **Parameter Efficiency:** The network uses only four fully‑connected layers before transitioning to convolutions, minimizing the parameter count for the latent mapping while delegating spatial generation to the convolutional stack.
- **Multi‑Stage Upsampling:** Three successive `PixelShuffle(2)` operations provide an 8× total spatial upsampling factor (2³), transforming the 16×16 intermediate tensor to the final 128×128 output.
- **Deterministic Output:** The sigmoid activation guarantees stable gradient behavior during training and ensures generated strokes remain within valid pixel ranges.

## Practical Code Examples

### Instantiating the Renderer and Generating Strokes

The following snippet demonstrates how to load the `FCN` class and produce brush strokes from random latent vectors:

```python
import torch
from baseline.Renderer.model import FCN

# Initialize the neural renderer

renderer = FCN()

# Dummy latent vectors: batch size 4, 10‑dimensional style codes

z = torch.randn(4, 10)

# Forward pass generates strokes of shape (B, 1, 128, 128)

stroke = renderer(z)

print(stroke.shape)           # torch.Size([4, 1, 128, 128])

print(stroke.min(), stroke.max())  # Values constrained to [0, 1]

```

### Training the Neural Renderer

This example shows a standard training loop using the Adam optimizer and MSE loss, as implemented in the repository’s training scripts:

```python
import torch
import torch.optim as optim
from baseline.Renderer.model import FCN

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = FCN().to(device)
optimizer = optim.Adam(model.parameters(), lr=1e-4)
criterion = torch.nn.MSELoss()

# Hypothetical data loader providing (style_vector, target_image) pairs

for epoch in range(num_epochs):
    for style_vec, target_img in train_loader:
        style_vec = style_vec.to(device)      # Shape: (B, 10)

        target_img = target_img.to(device)      # Shape: (B, 1, 128, 128)

        optimizer.zero_grad()
        output = model(style_vec)               # Shape: (B, 1, 128, 128)

        loss = criterion(output, target_img)
        loss.backward()
        optimizer.step()

```

### Visualizing Generated Strokes

To inspect the output of the neural renderer FCN architecture during inference or debugging:

```python
import matplotlib.pyplot as plt
import numpy as np

renderer = FCN()
z = torch.randn(1, 10)
stroke = renderer(z).detach().cpu().numpy().squeeze()  # Shape: (128, 128)

plt.imshow(stroke, cmap='gray')
plt.title('Generated Brush Stroke')
plt.axis('off')
plt.show()

```

## Repository File Structure

The neural renderer FCN architecture integrates with the broader Learning to Paint system through the following key files:

- **[`baseline/Renderer/model.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline/Renderer/model.py)** – Contains the `FCN` class definition and forward pass implementation (lines 6‑34).
- [`baseline/Renderer/stroke_gen.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline/Renderer/stroke_gen.py) – Utilities for converting FCN outputs into stroke sequences compatible with the painting environment.
- [`baseline/train_renderer.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline/train_renderer.py) – Training script that optimizes the FCN parameters using stroke‑image pairs.
- [`baseline/utils/util.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline/utils/util.py) – Helper functions for data loading, preprocessing, and tensor transformations.
- [`predict.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/predict.py) – Inference entry point demonstrating how to load a trained FCN checkpoint and generate strokes.

## Summary

- The **neural renderer FCN architecture** accepts a **10‑dimensional** latent vector and outputs a **128×128** single‑channel image.
- The network expands the latent code through **four fully‑connected layers** (10 → 4096 parameters) before reshaping into a spatial tensor.
- **Three pixel‑shuffle upsampling stages** progressively increase resolution from 16×16 to 128×128 while reducing channel depth, avoiding transposed convolution artifacts.
- All convolutional operations use **3×3 kernels** with stride 1 and padding 1, maintaining spatial fidelity throughout the upsampling pipeline.
- The implementation resides in [`baseline/Renderer/model.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline/Renderer/model.py) within the hzwer/iccv2019-learningtopaint repository.

## Frequently Asked Questions

### What is the input dimension of the FCN neural renderer?

The `FCN` class expects a **10‑dimensional** input vector representing the style or parameters of a brush stroke. This vector is processed by the fully‑connected layers to generate the initial 4096‑element feature representation that is reshaped into a 16×16 spatial grid with 16 channels.

### Why does the architecture use pixel‑shuffle instead of transposed convolutions?

The neural renderer FCN architecture employs **`nn.PixelShuffle(2)`** for upsampling because it reduces checkerboard artifacts commonly associated with transposed convolutions. Pixel‑shuffle rearranges channel elements into spatial dimensions, providing a learned upsampling mechanism that preserves high‑frequency details critical for realistic brush texture generation while maintaining stable gradients during backpropagation.

### What is the final image size produced by the neural renderer?

The network outputs a **single‑channel 128×128** pixel image. This resolution is achieved through three successive doubling operations (16→32→64→128) performed by the pixel‑shuffle layers, matching the stroke size used in the ICCV 2019 Learning to Paint paper’s experimental setup.

### How is the FCN class implemented in the Learning to Paint repository?

In [`baseline/Renderer/model.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline/Renderer/model.py), the `FCN` class inherits from `torch.nn.Module` and defines the forward pass as a sequential pipeline: four linear layers, a view operation to (B, 16, 16, 16), three convolutional blocks each followed by `PixelShuffle(2)`, and a final sigmoid activation with inversion. This compact implementation spans approximately 28 lines of code (lines 6‑34) and uses standard PyTorch operations for full differentiability.