# Neural Renderer Network Definition in hzwer/iccv2019-learningtopaint: FCN Architecture and File Location

> Find the neural renderer network definition in hzwer/iccv2019-learningtopaint. Learn how the FCN class in model.py maps stroke vectors to RGB images.

- Repository: [hzwer/iccv2019-learningtopaint](https://github.com/hzwer/iccv2019-learningtopaint)
- Tags: architecture
- Published: 2026-03-03

---

**The neural renderer network definition is implemented as the `FCN` class in [`baseline/Renderer/model.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline/Renderer/model.py) (and identically in [`baseline_modelfree/Renderer/model.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/Renderer/model.py)), defining a fully-convolutional network that maps a 10-dimensional stroke vector to a 128×128 RGB image.**

The `hzwer/iccv2019-learningtopaint` repository contains the official PyTorch implementation of the ICCV 2019 paper "Learning to Paint". At the core of this stroke-based rendering system lies the **neural renderer network definition**, which serves as the differentiable brush engine that converts low-dimensional stroke parameters into pixel-level outputs. This article identifies the exact file locations, examines the `FCN` class architecture, and provides practical code examples for integrating the renderer into training or inference pipelines.

## Neural Renderer Architecture Overview

The neural renderer is implemented as the **`FCN`** class, a fully-convolutional network that progressively expands a low-dimensional latent vector into a full-resolution image. According to the source code analysis, the network accepts a **10-dimensional input** representing brush stroke parameters and outputs a **128×128 image** with pixel values in the range `[0, 1]`.

The architecture consists of two distinct stages:

1. **Fully-connected tower**: Expands the 10-D vector through dense layers to a high-dimensional feature representation
2. **Convolutional upsampling pipeline**: Uses strided convolutions combined with `nn.PixelShuffle` to progressively increase spatial resolution from 16×16 to 128×128

```

FCN (nn.Module)
└─ Fully‑connected tower
   ├─ Linear(10 → 512)
   ├─ Linear(512 → 1024)
   ├─ Linear(1024 → 2048)
   └─ Linear(2048 → 4096)
   → reshaped to (batch, 16, 16, 16)

└─ Convolution‑up‑sampling pipeline
   ├─ Conv2d(16 → 32)  + ReLU
   ├─ Conv2d(32 → 32)  → PixelShuffle(2)   (↑×2)
   ├─ Conv2d(8  → 16)  + ReLU
   ├─ Conv2d(16 → 16)  → PixelShuffle(2)   (↑×2)
   ├─ Conv2d(4  → 8)   + ReLU
   ├─ Conv2d(8  → 4)   → PixelShuffle(2)   (↑×2)
   └─ Sigmoid → 1‑x → reshape to (batch, 128, 128)

```

## File Location of the Neural Renderer Network Definition

The `FCN` class definition is duplicated across two baseline variants in the repository. Both implementations are identical and are imported throughout the codebase in files such as [`predict.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/predict.py), [`train_renderer.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/train_renderer.py), [`test.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/test.py), and various DRL training scripts.

### Baseline Implementation Path

The primary **neural renderer network definition** resides in:

- [`baseline/Renderer/model.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline/Renderer/model.py)

This file contains the complete `FCN` class definition used by the standard supervised learning pipeline.

### Model-Free Variant Path

An identical copy exists for the model-free reinforcement learning variant:

- [`baseline_modelfree/Renderer/model.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/Renderer/model.py)

Both files define the same network architecture and can be imported interchangeably depending on which training paradigm you are using.

## FCN Class Implementation Details

The `FCN` class inherits from `nn.Module` and implements a custom forward pass that handles the dimensional transformations between the fully-connected and convolutional stages.

### Fully-Connected Tower

The first stage processes the input stroke vector through a deep MLP:

- **Input**: Tensor of shape `(batch_size, 10)`
- **Hidden layers**: 512, 1024, 2048 units with ReLU activations
- **Output**: 4096 units, reshaped to `(batch, 16, 16, 16)` to serve as the initial feature map

This expansion creates a rich, high-dimensional representation that contains the structural information needed to reconstruct the brush stroke.

### Convolutional Upsampling Pipeline

The second stage uses **PixelShuffle** (sub-pixel convolution) to upsample the feature map through three successive doubling operations (16→32→64→128):

1. Conv2d(16→32) → ReLU → Conv2d(32→32) → PixelShuffle(2)
2. Conv2d(8→16) → ReLU → Conv2d(16→16) → PixelShuffle(2)
3. Conv2d(4→8) → ReLU → Conv2d(8→4) → PixelShuffle(2)

Finally, a **Sigmoid** activation followed by the operation `1 - x` produces the final output image with values inverted to the `[0, 1]` range suitable for alpha compositing onto a canvas.

## Using the Neural Renderer in Code

The following examples demonstrate how to instantiate, train, and persist the neural renderer based on the actual usage patterns found in the repository.

### Instantiating and Forward Pass

Import the `FCN` class from the appropriate baseline directory and run inference:

```python
import torch
from baseline.Renderer.model import FCN   # or baseline_modelfree.Renderer.model

# Create the renderer

renderer = FCN()

# Dummy latent vector (batch size = 1, dim = 10)

z = torch.randn(1, 10)

# Generate an image

img = renderer(z)          # shape: (1, 128, 128)

print(img.shape)           # torch.Size([1, 128, 128])

```

### Training Loop Integration

When training the renderer (as done in [`baseline/train_renderer.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline/train_renderer.py)), use standard PyTorch optimization:

```python
optimizer = torch.optim.Adam(renderer.parameters(), lr=1e-4)

for step in range(num_steps):
    # Sample latent vectors and ground‑truth images

    z, target = get_batch()               # z: (B,10), target: (B,128,128)

    pred = renderer(z)                     # forward

    loss = torch.nn.functional.mse_loss(pred, target)

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

```

### Saving and Loading Checkpoints

Persist trained weights using PyTorch's state dict mechanism:

```python

# Save

torch.save(renderer.state_dict(), "renderer_fcn.pth")

# Load

renderer = FCN()
renderer.load_state_dict(torch.load("renderer_fcn.pth"))
renderer.eval()

```

## Related Files and Integration Points

The neural renderer network definition is utilized throughout the codebase by several key modules:

| File | Purpose |
|---|---|
| [`baseline/Renderer/stroke_gen.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline/Renderer/stroke_gen.py) | Generates the 10-dimensional stroke representation fed into `FCN` |
| [`predict.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/predict.py) | Entry point for inference that imports `FCN` and renders user-provided stroke sequences |
| [`baseline/train_renderer.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline/train_renderer.py) | Orchestrates data loading and optimization for the renderer |
| [`baseline_modelfree/train_renderer.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/train_renderer.py) | Training pipeline for the model-free reinforcement learning variant |
| [`test.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/test.py) | Evaluation scripts that use the trained `FCN` for metric calculation |

These files together constitute the complete workflow for stroke generation, neural rendering, and canvas compositing.

## Summary

- The **neural renderer network definition** is located in [`baseline/Renderer/model.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline/Renderer/model.py) and [`baseline_modelfree/Renderer/model.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/Renderer/model.py) as the `FCN` class.
- The network accepts a **10-dimensional stroke vector** and outputs a **128×128 RGB image** through a combination of fully-connected expansion and PixelShuffle upsampling.
- The architecture uses a **4096-unit bottleneck** reshaped to 16×16×16, followed by three 2× upsampling stages to reach full resolution.
- The class is imported across training, testing, and inference scripts including [`predict.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/predict.py) and [`train_renderer.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/train_renderer.py).
- Both baseline variants share **identical network definitions**, ensuring consistent stroke rendering across supervised and reinforcement learning pipelines.

## Frequently Asked Questions

### What is the input dimension of the neural renderer?

The `FCN` class accepts a **10-dimensional tensor** representing brush stroke parameters. These parameters typically encode stroke location, rotation, size, and color information generated by [`stroke_gen.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/stroke_gen.py) or predicted by the agent network.

### Where is the neural renderer trained in the codebase?

Training occurs in [`baseline/train_renderer.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline/train_renderer.py) for the supervised baseline and [`baseline_modelfree/train_renderer.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/train_renderer.py) for the model-free variant. Both scripts import the `FCN` class from their respective [`Renderer/model.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/Renderer/model.py) files and optimize the network using MSE loss between predicted and ground-truth brush strokes.

### What is the difference between the baseline and model-free renderer?

There is **no architectural difference** between the two renderers. Both the baseline ([`baseline/Renderer/model.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline/Renderer/model.py)) and model-free ([`baseline_modelfree/Renderer/model.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/Renderer/model.py)) implementations contain identical `FCN` class definitions. The distinction lies in how the 10-dimensional stroke vectors are generated—either from a learned model or from the reinforcement learning agent—while the neural renderer network definition remains constant.

### How does the FCN class generate high-resolution images from low-dimensional inputs?

The `FCN` class uses a **two-stage pipeline**: first, a fully-connected tower expands the 10-D vector to 4096 features, which are reshaped into a 16×16 spatial grid with 16 channels. Then, a series of convolutional layers with **PixelShuffle** operations progressively doubles the spatial resolution three times (16→32→64→128) while reducing channels, ultimately producing the final 128×128 output through a sigmoid activation.