# Stroke Models in the Learning-to-Paint Repository: Neural Renderer and Brush Checkpoints

> Explore stroke models in the learning-to-paint repository. Discover neural renderer and brush checkpoints like triangle round and bezierwotrans for diverse stroke prediction.

- Repository: [hzwer/iccv2019-learningtopaint](https://github.com/hzwer/iccv2019-learningtopaint)
- Tags: deep-dive
- Published: 2026-03-03

---

**The repository provides a fully-convolutional neural renderer (FCN) that predicts stroke opacity maps from 10-dimensional parameter vectors, alongside three pre-trained checkpoints—`triangle.pkl`, `round.pkl`, and `bezierwotrans.pkl`—that define distinct brush geometries.**

The *Learning-to-Paint* repository (ICCV 2019) by hzwer implements a reinforcement learning agent that learns to paint using brush strokes. Central to this system are specialized **stroke models** that translate high-level action parameters into rasterized canvas outputs, combining a trainable neural network architecture with deterministic geometric rasterization.

## The Neural Renderer Architecture (FCN)

The primary stroke prediction engine is a **Fully Convolutional Network (FCN)** defined in [`Renderer/model.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/Renderer/model.py) for both the `baseline` and `baseline_modelfree` variants. This network functions as a differentiable decoder that maps a low-dimensional parameter vector to a high-resolution opacity mask.

### Model Implementation

In [`baseline/Renderer/model.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline/Renderer/model.py) (mirrored in [`baseline_modelfree/Renderer/model.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/Renderer/model.py)), the `FCN` class implements the core **stroke model**. The architecture uses transposed convolutional layers to upsample a 10-dimensional input into a 128×128 pixel stroke map. During inference, this model acts as the decoder, predicting per-pixel opacity values that represent where a single brush stroke applies paint to the canvas.

### Input Parameter Format

The FCN accepts a **10-dimensional parameter vector** encoding a variable-width Bézier curve:

- `(x0, y0)`: Start point (normalized 0–1)
- `(x1, y1)`: Control point for curve shape
- `(x2, y2)`: End point (normalized 0–1)
- `(z0, z2)`: Brush radius at start and end (pixel units)
- `(w0, w2)`: Brush intensity at start and end (0–1)

During full pipeline execution, as seen in [`predict.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/predict.py), these 10 parameters are concatenated with 3 RGB color values, forming a 13-dimensional input vector. The FCN processes only the first 10 dimensions to generate the stroke mask, while the RGB values modulate the final color output.

## Pre-trained Renderer Checkpoints for Brush Shapes

While the FCN architecture remains constant, the repository distributes three distinct pre-trained checkpoints. Each checkpoint contains learned weights that specialize the renderer to produce a specific **brush geometry**, allowing users to swap stroke styles without modifying the network code.

### Triangular Brush Strokes

The `triangle.pkl` checkpoint configures the FCN to render **triangular** brush strokes. This geometry produces sharp, angular marks suitable for artistic styles requiring directional, pointed brushwork. Load this checkpoint in [`baseline/test.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline/test.py) or [`baseline_modelfree/test.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/test.py) to instantiate a triangular brush decoder.

### Circular Brush Strokes

The `round.pkl` checkpoint generates **circular** (round) brush strokes. This model creates soft, uniform marks typical of traditional round brushes, often preferred for portrait painting and smooth gradient rendering tasks where stroke edges should remain organic and blunt.

### Bezier Strokes Without Transformation

The `bezierwotrans.pkl` checkpoint implements the **default Bezier** stroke model from the original ICCV 2019 paper. This renderer produces strokes without additional geometric transformations, serving as the baseline brush type against which the authors benchmark their learning algorithms.

## The Stroke Generation Pipeline

Beyond the neural network, the repository implements a deterministic **stroke generator** that rasterizes parameter vectors into actual pixel arrays. This two-stage process combines the neural renderer's opacity prediction with OpenCV-based drawing functions to produce the final canvas.

### Low-Level Rasterization with stroke_gen

The [`Renderer/stroke_gen.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/Renderer/stroke_gen.py) file contains the `draw()` function, which provides a geometric implementation of the stroke model. This function takes the same 10-dimensional parameter vector used by the FCN and uses OpenCV to render a single Bézier curve with variable width onto a 128×128 canvas. It returns a numpy array representing the stroke mask, offering a non-learned alternative to the neural renderer.

### Compositing Strokes onto Canvas

The `decode()` function in [`predict.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/predict.py) demonstrates how the neural **stroke model** and rasterization integrate into the painting pipeline. The function feeds the 10-dimensional parameters through the FCN to obtain the stroke mask, reshapes the output to `(B, 1, 128, 128)`, and composites multiple strokes (typically grouped in batches of 5) onto the canvas using alpha blending:

```python
canvas = canvas * (1 - stroke) + color_stroke

```

Here, `stroke` represents the opacity mask predicted by the FCN (or generated by `stroke_gen`), while `color_stroke` represents the RGB-tinted version of that mask.

## Loading and Using Stroke Models

The following example demonstrates loading the triangular stroke model and generating a canvas from random parameters:

```python
import torch
from Renderer.model import FCN          # neural renderer (stroke model)

from Renderer.stroke_gen import draw    # rasterises a single stroke

# Load the triangular renderer checkpoint

ckpt_path = "triangle.pkl"   # download from README links

decoder = FCN()
decoder.load_state_dict(torch.load(ckpt_path, map_location="cpu"))
decoder.eval()

def decode(params, canvas, decoder, width=128):
    # params: (B, 13) → first 10 are Bézier params, last 3 are RGB

    stroke = 1 - decoder(params[:, :10])                # (B, 128, 128)

    stroke = stroke.view(-1, width, width, 1)           # add channel dim

    color_stroke = stroke * params[:, -3:].view(-1, 1, 1, 3)

    # Rearrange to (B, C, H, W)

    stroke = stroke.permute(0, 3, 1, 2)
    color_stroke = color_stroke.permute(0, 3, 1, 2)

    # Composite groups of 5 strokes

    stroke = stroke.view(-1, 5, 1, width, width)
    color_stroke = color_stroke.view(-1, 5, 3, width, width)
    
    for i in range(5):
        canvas = canvas * (1 - stroke[:, i]) + color_stroke[:, i]
    return canvas

# Run inference

B = 2
dummy_params = torch.randn(B, 13)
canvas = torch.zeros(B, 3, 128, 128)
canvas = decode(dummy_params, canvas, decoder)

```

To draw a single stroke using the low-level geometric generator instead of the neural network:

```python
import numpy as np
from Renderer.stroke_gen import draw

# f = (x0, y0, x1, y1, x2, y2, z0, z2, w0, w2)

f = (0.2, 0.2, 0.5, 0.5, 0.8, 0.8, 3, 6, 0.9, 0.3)
stroke_img = draw(f, width=128)   # Returns (128, 128) numpy array

```

## Summary

- The **FCN neural renderer** in [`baseline/Renderer/model.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline/Renderer/model.py) serves as the primary stroke model, converting 10-dimensional Bézier parameters into 128×128 opacity masks.
- Three pre-trained checkpoints—**`triangle.pkl`**, **`round.pkl`**, and **`bezierwotrans.pkl`**—provide distinct brush geometries (triangular, circular, and default Bezier) without requiring architectural modifications.
- The **[`stroke_gen.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/stroke_gen.py)** module handles deterministic rasterization of Bézier curves using OpenCV, offering both a training target for the FCN and a standalone rendering option.
- The **`decode()`** function in [`predict.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/predict.py) demonstrates the standard inference pattern: neural prediction of stroke masks followed by alpha-compositing with color parameters.

## Frequently Asked Questions

### What is the difference between the stroke model and the stroke generator?

The **stroke model** refers specifically to the neural FCN that learns to predict stroke opacity maps from the 10-dimensional parameter vectors. The **stroke generator** (`stroke_gen.draw`) is the deterministic, rule-based function that uses OpenCV to rasterize geometric Bézier curves. During training, the FCN learns to approximate the output of the stroke generator, while during inference, the FCN replaces the generator for faster, differentiable rendering.

### Can I use custom brush shapes with this repository?

Yes. You can train a new FCN checkpoint using the renderer training scripts to learn arbitrary brush geometries, or modify [`Renderer/stroke_gen.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/Renderer/stroke_gen.py) to implement custom primitives (e.g., textured brushes). The painting agent is decoupled from the specific stroke model implementation, allowing you to swap renderers by simply loading different `.pkl` checkpoint files into the decoder.

### What are the 10 input parameters to the stroke model?

The 10 parameters encode a variable-width Bézier curve: start point `(x0, y0)`, control point `(x1, y1)`, end point `(x2, y2)`, start radius `z0`, end radius `z2`, start intensity `w0`, and end intensity `w2`. All coordinates are normalized to [0, 1], while radii are in pixel units and intensities are in [0, 1]. During full pipeline execution in [`predict.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/predict.py), these are concatenated with 3 RGB color values to form a 13-dimensional action vector.

### Where can I download the pre-trained stroke model checkpoints?

The [`README.md`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/README.md) file in the repository root contains download links for `triangle.pkl`, `round.pkl`, and `bezierwotrans.pkl`. These files contain the state dictionaries for the FCN decoder trained specifically for each brush geometry, and are loaded by the inference scripts ([`baseline/test.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline/test.py), [`baseline_modelfree/test.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/test.py)) to initialize the stroke model.