Stroke Models in the Learning-to-Paint Repository: Neural Renderer and Brush Checkpoints

The repository provides a fully-convolutional neural renderer (FCN) that predicts stroke opacity maps from 10-dimensional parameter vectors, alongside three pre-trained checkpoints—triangle.pkl, round.pkl, and bezierwotrans.pkl—that define distinct brush geometries.

The Learning-to-Paint repository (ICCV 2019) by hzwer implements a reinforcement learning agent that learns to paint using brush strokes. Central to this system are specialized stroke models that translate high-level action parameters into rasterized canvas outputs, combining a trainable neural network architecture with deterministic geometric rasterization.

The Neural Renderer Architecture (FCN)

The primary stroke prediction engine is a Fully Convolutional Network (FCN) defined in Renderer/model.py for both the baseline and baseline_modelfree variants. This network functions as a differentiable decoder that maps a low-dimensional parameter vector to a high-resolution opacity mask.

Model Implementation

In baseline/Renderer/model.py (mirrored in baseline_modelfree/Renderer/model.py), the FCN class implements the core stroke model. The architecture uses transposed convolutional layers to upsample a 10-dimensional input into a 128×128 pixel stroke map. During inference, this model acts as the decoder, predicting per-pixel opacity values that represent where a single brush stroke applies paint to the canvas.

Input Parameter Format

The FCN accepts a 10-dimensional parameter vector encoding a variable-width Bézier curve:

• (x0, y0): Start point (normalized 0–1)
• (x1, y1): Control point for curve shape
• (x2, y2): End point (normalized 0–1)
• (z0, z2): Brush radius at start and end (pixel units)
• (w0, w2): Brush intensity at start and end (0–1)

During full pipeline execution, as seen in predict.py, these 10 parameters are concatenated with 3 RGB color values, forming a 13-dimensional input vector. The FCN processes only the first 10 dimensions to generate the stroke mask, while the RGB values modulate the final color output.

Pre-trained Renderer Checkpoints for Brush Shapes

While the FCN architecture remains constant, the repository distributes three distinct pre-trained checkpoints. Each checkpoint contains learned weights that specialize the renderer to produce a specific brush geometry, allowing users to swap stroke styles without modifying the network code.

Triangular Brush Strokes

The triangle.pkl checkpoint configures the FCN to render triangular brush strokes. This geometry produces sharp, angular marks suitable for artistic styles requiring directional, pointed brushwork. Load this checkpoint in baseline/test.py or baseline_modelfree/test.py to instantiate a triangular brush decoder.

Circular Brush Strokes

The round.pkl checkpoint generates circular (round) brush strokes. This model creates soft, uniform marks typical of traditional round brushes, often preferred for portrait painting and smooth gradient rendering tasks where stroke edges should remain organic and blunt.

Bezier Strokes Without Transformation

The bezierwotrans.pkl checkpoint implements the default Bezier stroke model from the original ICCV 2019 paper. This renderer produces strokes without additional geometric transformations, serving as the baseline brush type against which the authors benchmark their learning algorithms.

The Stroke Generation Pipeline

Beyond the neural network, the repository implements a deterministic stroke generator that rasterizes parameter vectors into actual pixel arrays. This two-stage process combines the neural renderer's opacity prediction with OpenCV-based drawing functions to produce the final canvas.

Low-Level Rasterization with stroke_gen

The Renderer/stroke_gen.py file contains the draw() function, which provides a geometric implementation of the stroke model. This function takes the same 10-dimensional parameter vector used by the FCN and uses OpenCV to render a single Bézier curve with variable width onto a 128×128 canvas. It returns a numpy array representing the stroke mask, offering a non-learned alternative to the neural renderer.

Compositing Strokes onto Canvas

The decode() function in predict.py demonstrates how the neural stroke model and rasterization integrate into the painting pipeline. The function feeds the 10-dimensional parameters through the FCN to obtain the stroke mask, reshapes the output to (B, 1, 128, 128), and composites multiple strokes (typically grouped in batches of 5) onto the canvas using alpha blending:

canvas = canvas * (1 - stroke) + color_stroke

Here, stroke represents the opacity mask predicted by the FCN (or generated by stroke_gen), while color_stroke represents the RGB-tinted version of that mask.

Loading and Using Stroke Models

The following example demonstrates loading the triangular stroke model and generating a canvas from random parameters:

import torch
from Renderer.model import FCN          # neural renderer (stroke model)

from Renderer.stroke_gen import draw    # rasterises a single stroke

# Load the triangular renderer checkpoint

ckpt_path = "triangle.pkl"   # download from README links

decoder = FCN()
decoder.load_state_dict(torch.load(ckpt_path, map_location="cpu"))
decoder.eval()

def decode(params, canvas, decoder, width=128):
    # params: (B, 13) → first 10 are Bézier params, last 3 are RGB

    stroke = 1 - decoder(params[:, :10])                # (B, 128, 128)

    stroke = stroke.view(-1, width, width, 1)           # add channel dim

    color_stroke = stroke * params[:, -3:].view(-1, 1, 1, 3)

    # Rearrange to (B, C, H, W)

    stroke = stroke.permute(0, 3, 1, 2)
    color_stroke = color_stroke.permute(0, 3, 1, 2)

    # Composite groups of 5 strokes

    stroke = stroke.view(-1, 5, 1, width, width)
    color_stroke = color_stroke.view(-1, 5, 3, width, width)
    
    for i in range(5):
        canvas = canvas * (1 - stroke[:, i]) + color_stroke[:, i]
    return canvas

# Run inference

B = 2
dummy_params = torch.randn(B, 13)
canvas = torch.zeros(B, 3, 128, 128)
canvas = decode(dummy_params, canvas, decoder)

To draw a single stroke using the low-level geometric generator instead of the neural network:

import numpy as np
from Renderer.stroke_gen import draw

# f = (x0, y0, x1, y1, x2, y2, z0, z2, w0, w2)

f = (0.2, 0.2, 0.5, 0.5, 0.8, 0.8, 3, 6, 0.9, 0.3)
stroke_img = draw(f, width=128)   # Returns (128, 128) numpy array

Summary

• The FCN neural renderer in baseline/Renderer/model.py serves as the primary stroke model, converting 10-dimensional Bézier parameters into 128×128 opacity masks.
• Three pre-trained checkpoints—triangle.pkl, round.pkl, and bezierwotrans.pkl—provide distinct brush geometries (triangular, circular, and default Bezier) without requiring architectural modifications.
• The stroke_gen.py module handles deterministic rasterization of Bézier curves using OpenCV, offering both a training target for the FCN and a standalone rendering option.
• The decode() function in predict.py demonstrates the standard inference pattern: neural prediction of stroke masks followed by alpha-compositing with color parameters.

Frequently Asked Questions

What is the difference between the stroke model and the stroke generator?

The stroke model refers specifically to the neural FCN that learns to predict stroke opacity maps from the 10-dimensional parameter vectors. The stroke generator (stroke_gen.draw) is the deterministic, rule-based function that uses OpenCV to rasterize geometric Bézier curves. During training, the FCN learns to approximate the output of the stroke generator, while during inference, the FCN replaces the generator for faster, differentiable rendering.

Can I use custom brush shapes with this repository?

Yes. You can train a new FCN checkpoint using the renderer training scripts to learn arbitrary brush geometries, or modify Renderer/stroke_gen.py to implement custom primitives (e.g., textured brushes). The painting agent is decoupled from the specific stroke model implementation, allowing you to swap renderers by simply loading different .pkl checkpoint files into the decoder.

What are the 10 input parameters to the stroke model?

The 10 parameters encode a variable-width Bézier curve: start point (x0, y0), control point (x1, y1), end point (x2, y2), start radius z0, end radius z2, start intensity w0, and end intensity w2. All coordinates are normalized to [0, 1], while radii are in pixel units and intensities are in [0, 1]. During full pipeline execution in predict.py, these are concatenated with 3 RGB color values to form a 13-dimensional action vector.

Where can I download the pre-trained stroke model checkpoints?

The README.md file in the repository root contains download links for triangle.pkl, round.pkl, and bezierwotrans.pkl. These files contain the state dictionaries for the FCN decoder trained specifically for each brush geometry, and are loaded by the inference scripts (baseline/test.py, baseline_modelfree/test.py) to initialize the stroke model.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}