How to Generate a Stroke Using the Neural Renderer with Custom Parameters in Python

You can generate custom brush strokes in the Learning to Paint project by passing a 10-dimensional parameter vector to the draw() function in baseline/Renderer/stroke_gen.py, which rasterizes a quadratic Bézier curve with variable radius and opacity into a 2D mask.

The ICCV 2019 Learning to Paint repository implements a neural renderer that converts compact action parameters into realistic brush strokes. While the full pipeline uses a learned decoder network, you can generate strokes independently using the lightweight pure-Python implementation provided in the source code. This approach allows you to specify custom geometries and opacity profiles without invoking the entire neural network.

Understanding the Stroke Parameter Format

The neural renderer interprets strokes as quadratic Bézier curves defined by 10 floating-point parameters. These parameters encode spatial coordinates, brush thickness, and transparency values that the rasterization engine converts into pixel masks.

The 10-Dimensional Parameter Vector

In baseline/Renderer/stroke_gen.py, the draw() function expects a parameter vector f structured as follows:

• x0, y0 (indices 0, 1): Start point coordinates normalized 0–1
• x1, y1 (indices 2, 3): Control point for the quadratic Bézier curve, normalized 0–1
• x2, y2 (indices 4, 5): End point coordinates normalized 0–1
• z0, z2 (indices 6, 7): Radius at start and end as a fraction of canvas width, 0–1
• w0, w2 (indices 8, 9): Opacity at start and end (0 = transparent, 1 = opaque)

Rendering a Single Stroke with stroke_gen.py

The baseline/Renderer/stroke_gen.py file provides a self-contained implementation that bypasses the neural network decoder. This module uses OpenCV to rasterize the Bézier curve directly from raw parameters.

The draw(f, width=128) function maps the normalized coordinates to pixel positions based on width * 2, calculates 100 intermediate points along the quadratic Bézier curve, and renders circular brushes at each point using cv2.circle. The function returns a 2D NumPy array where values are inverted (1 - mask) so that 1.0 represents fully painted pixels and 0.0 represents empty canvas.

Code Example: Rendering a Custom Stroke

# Example: generate a custom stroke with the Learning‑to‑Paint renderer

import cv2
import numpy as np
import matplotlib.pyplot as plt
from baseline.Renderer.stroke_gen import draw   # ← core rasterisation routine

# ------------------------------------------------------------------

# 1️⃣  Define the stroke parameters.

#    All coordinates are normalised (0‑1) relative to the canvas.

#    Here we draw a curve from the lower‑left to the upper‑right,

#    with a larger radius at the start and a tapering end.

# ------------------------------------------------------------------

params = [
    0.1, 0.9,   # x0, y0  (start point)

    0.5, 0.2,   # x1, y1  (control point)

    0.9, 0.1,   # x2, y2  (end point)

    0.2, 0.05,  # z0, z2  (radius start/end, as fraction of width)

    1.0, 0.0    # w0, w2  (opacity start/end)

]

# ------------------------------------------------------------------

# 2️⃣  Render the stroke.

#    `width` controls the resolution of the output mask.

# ------------------------------------------------------------------

stroke_mask = draw(params, width=256)          # returns a 2‑D float32 array (0‑1)

# ------------------------------------------------------------------

# 3️⃣  Visualise / save the result.

# ------------------------------------------------------------------

plt.imshow(stroke_mask, cmap='gray')
plt.title('Custom Stroke')
plt.axis('off')
plt.show()

# Save as PNG

out_path = 'custom_stroke.png'
cv2.imwrite(out_path, (stroke_mask * 255).astype(np.uint8))
print(f'Stroke saved to {out_path}')

Generating Batches of Strokes

For data augmentation or synthetic dataset creation, you can process multiple parameter vectors efficiently. The draw() function executes quickly enough to generate thousands of stroke masks in a loop.

batch_params = [
    # (x0, y0, x1, y1, x2, y2, z0, z2, w0, w2)

    [0.2, 0.8, 0.5, 0.5, 0.8, 0.2, 0.15, 0.05, 1.0, 0.3],
    [0.1, 0.1, 0.5, 0.9, 0.9, 0.1, 0.2, 0.1, 0.8, 0.0],
    # add more …

]

strokes = np.stack([draw(p, width=128) for p in batch_params], axis=0)   # shape: (B, H, W)

Integration with the Full Neural Renderer

While stroke_gen.py provides direct rasterization, the complete Learning to Paint system uses these same parameters within a learned pipeline. In baseline/Renderer/model.py, a small fully-connected decoder network learns to map the 10-dimensional vector to stroke masks.

During inference, as shown in predict.py, the actor network predicts stroke parameters, which then pass through a decode() function. This function calls stroke = 1 - Decoder(x[:, :10]) to obtain the rasterized output before blending it onto the canvas. By using the pure-Python draw() routine instead, you gain direct control over the geometry without neural network inference overhead.

Summary

• The draw() function in baseline/Renderer/stroke_gen.py converts 10-dimensional parameter vectors into rasterized stroke masks using quadratic Bézier curves.
• Parameters include normalized start/control/end points (x0–x2, y0–y2), radius values (z0, z2), and opacity values (w0, w2) ranging from 0 to 1.
• The implementation draws 100 intermediate points along the curve using OpenCV's cv2.circle, mapping coordinates to a resolution of width * 2 before resizing.
• For neural rendering workflows, the same parameter format feeds into the learned decoder in baseline/Renderer/model.py, as demonstrated in predict.py.
• Direct parameter manipulation enables visual debugging, synthetic data generation, and custom brush behaviors outside the trained model constraints.

Frequently Asked Questions

What Python dependencies are required to use the stroke generator?

You need OpenCV (cv2) and NumPy installed in your environment. The stroke_gen.py module has no PyTorch or TensorFlow dependencies, making it lightweight for pure rendering tasks. Matplotlib is recommended only for visualization purposes in the examples.

Can I use different brush shapes beyond the circular brush?

The current implementation in baseline/Renderer/stroke_gen.py uses cv2.circle to render each point along the Bézier curve, creating a tapered stroke effect when radii vary. To implement custom brush shapes, you would need to modify the drawing logic inside the loop that creates the 100 intermediate points, replacing the circle drawing with your custom mask application.

How does the width parameter affect the output resolution?

The width argument specifies the final output dimension of the square mask. Internally, the function renders at double this resolution (width * 2) to ensure anti-aliased quality, then downsamples to the requested size using OpenCV interpolation. This supersampling technique produces smoother curves than direct rendering at the target resolution.

Why are the stroke masks inverted (1 - mask) in the output?

The inversion ensures that pixel values of 1.0 represent fully painted areas and 0.0 represent transparent background, which aligns with the alpha blending logic used in the full renderer pipeline. When blending strokes onto a canvas, this format allows simple multiplication-based compositing where higher values indicate stronger paint presence.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}