How to Combine Rendered Strokes into a Video with Learning-to-Paint

To combine rendered strokes into a video, execute the baseline/test.py script to generate a chronological sequence of PNG images, then use the ffmpeg command documented in the repository to encode them into an MP4 file.

The Learning-to-Paint repository by hzwer implements a deep reinforcement learning agent that recreates images using a limited number of brush strokes. During inference, the code renders each stroke incrementally and saves intermediate canvas states as individual image files. You can then combine these frames into a time-lapse video that visualizes the painting process from blank canvas to final artwork.

How the Repository Generates Image Sequences

The stroke rendering pipeline creates a discrete series of images that represent the painting's progression. Understanding this internal mechanism ensures you know exactly which files to feed into your video encoder.

The Rendering Loop in baseline/test.py

The core inference logic resides in [baseline/test.py](https://github.com/hzwer/iccv2019-learningtopaint/blob/master/baseline/test.py). Inside the main loop (lines 21–33), the script processes the target image through multiple DRL steps. For each step, the decode function (lines 40–53) converts the agent's action vector into five distinct brush strokes, each possessing its own alpha mask and RGB color.

The canvas updates via alpha compositing using the formula:

canvas = canvas * (1 - stroke) + color_stroke

After each compositing operation, the intermediate canvas state is appended to a results list. Once all strokes for a step are rendered, the code calls save_img(res[j], args.imgid) (lines 24–31), which writes files to output/generated{N}.png where N increments sequentially (see the save_img implementation at lines 90–100).

Converting PNG Sequences to Video with FFmpeg

The repository standardizes on ffmpeg for video assembly, providing a specific command in [README.md](https://github.com/hzwer/iccv2019-learningtopaint/blob/master/README.md#L41) that handles the numbered image sequence correctly.

Recommended ffmpeg command:

ffmpeg -r 10 -f image2 -i output/generated%d.png -s 512x512 -c:v libx264 -pix_fmt yuv420p video.mp4 -q:v 0 -q:a 0

Parameter breakdown:

• -r 10: Sets the frame rate to 10 frames per second, creating a smooth time-lapse effect
• -f image2: Forces ffmpeg to treat input as an image sequence
• -i output/generated%d.png: Uses the wildcard pattern %d to match the numbered files (generated0.png, generated1.png, etc.)
• -s 512x512: Forces output resolution to 512×512 pixels, matching the original canvas dimensions
• -c:v libx264 -pix_fmt yuv420p: Encodes using H.264 with standard pixel format for broad compatibility

Step-by-Step Implementation

Follow these exact steps to generate your painting video from the source code.

Generate the Stroke Images

First, download the pretrained models and run the inference script to create the image sequence:

# Download pretrained renderer and actor models

wget "https://drive.google.com/uc?export=download&id=1-7dVdjCIZIxh8hHJnGTK-RA1-jL1tor4" -O renderer.pkl
wget "https://drive.google.com/uc?export=download&id=1a3vpKgjCVXHON4P7wodqhCgCMPgg1KeR" -O actor.pkl

# Generate the stroke sequence

python3 baseline/test.py \
    --max_step=100 \
    --actor=actor.pkl \
    --renderer=renderer.pkl \
    --img=image/test.png \
    --divide=4

This execution populates the output/ directory with files named generated0.png, generated1.png, and so on—one file per sub-stroke rendered by the neural network.

Assemble the Video

Navigate to the output directory and execute the ffmpeg command to stitch the frames into a video:

ffmpeg -r 10 -f image2 -i output/generated%d.png \
       -s 512x512 -c:v libx264 -pix_fmt yuv420p video.mp4 \
       -q:v 0 -q:a 0

You can modify the frame rate by changing -r 10 to a higher value (e.g., -r 30) for faster playback, or adjust resolution by modifying the -s flag if you modified the canvas size in the source.

Jupyter Notebook Alternative

If you prefer an interactive environment, the repository includes LearningToPaint.ipynb. Cell line 124 contains the identical ffmpeg command wrapped for notebook execution:

!ffmpeg -r 30 -f image2 -i output/generated%d.png -s 512x512 -c:v libx264 -pix_fmt yuv420p video.mp4 -q:v 0 -q:a 0

Execute this cell after the image generation cells complete to produce video.mp4 directly within your notebook environment.

Summary

• The baseline/test.py script generates a chronological sequence of PNG files in the output/ folder, with filenames following the pattern generated{N}.png
• Each PNG represents an intermediate canvas state after a specific brush stroke is applied using alpha compositing (canvas * (1 - stroke) + color_stroke)
• Use ffmpeg with the -f image2 input format and -i output/generated%d.png pattern to read the numbered sequence
• The recommended encoding settings (-c:v libx264 -pix_fmt yuv420p) ensure compatibility across video players while maintaining the original 512×512 resolution
• Both the command-line interface and the Jupyter notebook provide identical video generation capabilities

Frequently Asked Questions

What file naming convention does the repository use for intermediate canvases?

The save_img function in baseline/test.py (lines 90–100) writes files using the pattern output/generated{N}.png, where {N} represents a sequential integer starting from zero and incrementing after each stroke is rendered. This consecutive numbering allows ffmpeg's %d wildcard pattern to process the files in chronological order automatically.

How do I adjust the video frame rate or resolution?

Modify the -r flag in the ffmpeg command to change frames per second (e.g., -r 30 for 30 fps), and adjust the -s flag to alter resolution (e.g., -s 1024x1024 for upscaled output). Ensure the resolution matches or exceeds your source canvas dimensions to avoid distortion of the rendered strokes.

Can I generate the video directly from the Jupyter notebook?

Yes. The provided LearningToPaint.ipynb notebook contains an ffmpeg execution cell at line 124 that processes the output/generated%d.png sequence. Run this cell after the inference cells complete to generate the video without switching to a terminal.

Why does the canvas update use alpha compositing?

The decode function (lines 40–53 in baseline/test.py) generates strokes with alpha masks to simulate realistic brush transparency. The compositing formula canvas = canvas * (1 - stroke) + color_stroke blends new paint over existing canvas content, allowing later strokes to partially cover earlier ones and creating natural-looking layered brushwork essential for the final video's visual quality.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}