How to Run Inference and Test Trained Models in LearningToPaint (ICCV 2019)

To run inference on the LearningToPaint models, use baseline/test.py for command-line testing or predict.py for web-based Cog deployment, both requiring renderer.pkl and actor.pkl pretrained weights.

The hzwer/iccv2019-learningtopaint repository provides two ready-to-use scripts for running inference on trained neural painting agents. Whether you want to generate progressive painting sequences from static images or deploy the model as a web service, both workflows rely on the same core pipeline involving a neural renderer and a stroke-predicting agent.

Downloading Pretrained Weights

Before you can run inference or test the trained models, you must obtain the pretrained weight files. The repository expects two specific pickle files: renderer.pkl (the neural renderer) and actor.pkl (the paint agent policy network).

Download the official pretrained weights using wget:

wget "https://drive.google.com/uc?export=download&id=1-7dVdjCIZIxh8hHJnGTK-RA1-jL1tor4" -O renderer.pkl
wget "https://drive.google.com/uc?export=download&id=1a3vpKgjCVXHON4P7wodqhCgCMPgg1KeR" -O actor.pkl

Place these files in the repository root directory, or provide explicit paths via the --renderer and --actor command-line arguments when running inference scripts.

Command-Line Inference with baseline/test.py

The primary method to run inference is the baseline/test.py script. This utility loads the pretrained models, executes the painting loop, and saves intermediate canvas states as PNG images.

Running the Test Script

Execute inference on a single image using the following pattern:

python3 baseline/test.py \
    --max_step=100 \
    --actor=actor.pkl \
    --renderer=renderer.pkl \
    --img=image/test.png \
    --divide=4

Key Parameters Explained

• --max_step: Controls the number of "thinking" iterations. Each step emits 5 brush strokes, so 100 steps produce 500 total strokes.
• --divide: Splits the target image into a grid for higher-resolution results (optional). Use values like 4 or 5 to process large images in patches.
• --actor: Path to the ResNet-based policy network defined in baseline/DRL/actor.py.
• --renderer: Path to the fully convolutional neural renderer (FCN) defined in baseline/Renderer/model.py.

Output Handling

The script creates an output/ directory and writes a sequence of PNG files (generated0.png, generated1.png, …) showing the progressive painting process. These frames capture the canvas state after each thinking step, utilizing the decode, large2small, small2large, and smooth helper functions from the source code to manage stroke rendering and image transitions.

Web-Based Deployment with predict.py

For production environments or web demos, use the Cog predictor wrapper in predict.py. This script encapsulates the same inference pipeline as baseline/test.py but returns an animated GIF suitable for API responses.

Running the Cog Predictor

Install Cog and run inference as a web service:

python3 -m cog predict -i image=test.png -i renderer=renderer.pkl -i actor=actor.pkl

The predictor class defined in predict.py handles model loading onto the appropriate device and executes the same agent-renderer loop found in the baseline test script. This approach is ideal for containerized deployments where you need to run inference or test the trained models via HTTP endpoints rather than local file system operations.

Post-Processing: Creating Videos from Output Frames

The PNG sequence generated by baseline/test.py can be assembled into a video file to visualize the painting process. Use ffmpeg to convert the frame sequence:

ffmpeg -r 10 -f image2 -i output/generated%d.png -s 512x512 \
       -c:v libx264 -pix_fmt yuv420p video.mp4 -q:v 0 -q:a 0

This command reads all generated PNG files at 10 frames per second and encodes them into video.mp4 with YUV420p pixel formatting for maximum compatibility.

Summary

• Two inference paths exist: baseline/test.py for local batch processing and predict.py for Cog-based web services.
• Required assets: You must download renderer.pkl and actor.pkl before running either script.
• Core implementation: Both scripts rely on the neural renderer (baseline/Renderer/model.py) and the actor network (baseline/DRL/actor.py), using helper functions like decode and smooth to manage stroke generation.
• Output format: Command-line testing produces numbered PNG frames, while the Cog predictor returns an animated GIF.
• Video conversion: Use ffmpeg to compile PNG sequences into shareable MP4 files showing the progressive painting process.

Frequently Asked Questions

Where can I find the pretrained model weights for inference?

The official pretrained weights are hosted on Google Drive and linked in the README.md Testing section. You need two files: renderer.pkl (the FCN neural renderer) and actor.pkl (the ResNet policy network). Download them using the wget commands provided in the repository documentation or copy them manually to your working directory.

What is the difference between --max_step and the total number of brush strokes?

The --max_step parameter controls the number of agent "thinking" iterations, not individual strokes. According to the implementation in baseline/test.py, each thinking step generates 5 brush strokes. Therefore, setting --max_step=100 produces 500 total strokes across the painting sequence, with the canvas state saved after each step.

Can I run inference on high-resolution images?

Yes. Use the --divide argument in baseline/test.py to enable a multi-patch processing mode. This splits the target image into a grid (e.g., --divide=4 creates a 4x4 grid), processes each patch independently using the large2small and small2large utilities, and reconstructs the full-resolution output. This approach allows the fixed-size neural renderer to handle images larger than its training resolution.

How do I modify the inference code to use my own trained models?

Replace the paths passed to --actor and --renderer with your custom pickle files. Ensure your trained models maintain the same architecture as the baseline implementations: the renderer must follow the FCN structure in baseline/Renderer/model.py, and the actor must match the ResNet-based policy in baseline/DRL/actor.py. Both scripts accept explicit file paths, so no code modification is required if your weights use compatible formats.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}