# How to Run Inference and Test Trained Models in LearningToPaint (ICCV 2019)

> Learn to run inference and test trained models for LearningToPaint ICCV 2019. Use command-line or web deployment with pretrained weights for easy model testing.

- Repository: [hzwer/iccv2019-learningtopaint](https://github.com/hzwer/iccv2019-learningtopaint)
- Tags: how-to-guide
- Published: 2026-03-03

---

**To run inference on the LearningToPaint models, use [`baseline/test.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline/test.py) for command-line testing or [`predict.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/predict.py) for web-based Cog deployment, both requiring `renderer.pkl` and `actor.pkl` pretrained weights.**

The `hzwer/iccv2019-learningtopaint` repository provides two ready-to-use scripts for running inference on trained neural painting agents. Whether you want to generate progressive painting sequences from static images or deploy the model as a web service, both workflows rely on the same core pipeline involving a neural renderer and a stroke-predicting agent.

## Downloading Pretrained Weights

Before you can run inference or test the trained models, you must obtain the pretrained weight files. The repository expects two specific pickle files: `renderer.pkl` (the neural renderer) and `actor.pkl` (the paint agent policy network).

Download the official pretrained weights using `wget`:

```bash
wget "https://drive.google.com/uc?export=download&id=1-7dVdjCIZIxh8hHJnGTK-RA1-jL1tor4" -O renderer.pkl
wget "https://drive.google.com/uc?export=download&id=1a3vpKgjCVXHON4P7wodqhCgCMPgg1KeR" -O actor.pkl

```

Place these files in the repository root directory, or provide explicit paths via the `--renderer` and `--actor` command-line arguments when running inference scripts.

## Command-Line Inference with [`baseline/test.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline/test.py)

The primary method to run inference is the [`baseline/test.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline/test.py) script. This utility loads the pretrained models, executes the painting loop, and saves intermediate canvas states as PNG images.

### Running the Test Script

Execute inference on a single image using the following pattern:

```bash
python3 baseline/test.py \
    --max_step=100 \
    --actor=actor.pkl \
    --renderer=renderer.pkl \
    --img=image/test.png \
    --divide=4

```

### Key Parameters Explained

- **`--max_step`**: Controls the number of "thinking" iterations. Each step emits 5 brush strokes, so 100 steps produce 500 total strokes.
- **`--divide`**: Splits the target image into a grid for higher-resolution results (optional). Use values like 4 or 5 to process large images in patches.
- **`--actor`**: Path to the ResNet-based policy network defined in [`baseline/DRL/actor.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline/DRL/actor.py).
- **`--renderer`**: Path to the fully convolutional neural renderer (FCN) defined in [`baseline/Renderer/model.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline/Renderer/model.py).

### Output Handling

The script creates an `output/` directory and writes a sequence of PNG files (`generated0.png`, `generated1.png`, …) showing the progressive painting process. These frames capture the canvas state after each thinking step, utilizing the `decode`, `large2small`, `small2large`, and `smooth` helper functions from the source code to manage stroke rendering and image transitions.

## Web-Based Deployment with [`predict.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/predict.py)

For production environments or web demos, use the Cog predictor wrapper in [`predict.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/predict.py). This script encapsulates the same inference pipeline as [`baseline/test.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline/test.py) but returns an animated GIF suitable for API responses.

### Running the Cog Predictor

Install Cog and run inference as a web service:

```bash
python3 -m cog predict -i image=test.png -i renderer=renderer.pkl -i actor=actor.pkl

```

The predictor class defined in [`predict.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/predict.py) handles model loading onto the appropriate device and executes the same agent-renderer loop found in the baseline test script. This approach is ideal for containerized deployments where you need to run inference or test the trained models via HTTP endpoints rather than local file system operations.

## Post-Processing: Creating Videos from Output Frames

The PNG sequence generated by [`baseline/test.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline/test.py) can be assembled into a video file to visualize the painting process. Use `ffmpeg` to convert the frame sequence:

```bash
ffmpeg -r 10 -f image2 -i output/generated%d.png -s 512x512 \
       -c:v libx264 -pix_fmt yuv420p video.mp4 -q:v 0 -q:a 0

```

This command reads all generated PNG files at 10 frames per second and encodes them into `video.mp4` with YUV420p pixel formatting for maximum compatibility.

## Summary

- **Two inference paths exist**: [`baseline/test.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline/test.py) for local batch processing and [`predict.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/predict.py) for Cog-based web services.
- **Required assets**: You must download `renderer.pkl` and `actor.pkl` before running either script.
- **Core implementation**: Both scripts rely on the neural renderer ([`baseline/Renderer/model.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline/Renderer/model.py)) and the actor network ([`baseline/DRL/actor.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline/DRL/actor.py)), using helper functions like `decode` and `smooth` to manage stroke generation.
- **Output format**: Command-line testing produces numbered PNG frames, while the Cog predictor returns an animated GIF.
- **Video conversion**: Use `ffmpeg` to compile PNG sequences into shareable MP4 files showing the progressive painting process.

## Frequently Asked Questions

### Where can I find the pretrained model weights for inference?

The official pretrained weights are hosted on Google Drive and linked in the README.md Testing section. You need two files: `renderer.pkl` (the FCN neural renderer) and `actor.pkl` (the ResNet policy network). Download them using the `wget` commands provided in the repository documentation or copy them manually to your working directory.

### What is the difference between `--max_step` and the total number of brush strokes?

The `--max_step` parameter controls the number of agent "thinking" iterations, not individual strokes. According to the implementation in [`baseline/test.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline/test.py), each thinking step generates 5 brush strokes. Therefore, setting `--max_step=100` produces 500 total strokes across the painting sequence, with the canvas state saved after each step.

### Can I run inference on high-resolution images?

Yes. Use the `--divide` argument in [`baseline/test.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline/test.py) to enable a multi-patch processing mode. This splits the target image into a grid (e.g., `--divide=4` creates a 4x4 grid), processes each patch independently using the `large2small` and `small2large` utilities, and reconstructs the full-resolution output. This approach allows the fixed-size neural renderer to handle images larger than its training resolution.

### How do I modify the inference code to use my own trained models?

Replace the paths passed to `--actor` and `--renderer` with your custom pickle files. Ensure your trained models maintain the same architecture as the baseline implementations: the renderer must follow the FCN structure in [`baseline/Renderer/model.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline/Renderer/model.py), and the actor must match the ResNet-based policy in [`baseline/DRL/actor.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline/DRL/actor.py). Both scripts accept explicit file paths, so no code modification is required if your weights use compatible formats.