# Where Is the DDPG Agent Implementation in the Learning-to-Paint Repository?

> Find the DDPG agent implementation in hzwer/iccv2019-learningtopaint at DRL/ddpg.py. Discover actor-critic networks, experience replay, and training loop details for stroke-based painting.

- Repository: [hzwer/iccv2019-learningtopaint](https://github.com/hzwer/iccv2019-learningtopaint)
- Tags: how-to-guide
- Published: 2026-03-03

---

**The DDPG agent implementation resides in [`baseline_modelfree/DRL/ddpg.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/DRL/ddpg.py) (with a duplicate in [`baseline/DRL/ddpg.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline/DRL/ddpg.py)), defining the complete `DDPG` class that manages actor-critic networks, experience replay, target network updates, and the training loop for the stroke-based painting system.**

The `hzwer/iccv2019-learningtopaint` repository implements a Deep Reinforcement Learning agent that learns to paint images using continuous brush strokes. Understanding the DDPG agent implementation is essential for modifying training behavior, adjusting exploration strategies, or adapting the model to new rendering tasks. This guide identifies the exact file locations and breaks down the architectural components of the Deep Deterministic Policy Gradient agent as implemented in the source code.

## Locating the Main DDPG Agent File

The primary DDPG agent implementation is located at:

- **[`baseline_modelfree/DRL/ddpg.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/DRL/ddpg.py)**

A duplicate of this file exists at [`baseline/DRL/ddpg.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline/DRL/ddpg.py) for the non-model-free baseline variant. The `DDPG` class defined in these files encapsulates the entire agent logic, including network initialization, the replay buffer, GAN-based reward computation, and the core training algorithms.

## Architectural Components of the DDPG Agent

The `DDPG` class composes several distinct components that work together to enable continuous control of brush parameters.

### Actor Network

The actor is instantiated as `ResNet(9, 18, 65)` and defined in [`baseline_modelfree/DRL/actor.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/DRL/actor.py). It accepts a 9-channel state tensor representing the current canvas (3 channels), target image (3 channels), step number mask (1 channel), and coordinate convolution channels (2 channels). The network outputs continuous brush parameters that define the stroke position, color, and shape.

### Critic Network

The critic is defined in [`baseline_modelfree/DRL/critic.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/DRL/critic.py) as `ResNet_wobn(9, 18, 1)`, indicating a ResNet architecture with weight-only batch normalization. It evaluates state-action pairs to produce a Q-value estimating the expected return.

### Target Networks and Update Mechanisms

The agent maintains target network copies (`actor_target` and `critic_target`) for stable temporal-difference learning. Target updates are handled by helper functions in [`baseline_modelfree/utils/util.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/utils/util.py):

- **`hard_update`**: Copies weights directly from source to target
- **`soft_update`**: Gradual blending using the `tau` parameter (default 0.001)

### Experience Replay Buffer

The replay buffer is instantiated as `rpm` using the implementation in [`baseline_modelfree/DRL/rpm.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/DRL/rpm.py). This ring buffer stores transition tuples `(s, a, r, s′, done)` and supports batch sampling during policy updates. The default buffer size (`rmsize`) is set to 800 transitions.

### GAN-Based Reward Calculation

The agent integrates adversarial reward signals through [`baseline_modelfree/DRL/wgan.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/DRL/wgan.py). The methods `update` and `cal_reward` compute a GAN-based reward that complements the standard L2 reconstruction loss, providing richer training signal for stroke quality.

### Stroke Decoder

The fixed stroke decoder (`FCN`) is loaded from `renderer.pkl` and defined in [`baseline_modelfree/Renderer/model.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/Renderer/model.py). The `decode` function converts the actor's continuous output into a rendered brush stroke that is applied to the canvas.

## Core Methods in the DDPG Class

The `DDPG` class exposes several key methods that orchestrate the training and inference pipeline:

- **`play(state, target=False)`**: Executes a forward pass through the actor network (or target actor if `target=True`). The input state has shape `(B, 9, 128, 128)`.

- **`evaluate(state, action, target=False)`**: Returns the Q-value from the critic (or target critic) and computes the GAN-based reward.

- **`update_policy(lr)`**: Performs the core DDPG update step. Samples a batch from the replay buffer, computes TD-targets using target networks, updates the critic with MSE loss, then updates the actor by maximizing the expected Q-value. Finally applies soft target updates.

- **`select_action(state, return_fix=False, noise_factor=0)`**: Returns deterministic actions from the actor. When `noise_factor > 0`, adds Gaussian exploration noise to encourage exploration.

- **`observe(reward, state, done, step)`**: Stores the transition tuple in the replay buffer after each environment step.

- **`load_weights(path)` / **`save_model(path)`**: Handles serialization of actor weights, critic weights, and GAN components.

- **`choose_device()`**: Automatically moves all network parameters to the appropriate device (CPU or GPU).

## Working with the DDPG Agent: Code Examples

### Instantiating the Agent

```python
from baseline_modelfree.DRL.ddpg import DDPG

agent = DDPG(
    batch_size=64,
    env_batch=1,
    max_step=40,
    tau=0.001,
    discount=0.9,
    rmsize=800,
    writer=None,          # optional TensorBoard writer

    resume=None,          # path to checkpoint (if any)

    output_path="./ckpt"  # where checkpoints will be saved

)

```

### Selecting Actions and Exploration

```python

# state shape: (B, 9, 128, 128) - batch, 9 channels, 128x128 resolution

action = agent.select_action(state, noise_factor=0.1)  # Gaussian exploration noise

```

### Training Loop Implementation

```python
for episode in range(num_episodes):
    obs = env.reset()
    agent.reset(obs, factor=0.2)  # initialise noise level

    
    for step in range(agent.max_step):
        # 1) Choose action

        action = agent.select_action(obs)
        
        # 2) Apply action in environment

        next_obs, reward, done, info = env.step(action)
        
        # 3) Store transition in replay buffer

        agent.observe(reward, next_obs, done, step)
        
        # 4) Update policy every few steps

        if step % 5 == 0:
            lr = [1e-3, 1e-3]  # learning rates for critic & actor

            policy_loss, value_loss = agent.update_policy(lr)
        
        if done:
            break
    
    # Save checkpoint

    agent.save_model("./ckpt/episode_{}".format(episode))

```

### Model Persistence

```python

# Load pretrained weights

agent.load_weights("./ckpt/episode_10")

# Save current state

agent.save_model("./ckpt/final_model")

```

## Related Files in the DDPG Pipeline

| Path | Role |
|------|------|
| **[`baseline_modelfree/DRL/ddpg.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/DRL/ddpg.py)** | Main DDPG agent class definition |
| **[`baseline_modelfree/DRL/actor.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/DRL/actor.py)** | Actor network (`ResNet`) architecture |
| **[`baseline_modelfree/DRL/critic.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/DRL/critic.py)** | Critic network (`ResNet_wobn`) architecture |
| **[`baseline_modelfree/DRL/rpm.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/DRL/rpm.py)** | Replay buffer implementation |
| **[`baseline_modelfree/DRL/wgan.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/DRL/wgan.py)** | GAN components for reward calculation |
| **[`baseline_modelfree/Renderer/model.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/Renderer/model.py)** | `FCN` decoder for rendering strokes |
| **[`baseline_modelfree/utils/util.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/utils/util.py)** | Target update utilities (`hard_update`, `soft_update`) |

## Summary

- The **DDPG agent implementation** is located in [`baseline_modelfree/DRL/ddpg.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/DRL/ddpg.py) (primary) and [`baseline/DRL/ddpg.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline/DRL/ddpg.py) (secondary).
- The `DDPG` class encapsulates **actor** (`ResNet`) and **critic** (`ResNet_wobn`) networks along with their target copies.
- **Soft target updates** use a `tau` of 0.001 via utilities in [`baseline_modelfree/utils/util.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/utils/util.py).
- The **replay buffer** (`rpm`) stores transitions for off-policy learning with a default capacity of 800.
- **GAN-based rewards** are computed using [`baseline_modelfree/DRL/wgan.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/DRL/wgan.py) to augment the L2 loss signal.
- The **stroke decoder** (`FCN`) converts network outputs into rendered brush strokes via [`baseline_modelfree/Renderer/model.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/Renderer/model.py).

## Frequently Asked Questions

### Where exactly is the DDPG class defined in the Learning-to-Paint repository?

The `DDPG` class is defined in [`baseline_modelfree/DRL/ddpg.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/DRL/ddpg.py). A functionally identical copy exists in [`baseline/DRL/ddpg.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline/DRL/ddpg.py) for the alternative baseline implementation. Both files contain the complete agent logic including network definitions, replay buffer management, and the training loop.

### What neural network architectures does the DDPG agent use?

According to the source code in [`baseline_modelfree/DRL/actor.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/DRL/actor.py) and [`baseline_modelfree/DRL/critic.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/DRL/critic.py), the agent uses a `ResNet(9, 18, 65)` for the actor and a `ResNet_wobn(9, 18, 1)` for the critic. The "9" input channels correspond to the canvas (3), target image (3), step mask (1), and coordinate convolution (2) at 128×128 resolution.

### How does the DDPG agent compute rewards during training?

The agent combines traditional L2 reconstruction loss with adversarial rewards from a WGAN. The `cal_reward` method in [`baseline_modelfree/DRL/wgan.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/DRL/wgan.py) evaluates the quality of generated strokes, providing an additional training signal beyond pixel-wise differences.

### What is the purpose of the replay buffer in this implementation?

The replay buffer, implemented as `rpm` in [`baseline_modelfree/DRL/rpm.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/DRL/rpm.py), stores transition tuples `(state, action, reward, next_state, done)` as a ring buffer. During `update_policy`, the agent samples batches from this buffer to perform stable off-policy updates with decorrelated data, which is essential for training continuous control agents with DDPG.