Where Is the DDPG Agent Implementation in the Learning-to-Paint Repository?

The DDPG agent implementation resides in baseline_modelfree/DRL/ddpg.py (with a duplicate in baseline/DRL/ddpg.py), defining the complete DDPG class that manages actor-critic networks, experience replay, target network updates, and the training loop for the stroke-based painting system.

The hzwer/iccv2019-learningtopaint repository implements a Deep Reinforcement Learning agent that learns to paint images using continuous brush strokes. Understanding the DDPG agent implementation is essential for modifying training behavior, adjusting exploration strategies, or adapting the model to new rendering tasks. This guide identifies the exact file locations and breaks down the architectural components of the Deep Deterministic Policy Gradient agent as implemented in the source code.

Locating the Main DDPG Agent File

The primary DDPG agent implementation is located at:

• baseline_modelfree/DRL/ddpg.py

A duplicate of this file exists at baseline/DRL/ddpg.py for the non-model-free baseline variant. The DDPG class defined in these files encapsulates the entire agent logic, including network initialization, the replay buffer, GAN-based reward computation, and the core training algorithms.

Architectural Components of the DDPG Agent

The DDPG class composes several distinct components that work together to enable continuous control of brush parameters.

Actor Network

The actor is instantiated as ResNet(9, 18, 65) and defined in baseline_modelfree/DRL/actor.py. It accepts a 9-channel state tensor representing the current canvas (3 channels), target image (3 channels), step number mask (1 channel), and coordinate convolution channels (2 channels). The network outputs continuous brush parameters that define the stroke position, color, and shape.

Critic Network

The critic is defined in baseline_modelfree/DRL/critic.py as ResNet_wobn(9, 18, 1), indicating a ResNet architecture with weight-only batch normalization. It evaluates state-action pairs to produce a Q-value estimating the expected return.

Target Networks and Update Mechanisms

The agent maintains target network copies (actor_target and critic_target) for stable temporal-difference learning. Target updates are handled by helper functions in baseline_modelfree/utils/util.py:

• hard_update: Copies weights directly from source to target
• soft_update: Gradual blending using the tau parameter (default 0.001)

Experience Replay Buffer

The replay buffer is instantiated as rpm using the implementation in baseline_modelfree/DRL/rpm.py. This ring buffer stores transition tuples (s, a, r, s′, done) and supports batch sampling during policy updates. The default buffer size (rmsize) is set to 800 transitions.

GAN-Based Reward Calculation

The agent integrates adversarial reward signals through baseline_modelfree/DRL/wgan.py. The methods update and cal_reward compute a GAN-based reward that complements the standard L2 reconstruction loss, providing richer training signal for stroke quality.

Stroke Decoder

The fixed stroke decoder (FCN) is loaded from renderer.pkl and defined in baseline_modelfree/Renderer/model.py. The decode function converts the actor's continuous output into a rendered brush stroke that is applied to the canvas.

Core Methods in the DDPG Class

The DDPG class exposes several key methods that orchestrate the training and inference pipeline:

• play(state, target=False): Executes a forward pass through the actor network (or target actor if target=True). The input state has shape (B, 9, 128, 128).

• evaluate(state, action, target=False): Returns the Q-value from the critic (or target critic) and computes the GAN-based reward.

• update_policy(lr): Performs the core DDPG update step. Samples a batch from the replay buffer, computes TD-targets using target networks, updates the critic with MSE loss, then updates the actor by maximizing the expected Q-value. Finally applies soft target updates.

• select_action(state, return_fix=False, noise_factor=0): Returns deterministic actions from the actor. When noise_factor > 0, adds Gaussian exploration noise to encourage exploration.

• observe(reward, state, done, step): Stores the transition tuple in the replay buffer after each environment step.

• **load_weights(path) / save_model(path): Handles serialization of actor weights, critic weights, and GAN components.

• choose_device(): Automatically moves all network parameters to the appropriate device (CPU or GPU).

Working with the DDPG Agent: Code Examples

Instantiating the Agent

from baseline_modelfree.DRL.ddpg import DDPG

agent = DDPG(
    batch_size=64,
    env_batch=1,
    max_step=40,
    tau=0.001,
    discount=0.9,
    rmsize=800,
    writer=None,          # optional TensorBoard writer

    resume=None,          # path to checkpoint (if any)

    output_path="./ckpt"  # where checkpoints will be saved

)

Selecting Actions and Exploration

# state shape: (B, 9, 128, 128) - batch, 9 channels, 128x128 resolution

action = agent.select_action(state, noise_factor=0.1)  # Gaussian exploration noise

Training Loop Implementation

for episode in range(num_episodes):
    obs = env.reset()
    agent.reset(obs, factor=0.2)  # initialise noise level

    
    for step in range(agent.max_step):
        # 1) Choose action

        action = agent.select_action(obs)
        
        # 2) Apply action in environment

        next_obs, reward, done, info = env.step(action)
        
        # 3) Store transition in replay buffer

        agent.observe(reward, next_obs, done, step)
        
        # 4) Update policy every few steps

        if step % 5 == 0:
            lr = [1e-3, 1e-3]  # learning rates for critic & actor

            policy_loss, value_loss = agent.update_policy(lr)
        
        if done:
            break
    
    # Save checkpoint

    agent.save_model("./ckpt/episode_{}".format(episode))

Model Persistence

# Load pretrained weights

agent.load_weights("./ckpt/episode_10")

# Save current state

agent.save_model("./ckpt/final_model")

Related Files in the DDPG Pipeline

Path	Role
baseline_modelfree/DRL/ddpg.py	Main DDPG agent class definition
baseline_modelfree/DRL/actor.py	Actor network (ResNet) architecture
baseline_modelfree/DRL/critic.py	Critic network (ResNet_wobn) architecture
baseline_modelfree/DRL/rpm.py	Replay buffer implementation
baseline_modelfree/DRL/wgan.py	GAN components for reward calculation
baseline_modelfree/Renderer/model.py	FCN decoder for rendering strokes
baseline_modelfree/utils/util.py	Target update utilities (hard_update, soft_update)

Summary

• The DDPG agent implementation is located in baseline_modelfree/DRL/ddpg.py (primary) and baseline/DRL/ddpg.py (secondary).
• The DDPG class encapsulates actor (ResNet) and critic (ResNet_wobn) networks along with their target copies.
• Soft target updates use a tau of 0.001 via utilities in baseline_modelfree/utils/util.py.
• The replay buffer (rpm) stores transitions for off-policy learning with a default capacity of 800.
• GAN-based rewards are computed using baseline_modelfree/DRL/wgan.py to augment the L2 loss signal.
• The stroke decoder (FCN) converts network outputs into rendered brush strokes via baseline_modelfree/Renderer/model.py.

Frequently Asked Questions

Where exactly is the DDPG class defined in the Learning-to-Paint repository?

The DDPG class is defined in baseline_modelfree/DRL/ddpg.py. A functionally identical copy exists in baseline/DRL/ddpg.py for the alternative baseline implementation. Both files contain the complete agent logic including network definitions, replay buffer management, and the training loop.

What neural network architectures does the DDPG agent use?

According to the source code in baseline_modelfree/DRL/actor.py and baseline_modelfree/DRL/critic.py, the agent uses a ResNet(9, 18, 65) for the actor and a ResNet_wobn(9, 18, 1) for the critic. The "9" input channels correspond to the canvas (3), target image (3), step mask (1), and coordinate convolution (2) at 128×128 resolution.

How does the DDPG agent compute rewards during training?

The agent combines traditional L2 reconstruction loss with adversarial rewards from a WGAN. The cal_reward method in baseline_modelfree/DRL/wgan.py evaluates the quality of generated strokes, providing an additional training signal beyond pixel-wise differences.

What is the purpose of the replay buffer in this implementation?

The replay buffer, implemented as rpm in baseline_modelfree/DRL/rpm.py, stores transition tuples (state, action, reward, next_state, done) as a ring buffer. During update_policy, the agent samples batches from this buffer to perform stable off-policy updates with decorrelated data, which is essential for training continuous control agents with DDPG.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}