How the ResNet-18 Actor Network Functions in the DDPG Agent for Neural Painting

The ResNet-18 actor network in this DDPG agent processes 9-channel visual state tensors through deep residual convolutional layers to output 65 continuous parameters representing five brush strokes, enabling end-to-end reinforcement learning of painting policies.

The hzwer/iccv2019-learningtopaint repository implements a Deep Deterministic Policy Gradient (DDPG) agent that employs a custom ResNet-18 architecture as its actor network to map high-dimensional visual states to continuous brush-stroke actions. This ResNet-18 actor network functions as the policy backbone, transforming 128×128 pixel inputs into precise painting parameters through residual learning blocks optimized for the neural painting task.

Input State Representation: What the Actor Receives

In baseline_modelfree/DRL/ddpg.py, the DDPG.play method constructs a rich 9-channel state tensor that serves as the actor's input. The raw environment state is concatenated into a tensor of shape (batch, 9, 128, 128):

state = torch.cat(
    (state[:, :6].float() / 255,                # target (3) + canvas (3)

     state[:, 6:7].float() / self.max_step,    # normalized step number (1)

     coord.expand(state.shape[0], 2, 128, 128) # two coordinate-conv channels (2)

    , 1)

This input comprises three color channels for the target image, three for the current canvas, one channel representing the normalized step number, and two fixed coordinate map channels. The actor instantiation in the DDPG constructor reflects this input dimensionality:

self.actor = ResNet(9, 18, 65)   # [inputs, depth, outputs]

Architecture of the ResNet-18 Actor Network

The actor implementation resides in baseline_modelfree/DRL/actor.py, where the ResNet class defines a standard ResNet-18 backbone modified for continuous control. The architecture processes visual inputs through the following components:

Stem Layer. The network begins with conv1 = conv3x3(num_inputs, 64, stride=2) followed by BatchNorm and ReLU, downsampling the 128×128 input while expanding to 64 feature channels.

Residual Blocks. Four sequential layers (layer1 through layer4) are constructed using _make_layer with the ResNet-18 block configuration (BasicBlock). Each layer applies stride-2 convolutions, progressively halving spatial resolution while increasing channel depth through the standard 64→128→256→512 progression.

Output Head. Global average pooling (F.avg_pool2d(x, 4)) compresses the final 512 feature maps into a 512-dimensional vector. A fully-connected layer (self.fc = nn.Linear(512, num_outputs)) projects this to 65 outputs, followed by a sigmoid activation (torch.sigmoid) that constrains values to [0, 1] to match the valid stroke-parameter range.

The complete forward pass executes as: stem convolution → batch normalization → ReLU → four residual blocks → average pooling → flatten → fully-connected layer → sigmoid.

Integration into the DDPG Training Loop

The ResNet-18 actor integrates into the DDPG framework through several key mechanisms defined in baseline_modelfree/DRL/ddpg.py:

Policy Evaluation. During inference, self.actor(state) generates deterministic actions given the current visual state. The select_action method queries the network and optionally adds exploration noise via the noise_factor parameter.

Target Network Stabilization. For stable temporal-difference learning, the DDPG agent maintains a target actor (self.actor_target) with identical ResNet-18 architecture. The play method accepts a target boolean flag to switch between the online and target networks when computing Q-learning targets.

Policy Gradient Updates. During training, the critic network evaluates the actor's proposed actions. The gradient of -Q.mean() is back-propagated through the actor (policy_loss.backward()), and the Adam optimizer updates the ResNet-18 weights to maximize expected return.

Action Decoding. The raw 65-dimensional output requires interpretation by the decode function in ddpg.py, which reshapes the vector into five distinct brush strokes. Each stroke consists of 10 shape parameters and 3 color values (5 × 13 = 65), defining the brush location, size, rotation, and RGB color applied to the canvas.

Output Interpretation: From 65-Dim Vector to Brush Strokes

The actor's output dimensionality of 65 corresponds to a fixed set of five brush strokes per decision step. As implemented in the decoding logic, these parameters partition into:

• 10 shape parameters per stroke: controlling position, size, rotation, and brush curvature
• 3 color parameters per stroke: RGB values for the brush color

This design choice enables the agent to paint multi-stroke compositions in parallel while maintaining a compact, continuous action space suitable for DDPG's deterministic policy gradient method.

Summary

• The ResNet-18 actor in baseline_modelfree/DRL/actor.py accepts 9-channel visual inputs (target, canvas, step number, and coordinate maps) of shape (batch, 9, 128, 128).
• The architecture follows standard ResNet-18 design with four residual layers, global average pooling, and a fully-connected head outputting 65 continuous values constrained to [0, 1] via sigmoid.
• In baseline_modelfree/DRL/ddpg.py, the actor is instantiated as ResNet(9, 18, 65) and used for both online policy evaluation and target network calculations during DDPG updates.
• The 65-dimensional output decodes into five brush strokes (10 shape + 3 color parameters each), which are rendered onto the canvas to generate the next painting state.

Frequently Asked Questions

Why does the actor use 9 input channels instead of a standard RGB image?

The 9-channel representation encodes the complete painting state necessary for sequential decision-making. According to the source code in baseline_modelfree/DRL/ddpg.py, the tensor concatenates the 3-channel target image, 3-channel current canvas, 1-channel normalized timestep, and 2-channel coordinate maps. This provides the ResNet-18 actor with spatial awareness, progress indication, and color reference required to determine optimal brush strokes.

What is the purpose of the sigmoid activation on the actor output?

The final torch.sigmoid layer in baseline_modelfree/DRL/actor.py constrains all 65 output parameters to the range [0, 1]. This normalization ensures that decoded stroke parameters (positions, sizes, colors) remain within valid bounds for the painting renderer, preventing out-of-range values that could produce invalid brush configurations or rendering artifacts.

How does the target actor network differ from the online actor?

Both networks share identical ResNet-18 architectures, but the target actor maintains separate parameters that are softly updated (polyak averaging) rather than trained directly. As implemented in the DDPG agent, the target network computes next-state action predictions for stable Q-target calculation, reducing moving-target problems during temporal-difference learning while the online actor receives policy gradient updates.

Why was ResNet-18 chosen over deeper architectures like ResNet-50?

ResNet-18 provides sufficient representational capacity to map 128×128 visual states to 65 continuous parameters while maintaining computational efficiency for reinforcement learning. The shallow depth (18 layers versus 50+) enables faster forward and backward passes during the thousands of environment steps required for DDPG training, balancing model capacity with training throughput for the neural painting task.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}