What Is the Role of the Critic Network in the DDPG Agent?

The critic network in the DDPG agent implements the Q-function (Q(s,a)) to evaluate state-action pairs, providing bootstrapped target values for stable training and gradient signals that guide the actor toward maximizing expected cumulative reward.

In the hzwer/iccv2019-learningtopaint repository, the Deep Deterministic Policy Gradient (DDPG) agent learns to synthesize paintings through iterative brush strokes. The critic network serves as the value estimator, predicting the expected return for taking a specific action (stroke parameters and color) in a given painting state (canvas configuration).

Core Function of the DDPG Critic Network

The DDPG architecture maintains two distinct neural networks: the actor, which proposes deterministic actions, and the critic, which judges the quality of those actions. In this implementation, the critic is instantiated as ResNet_wobn, a ResNet variant without batch normalization that outputs scalar Q-values.

The critic performs three critical functions during training:

1. Evaluates current state-action pairs using the online network
2. Computes target Q-values using a slowly-updated target network for stability
3. Provides gradients to the actor via backpropagation through the Q-function

According to the source code in baseline_modelfree/DRL/ddpg.py, the critic initialization and target network setup occur at lines 51-55:

self.critic = ResNet_wobn(9, 18, 1)          # Online critic: 9 input channels, depth 18

self.critic_target = ResNet_wobn(9, 18, 1)   # Target critic for stable bootstrapping

hard_update(self.critic_target, self.critic) # Copy weights initially

ResNet_wobn Architecture and Implementation

The critic network (ResNet_wobn) processes a 9-channel merged environment state consisting of the current canvas, target image, normalized step counter, and coordinate maps. It concatenates this state with the action embedding and outputs a single scalar Q-value per batch element.

The architecture is defined in baseline_modelfree/DRL/critic.py (lines 94-142):

class ResNet_wobn(nn.Module):
    def __init__(self, num_inputs, depth, num_outputs):
        super(ResNet_wobn, self).__init__()
        # ResNet layers without batch normalization

        self.conv0 = conv3x3(num_inputs, 32, 2)
        self.layer1 = self._make_layer(block, 64, num_blocks[0], stride=2)
        self.layer2 = self._make_layer(block, 128, num_blocks[1], stride=2)
        self.layer3 = self._make_layer(block, 256, num_blocks[2], stride=2)
        self.layer4 = self._make_layer(block, 512, num_blocks[3], stride=2)
        # Final 1x1 convolution outputs single channel Q-value

        self.conv4 = weightNorm(nn.Conv2d(512, 1, 1, 1, 0))
        
    def forward(self, input):
        x, a = input                     # x = merged state (B,9,128,128), a = action

        a = self.a2img(a)                # Embed action as image-like feature map

        x = self.relu_1(self.conv0(x))
        x = torch.cat([x, a], 1)         # Fuse state and action features

        x = self.layer1(x)
        x = self.layer2(x) 
        x = self.layer3(x)
        x = self.layer4(x)
        x = self.conv4(x)
        return x.view(x.size(0), 64)     # Reshape to scalar Q per sample: (B, 64) -> effectively scalar per batch item

The network receives the merged state created by concatenating:

• Current canvas (3 channels)
• Target ground truth image (3 channels)
• Step counter normalized by max steps (1 channel)
• Coordinate convolution features (2 channels)

How the Critic Drives Training in DDPG

The critic enables learning through a temporal difference (TD) learning scheme involving both online and target networks.

Target Network Initialization and Soft Updates

To prevent training instability from oscillating target values, the implementation maintains a target critic that slowly tracks the online network. In ddpg.py lines 51-53, the target network is updated with a soft update coefficient (\tau = 0.001):

soft_update(self.critic_target, self.critic, self.tau)  # τ = 0.001

This ensures that target Q-values change gradually, providing stable learning targets for the value function approximation.

Computing Target Q-Values

During the learning step, the target critic estimates the value of the next state-action pair. The implementation in ddpg.py (lines 30-34) computes this as:

next_action = self.play(next_state, True)                    # Target policy action

target_q, _ = self.evaluate(next_state, next_action, True)   # Use target critic (target=True)

The evaluate method constructs the merged state tensor and passes it through the specified critic network (online or target).

Value Loss and Backpropagation

The online critic evaluates the current state-action pair to produce cur_q, which is compared against the TD target to form the value loss. This occurs in ddpg.py lines 35-42:

cur_q, step_reward = self.evaluate(state, action)        # Online Q-value

target_q = self.discount * (1 - terminal.float()).view(-1,1) * target_q
value_loss = criterion(cur_q, target_q)                  # Typically MSE loss

self.critic.zero_grad()
value_loss.backward()                                    # Compute gradients

self.critic_optim.step()                                 # Update critic weights

The critic's gradients flow back through the network, updating its weights to better approximate the true action-value function.

Code Walkthrough: Critic Updates in Practice

The complete learning step demonstrates how the critic network orchestrates the DDPG update loop. The agent samples transitions from replay memory and uses both critic networks to compute the TD error:

# Evaluate current state with online critic

cur_q, _ = self.evaluate(state, action)

# Compute target Q-value using target critic and target policy

next_action = self.play(next_state, True)
target_q, _ = self.evaluate(next_state, next_action, target=True)

# Apply discount factor (0.99) and handle terminal states

target_q = self.discount * (1 - terminal.float()).view(-1, 1) * target_q

# Compute temporal difference loss

value_loss = criterion(cur_q, target_q)  # Mean squared error

# Update online critic

self.critic.zero_grad()
value_loss.backward()
self.critic_optim.step()

# Soft update target critic: θ_target = τ*θ + (1-τ)*θ_target

soft_update(self.critic_target, self.critic, self.tau)

This pattern ensures that the critic learns to accurately estimate painting quality while providing reliable target values for the actor's policy gradient updates.

Summary

• The critic network (ResNet_wobn) implements the Q-function (Q(s,a)) to evaluate painting actions, receiving a 9-channel merged state and outputting scalar value estimates.
• Target network stabilization uses a slowly-updated copy of the critic (soft update (\tau = 0.001)) to provide consistent bootstrapped targets and prevent training divergence.
• Value loss computation compares online critic predictions against TD targets using MSE loss, with gradients backpropagated through baseline_modelfree/DRL/ddpg.py lines 41-42.
• Actor guidance relies on the critic's gradients to inform policy updates, though the actor optimization is driven by maximizing the critic's Q-values with respect to action parameters.

Frequently Asked Questions

What is the difference between the critic and actor in DDPG?

The actor network implements the policy (\mu(s)), deterministically mapping painting states to continuous stroke actions (position, color, shape). The critic network implements the value function (Q(s,a)), estimating the expected cumulative reward for taking a specific action in a given state. While the actor proposes what to paint, the critic evaluates how good that painting decision is.

Why does the DDPG critic use a target network?

The target critic provides stable bootstrapped targets for temporal difference learning. Without it, the learning target would change every time the critic updates, causing training instability similar to chasing a moving target. As implemented in ddpg.py, the target network updates slowly via soft updates ((\tau = 0.001)), ensuring that target Q-values evolve gradually while the online network learns from consistent objectives.

What input does the critic network receive in LearningToPaint?

The critic receives a 9-channel merged tensor constructed in the evaluate method of ddpg.py. This includes the current canvas (3 channels), target image (3 channels), normalized step counter indicating progress through the episode, and coordinate convolution features (2 channels). The action vector is embedded as an additional image-like feature map and concatenated to the state representation before processing through the ResNet layers.

How is the critic loss computed in this implementation?

The critic loss is the mean squared error between the online critic's prediction (cur_q) and the temporal difference target. The target combines immediate reward with the discounted Q-value of the next state-action pair estimated by the target critic. Specifically, value_loss = criterion(cur_q, target_q) where target_q incorporates the discount factor (0.99) and terminal state masking, as shown in baseline_modelfree/DRL/ddpg.py lines 36-40.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}