# What Is the Role of the Critic Network in the DDPG Agent?

> Understand the critic network's role in DDPG agents. It evaluates state-action pairs with the Q-function for stable training and guides the actor to maximize reward.

- Repository: [hzwer/iccv2019-learningtopaint](https://github.com/hzwer/iccv2019-learningtopaint)
- Tags: deep-dive
- Published: 2026-03-03

---

**The critic network in the DDPG agent implements the Q-function \(Q(s,a)\) to evaluate state-action pairs, providing bootstrapped target values for stable training and gradient signals that guide the actor toward maximizing expected cumulative reward.**

In the `hzwer/iccv2019-learningtopaint` repository, the Deep Deterministic Policy Gradient (DDPG) agent learns to synthesize paintings through iterative brush strokes. The critic network serves as the value estimator, predicting the expected return for taking a specific action (stroke parameters and color) in a given painting state (canvas configuration).

## Core Function of the DDPG Critic Network

The DDPG architecture maintains two distinct neural networks: the **actor**, which proposes deterministic actions, and the **critic**, which judges the quality of those actions. In this implementation, the critic is instantiated as `ResNet_wobn`, a ResNet variant without batch normalization that outputs scalar Q-values.

The critic performs three critical functions during training:

1. **Evaluates current state-action pairs** using the online network
2. **Computes target Q-values** using a slowly-updated target network for stability
3. **Provides gradients** to the actor via backpropagation through the Q-function

According to the source code in [`baseline_modelfree/DRL/ddpg.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/DRL/ddpg.py), the critic initialization and target network setup occur at lines 51-55:

```python
self.critic = ResNet_wobn(9, 18, 1)          # Online critic: 9 input channels, depth 18

self.critic_target = ResNet_wobn(9, 18, 1)   # Target critic for stable bootstrapping

hard_update(self.critic_target, self.critic) # Copy weights initially

```

## ResNet_wobn Architecture and Implementation

The critic network (`ResNet_wobn`) processes a 9-channel merged environment state consisting of the current canvas, target image, normalized step counter, and coordinate maps. It concatenates this state with the action embedding and outputs a single scalar Q-value per batch element.

The architecture is defined in [`baseline_modelfree/DRL/critic.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/DRL/critic.py) (lines 94-142):

```python
class ResNet_wobn(nn.Module):
    def __init__(self, num_inputs, depth, num_outputs):
        super(ResNet_wobn, self).__init__()
        # ResNet layers without batch normalization

        self.conv0 = conv3x3(num_inputs, 32, 2)
        self.layer1 = self._make_layer(block, 64, num_blocks[0], stride=2)
        self.layer2 = self._make_layer(block, 128, num_blocks[1], stride=2)
        self.layer3 = self._make_layer(block, 256, num_blocks[2], stride=2)
        self.layer4 = self._make_layer(block, 512, num_blocks[3], stride=2)
        # Final 1x1 convolution outputs single channel Q-value

        self.conv4 = weightNorm(nn.Conv2d(512, 1, 1, 1, 0))
        
    def forward(self, input):
        x, a = input                     # x = merged state (B,9,128,128), a = action

        a = self.a2img(a)                # Embed action as image-like feature map

        x = self.relu_1(self.conv0(x))
        x = torch.cat([x, a], 1)         # Fuse state and action features

        x = self.layer1(x)
        x = self.layer2(x) 
        x = self.layer3(x)
        x = self.layer4(x)
        x = self.conv4(x)
        return x.view(x.size(0), 64)     # Reshape to scalar Q per sample: (B, 64) -> effectively scalar per batch item

```

The network receives the **merged state** created by concatenating:
- Current canvas (3 channels)
- Target ground truth image (3 channels) 
- Step counter normalized by max steps (1 channel)
- Coordinate convolution features (2 channels)

## How the Critic Drives Training in DDPG

The critic enables learning through a temporal difference (TD) learning scheme involving both online and target networks.

### Target Network Initialization and Soft Updates

To prevent training instability from oscillating target values, the implementation maintains a target critic that slowly tracks the online network. In [`ddpg.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/ddpg.py) lines 51-53, the target network is updated with a soft update coefficient \(\tau = 0.001\):

```python
soft_update(self.critic_target, self.critic, self.tau)  # τ = 0.001

```

This ensures that target Q-values change gradually, providing stable learning targets for the value function approximation.

### Computing Target Q-Values

During the learning step, the target critic estimates the value of the next state-action pair. The implementation in [`ddpg.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/ddpg.py) (lines 30-34) computes this as:

```python
next_action = self.play(next_state, True)                    # Target policy action

target_q, _ = self.evaluate(next_state, next_action, True)   # Use target critic (target=True)

```

The `evaluate` method constructs the merged state tensor and passes it through the specified critic network (online or target).

### Value Loss and Backpropagation

The online critic evaluates the current state-action pair to produce `cur_q`, which is compared against the TD target to form the value loss. This occurs in [`ddpg.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/ddpg.py) lines 35-42:

```python
cur_q, step_reward = self.evaluate(state, action)        # Online Q-value

target_q = self.discount * (1 - terminal.float()).view(-1,1) * target_q
value_loss = criterion(cur_q, target_q)                  # Typically MSE loss

self.critic.zero_grad()
value_loss.backward()                                    # Compute gradients

self.critic_optim.step()                                 # Update critic weights

```

The critic's gradients flow back through the network, updating its weights to better approximate the true action-value function.

## Code Walkthrough: Critic Updates in Practice

The complete learning step demonstrates how the critic network orchestrates the DDPG update loop. The agent samples transitions from replay memory and uses both critic networks to compute the TD error:

```python

# Evaluate current state with online critic

cur_q, _ = self.evaluate(state, action)

# Compute target Q-value using target critic and target policy

next_action = self.play(next_state, True)
target_q, _ = self.evaluate(next_state, next_action, target=True)

# Apply discount factor (0.99) and handle terminal states

target_q = self.discount * (1 - terminal.float()).view(-1, 1) * target_q

# Compute temporal difference loss

value_loss = criterion(cur_q, target_q)  # Mean squared error

# Update online critic

self.critic.zero_grad()
value_loss.backward()
self.critic_optim.step()

# Soft update target critic: θ_target = τ*θ + (1-τ)*θ_target

soft_update(self.critic_target, self.critic, self.tau)

```

This pattern ensures that the critic learns to accurately estimate painting quality while providing reliable target values for the actor's policy gradient updates.

## Summary

- The **critic network** (`ResNet_wobn`) implements the Q-function \(Q(s,a)\) to evaluate painting actions, receiving a 9-channel merged state and outputting scalar value estimates.
- **Target network stabilization** uses a slowly-updated copy of the critic (soft update \(\tau = 0.001\)) to provide consistent bootstrapped targets and prevent training divergence.
- **Value loss computation** compares online critic predictions against TD targets using MSE loss, with gradients backpropagated through [`baseline_modelfree/DRL/ddpg.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/DRL/ddpg.py) lines 41-42.
- **Actor guidance** relies on the critic's gradients to inform policy updates, though the actor optimization is driven by maximizing the critic's Q-values with respect to action parameters.

## Frequently Asked Questions

### What is the difference between the critic and actor in DDPG?

The **actor** network implements the policy \(\mu(s)\), deterministically mapping painting states to continuous stroke actions (position, color, shape). The **critic** network implements the value function \(Q(s,a)\), estimating the expected cumulative reward for taking a specific action in a given state. While the actor proposes what to paint, the critic evaluates how good that painting decision is.

### Why does the DDPG critic use a target network?

The target critic provides **stable bootstrapped targets** for temporal difference learning. Without it, the learning target would change every time the critic updates, causing training instability similar to chasing a moving target. As implemented in [`ddpg.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/ddpg.py), the target network updates slowly via soft updates (\(\tau = 0.001\)), ensuring that target Q-values evolve gradually while the online network learns from consistent objectives.

### What input does the critic network receive in LearningToPaint?

The critic receives a **9-channel merged tensor** constructed in the `evaluate` method of [`ddpg.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/ddpg.py). This includes the current canvas (3 channels), target image (3 channels), normalized step counter indicating progress through the episode, and coordinate convolution features (2 channels). The action vector is embedded as an additional image-like feature map and concatenated to the state representation before processing through the ResNet layers.

### How is the critic loss computed in this implementation?

The critic loss is the **mean squared error** between the online critic's prediction (`cur_q`) and the temporal difference target. The target combines immediate reward with the discounted Q-value of the next state-action pair estimated by the target critic. Specifically, `value_loss = criterion(cur_q, target_q)` where `target_q` incorporates the discount factor (0.99) and terminal state masking, as shown in [`baseline_modelfree/DRL/ddpg.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/DRL/ddpg.py) lines 36-40.