# Observation Space in the Learning to Paint Environment: Structure and Implementation

> Explore the observation space in the Learning to Paint environment. Discover its 7-channel tensor structure combining canvas, target, and step counter for efficient image generation.

- Repository: [hzwer/iccv2019-learningtopaint](https://github.com/hzwer/iccv2019-learningtopaint)
- Tags: deep-dive
- Published: 2026-03-03

---

**The observation space in the Learning to Paint environment is a 7-channel tensor combining the current canvas (3 RGB channels), the ground-truth target image (3 RGB channels), and a single-channel step counter map, resulting in a shape of `(batch_size, 7, width, width)`.**

The `Paint` environment class in the hzwer/iccv2019-learningtopaint repository defines how reinforcement learning agents perceive the state of the painting task. Understanding the observation space structure is essential for implementing custom agents or debugging the training pipeline. This article examines the exact tensor composition, channel ordering, and source code implementation based on the actual repository files.

## Tensor Composition of the Observation Space

The observation space consists of three distinct components concatenated along the channel dimension:

- **Current Canvas**: 3 channels representing the RGB image the agent has painted so far
- **Ground-Truth Target**: 3 channels containing the reference image the agent attempts to reproduce  
- **Step Counter Map**: 1 channel filled with the current step number `T` acting as a temporal cue

These combine to form a tensor with **7 channels total**. The shape is declared explicitly in the constructor at lines 32–33 of [`baseline/env.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline/env.py):

```python
self.observation_space = (self.batch_size, width, width, 7)

```

The same definition exists in the model-free variant at [`baseline_modelfree/env.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/env.py) lines 32–33, ensuring consistency across both implementations.

## How the Observation Tensor Is Constructed

The `observation()` method assembles the tensor dynamically during each training step. In [`baseline/env.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline/env.py), the method concatenates the canvas, ground truth, and step counter using PyTorch operations:

```python
def observation(self):
    # canvas: batch_size × 3 × width × width

    # gt (target): batch_size × 3 × width × width  

    # T (step): batch_size × 1 × width × width

    T = torch.ones([self.batch_size, 1, width, width], dtype=torch.uint8) * self.stepnum
    return torch.cat((self.canvas, self.gt, T.to(device)), 1)  # → batch_size × 7 × width × width

```

The step counter map provides crucial temporal information to the agent, indicating how many strokes have been applied in the current episode. This allows the agent to adjust its strategy based on the remaining steps.

## Implementation Across Environment Variants

The observation space implementation is identical in both the model-based and model-free baselines:

- **[`baseline/env.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline/env.py)**: Defines the `Paint` class for the model-based approach with full differentiable rendering
- **[`baseline_modelfree/env.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline_modelfree/env.py)**: Provides the same environment structure for the model-free DRL implementation

Both files declare the same `observation_space` tuple and implement the identical `observation()` method signature. The canvas updates are handled through the `decode()` function in [`baseline/DRL/ddpg.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline/DRL/ddpg.py), which modifies the canvas state that subsequently appears in the next observation.

## Practical Code Examples

### Inspecting the Observation Shape

Initialize the environment and verify the tensor dimensions match the declared observation space:

```python
from baseline.env import Paint

# Initialize with batch size 4 and max 10 steps per episode

env = Paint(batch_size=4, max_step=10)
env.load_data()

# Reset environment in training mode

obs = env.reset(test=False)

print("Observation shape:", obs.shape)

# Output: torch.Size([4, 7, 128, 128])

```

### Extracting Individual Components

Decompose the 7-channel tensor into its constituent parts for visualization or analysis:

```python
import matplotlib.pyplot as plt

# Extract first sample from batch

sample = obs[0]  # Shape: [7, 128, 128]

# Decompose channels

canvas = sample[:3].permute(1, 2, 0).cpu().numpy()
gt = sample[3:6].permute(1, 2, 0).cpu().numpy()
step_map = sample[6].cpu().numpy()

# Display components

fig, axes = plt.subplots(1, 3, figsize=(12, 4))
axes[0].imshow(canvas)
axes[0].set_title('Current Canvas')
axes[1].imshow(gt)
axes[1].set_title('Ground Truth')
axes[2].imshow(step_map, cmap='viridis')
axes[2].set_title(f'Step Map (T={step_map[0,0]})')
plt.show()

```

## Summary

- The observation space in hzwer/iccv2019-learningtopaint is a **7-channel tensor** with shape `(batch_size, 7, width, width)`
- Channels 0–2 contain the **current canvas**, channels 3–5 contain the **ground-truth target**, and channel 6 contains the **step counter**
- The `observation()` method in [`baseline/env.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline/env.py) dynamically constructs this tensor by concatenating three separate tensors along the channel dimension
- Both model-based and model-free variants implement identical observation space structures for consistent agent training

## Frequently Asked Questions

### What is the exact tensor shape returned by `env.reset()`?

The `reset()` method returns a tensor of shape `(batch_size, 7, width, width)` where `batch_size` is specified during environment initialization (typically 64 or 128), and `width` is the image resolution (default 128). The 7 channels correspond to RGB canvas (3), RGB target (3), and the step counter (1).

### Why does the observation include a step counter map instead of a scalar?

The step counter is broadcast into a full `(width, width)` spatial map rather than provided as a single scalar value. This design choice allows convolutional neural network policies to easily incorporate temporal information through standard spatial convolutions, as the step number appears as a uniform channel that can be processed alongside image features.

### How does the canvas state update between observations?

The canvas updates occur through the `decode()` function in [`baseline/DRL/ddpg.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/baseline/DRL/ddpg.py), which renders brush strokes based on agent actions. After each step, the modified canvas tensor replaces `self.canvas` in the environment state, making the updated painting visible in the next observation returned by `env.observation()`.

### Is the observation space different in the model-free variant?

No, the observation space is identical between the model-based (`baseline/`) and model-free (`baseline_modelfree/`) implementations. Both use the same 7-channel structure defined in their respective [`env.py`](https://github.com/hzwer/iccv2019-learningtopaint/blob/main/env.py) files, ensuring that agents can be transferred between environments without modification to the observation processing logic.