Key Command-Line Arguments for Training the Paint Agent in LearningToPaint

The training script baseline/train.py exposes 16 hyper-parameters—ranging from replay buffer capacity (--rmsize) to total training steps (--train_times)—that control the DDPG reinforcement learning loop, vectorized environment interaction, and checkpoint management.

The hzwer/iccv2019-learningtopaint repository implements a reinforcement learning agent that learns to paint stroke by stroke using Deep Deterministic Policy Gradient (DDPG). To reproduce the ICCV 2019 results or fine-tune the model, you must configure the training pipeline through command-line arguments defined in the main entry point. These parameters govern everything from exploration noise to validation frequency.

Core Training Hyperparameters

The DDPG algorithm's behavior is governed by several key arguments parsed at lines 78–98 of baseline/train.py.

• --warmup (default: 400): Number of initial time-steps used to pre-fill the replay buffer before any gradient updates occur. This ensures the agent learns from diverse states rather than empty memories.
• --discount (default: 0.95⁵ ≈ 0.774): The discount factor γ applied to future rewards in the Bellman equation, controlling how far ahead the agent looks when calculating returns.
• --batch_size (default: 96): Number of transitions sampled from replay memory for each parameter update step.
• --rmsize (default: 800): Maximum capacity of the replay memory buffer, determining how many past experiences (state-action-reward tuples) are retained for sampling.
• --env_batch (default: 96): Number of parallel painting environments running simultaneously via the fastenv wrapper in DRL/multi.py. Higher values increase sample throughput but require more GPU memory.
• --tau (default: 0.001): Soft update coefficient for target network Polyak averaging. This controls how slowly the target actor and critic networks track the learned parameters.
• --max_step (default: 40): Maximum episode length in brush strokes before forced termination. Each step represents one painting action on the canvas.
• --noise_factor (default: 0): Scale of parameter-space noise added to the policy network for exploration, implemented in the DDPG class (DRL/ddpg.py).

Validation and Logging Configuration

Monitoring training progress requires periodic evaluation and tensorboard logging, managed by DRL/evaluator.py and utils/tensorboard.py.

• --train_times (default: 2,000,000): Total number of training steps (environment interactions) to execute before termination.
• --episode_train_times (default: 10): Number of policy-gradient updates performed immediately after each episode completes.
• --validate_interval (default: 50): Frequency (in episodes) at which the Evaluator runs a validation pass to compute mean reward and distance metrics.
• --validate_episodes (default: 5): Number of episodes executed during each validation phase to average performance statistics.
• --debug (default: False): Enables verbose logging of intermediate statistics during training iterations.

Checkpointing and Reproducibility

Controlling random seeds and model persistence ensures experiments remain reproducible and recoverable.

• --resume (default: None): Path to a checkpoint file (e.g., ./model/paint_latest.pth) to load before training begins, enabling fine-tuning or crash recovery.
• --output (default: ./model): Directory where trained model checkpoints and intermediate saves are written.
• --seed (default: 1234): Random seed propagated to NumPy, PyTorch, and Python’s random module to ensure deterministic behavior across runs.

Usage Examples

Start basic training with default hyper-parameters suitable for the paper's experimental setup:

python baseline/train.py

Increase the replay buffer capacity, extend warmup steps, and enable debug logging for extended training runs:

python baseline/train.py \
    --rmsize 2000 \
    --warmup 1000 \
    --train_times 5000000 \
    --output ./checkpoints \
    --debug

Resume training from a previous checkpoint while adjusting the remaining training steps:

python baseline/train.py \
    --resume ./checkpoints/paint_latest.pth \
    --output ./checkpoints \
    --train_times 3000000

Summary

• The 16 command-line arguments in baseline/train.py (lines 78–98) configure the DDPG agent's learning dynamics, environment parallelism, and I/O behavior.
• Replay memory size (--rmsize) and vectorized environments (--env_batch) directly impact sample efficiency and GPU utilization.
• Validation occurs every --validate_interval episodes using --validate_episodes roll-outs to track policy improvement.
• Checkpointing via --resume and --output enables experiment continuity and model versioning.

Frequently Asked Questions

How do I resume training from a checkpoint?

Use the --resume argument followed by the path to your saved model file. The DDPG class in DRL/ddpg.py loads the actor-critic weights, optimizer states, and training step counters, allowing you to continue from the exact state where training stopped.

What is the difference between --train_times and --episode_train_times?

--train_times sets the total number of environment interactions (the global step counter) before the script terminates, while --episode_train_times specifies how many gradient updates occur immediately after each episode ends. The former controls overall training duration; the latter controls learning intensity per episode.

How does --env_batch affect training performance?

This parameter determines how many parallel painting environments run simultaneously through the fastenv wrapper. Increasing --env_batch improves sample throughput and GPU utilization but requires proportional increases in VRAM. The default value of 96 matches the default --batch_size to maintain balanced data collection and consumption rates.

What does the --warmup parameter do?

During the warmup phase (default 400 steps), the agent populates the replay buffer with random actions without performing any policy updates. This prevents the DDPG networks from training on empty or highly correlated initial data, stabilizing early learning dynamics.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}