How to Configure Population-Based Training (PBT) with Hydra Configuration Files in EvoRL

To configure Population-Based Training (PBT) in EvoRL, you compose a Hydra configuration file that combines a PBT workflow class with a target RL algorithm workflow, defining population size, hyperparameter search spaces, and exploit-and-explore ratios.

EvoRL is an open-source evolutionary reinforcement learning framework that leverages Hydra for declarative experiment configuration. When you configure Population-Based Training (PBT) with Hydra configuration files in EvoRL, you are essentially orchestrating a meta-learning loop where a population of agents evolves hyperparameters while training on a base algorithm like PPO or SAC.

Understanding PBT Architecture in EvoRL

EvoRL separates concerns between the meta-level PBT controller and the underlying RL training loop. In evorl/algorithms/meta/pbt_workflow.py, the PBTWorkflow class (and its templates PBTWorkflowTemplate and PBTOffpolicyWorkflowTemplate) implements the core logic for population management, exploit-and-explore operations, and metric aggregation.

The configuration system requires two distinct components:

• PBT Workflow: The meta-controller that manages the population (configured via workflow_cls)
• Target Workflow: The base RL algorithm configuration (imported via Hydra's defaults mechanism)

Key Configuration Components

Hydra Defaults and Composition

Every PBT configuration file starts with a defaults block that composes the target workflow into the configuration namespace. This creates a nested target_workflow key containing all parameters from the base algorithm.

defaults:
  - /agent@target_workflow: ppo   # Imports configs/agent/ppo.yaml as target_workflow

  - _self_

The /agent@target_workflow syntax instructs Hydra to load the PPO configuration and place it under the target_workflow key, allowing the PBT meta-controller to access and modify the base algorithm's hyperparameters.

Population Hyperparameters

The PBT-specific parameters control the evolutionary process:

• pop_size: Total number of agents in the population (must be divisible by the number of devices; the workflow automatically rescales if needed)
• bottom_ratio: Fraction of population identified as underperforming (exploited and replaced)
• top_ratio: Fraction of population identified as high-performing (copied to replace bottom individuals)
• perturb_factor: Multiplicative noise applied during exploration (e.g., 0.2 for ±20% perturbation)
• search_space: Bounds for each tunable hyperparameter with low and high values

Warm-up Phase

The warmup_steps parameter defines an initial training period where standard RL updates occur without any exploit-and-explore operations. The workflow checks state.metrics.iterations + 1 <= ceil(warmup_steps / workflow_steps_per_iter) to determine when to begin PBT operations.

Step-by-Step Configuration Guide

To create a custom PBT experiment, follow these steps:

1. Copy an existing PBT configuration from configs/agent/pbt-ppo.yaml or configs/agent/meta/pbt-paramppo.yaml to a new file (e.g., my_pbt.yaml).

2. Select your target algorithm by modifying the defaults block:

   defaults:
     - /agent@target_workflow: sac   # Switch from PPO to SAC
   
     - _self_

3. Adjust population parameters based on your computational resources:

   pop_size: 16
   bottom_ratio: 0.25
   top_ratio: 0.25
   perturb_factor:
     lr: 0.3
     entropy_coeff: 0.2

4. Define the search space for hyperparameters you want to evolve:

   search_space:
     lr:
       low: 1e-5
       high: 5e-3
     entropy_coeff:
       low: 0.0
       high: 0.1

5. Configure warm-up and training duration:

   warmup_steps: 20
   num_iters: 300
   workflow_steps_per_iter: 10

Running a PBT Experiment

Execute your configured PBT experiment using the Hydra CLI from the repository root:

python -m scripts.train \
    workflow_cls=evorl.algorithms.meta.pbt.PBTWorkflow \
    +agent=my_pbt.yaml \
    env=env/brax/walker2d \
    seed=123 \
    recorders=[log]

Key command components:

• workflow_cls=evorl.algorithms.meta.pbt.PBTWorkflow – Instantiates the PBT meta-controller from evorl/algorithms/meta/pbt_workflow.py
• +agent=my_pbt.yaml – Loads your custom PBT configuration (the + prefix adds this to the default config group)
• env=env/brax/walker2d – Selects the Brax Walker2D environment (config files located in configs/env/brax/)
• recorders=[log] – Restricts logging to console output (omit or modify to enable WandB)

The scripts/train.py entry point resolves the composed configuration through Hydra, builds the workflow via workflow_cls.build_from_config, and executes the init → learn → close lifecycle while automatically handling multi-device sharding for the population.

Summary

• EvoRL implements PBT through the PBTWorkflow class in evorl/algorithms/meta/pbt_workflow.py, which orchestrates populations of RL agents.
• Hydra composition allows you to combine a PBT meta-configuration with any target workflow (PPO, SAC, etc.) using the defaults block and /agent@target_workflow syntax.
• Key parameters include pop_size, bottom_ratio, top_ratio, perturb_factor, and search_space to control the evolutionary process.
• Warm-up steps allow initial convergence before PBT operations begin, configured via warmup_steps.
• Execution requires specifying workflow_cls=evorl.algorithms.meta.pbt.PBTWorkflow and your custom agent config when running python -m scripts.train.

Frequently Asked Questions

What is the difference between PBTWorkflowTemplate and PBTOffpolicyWorkflowTemplate?

PBTWorkflowTemplate in evorl/algorithms/meta/pbt_workflow.py is designed for on-policy algorithms like PPO, while PBTOffpolicyWorkflowTemplate handles off-policy algorithms such as SAC or TD3. Both inherit from PBTWorkflowBase and implement the exploit_and_explore method, but they differ in how they manage replay buffers and update frequencies to accommodate the underlying algorithm's sample efficiency characteristics.

How do I tune the perturb_factor for different hyperparameters?

The perturb_factor dictionary in your PBT config controls the relative magnitude of multiplicative noise applied during exploration. Set higher values (e.g., 0.3 to 0.5) for hyperparameters where you expect high sensitivity or need aggressive exploration, such as learning rate or entropy coefficient. Use lower values (e.g., 0.1 to 0.2) for more stable parameters like discount factor. The perturbation is applied as new_value = old_value * (1 + perturb_factor * random_uniform(-1, 1)) within the bounds defined in search_space.

Can I use PBT with custom RL algorithms not included in EvoRL?

Yes, you can configure PBT to work with custom workflows by ensuring your custom algorithm implements the EvoRL workflow interface and can be instantiated through Hydra. Create a configuration file for your custom agent in configs/agent/, then reference it in your PBT config's defaults block using /agent@target_workflow: your_custom_agent. The PBTWorkflow class will treat your custom algorithm as the target workflow, evolving its hyperparameters according to the PBT schedule while your custom workflow handles the underlying RL training logic.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}