# How to Configure Population-Based Training (PBT) with Hydra Configuration Files in EvoRL

> Learn to configure Population-Based Training PBT in EvoRL using Hydra. Combine PBT and RL workflows, define population size, hyperparameter search spaces, and exploit-explore ratios.

- Repository: [EMI-Group/evorl](https://github.com/emi-group/evorl)
- Tags: how-to-guide
- Published: 2026-03-01

---

**To configure Population-Based Training (PBT) in EvoRL, you compose a Hydra configuration file that combines a PBT workflow class with a target RL algorithm workflow, defining population size, hyperparameter search spaces, and exploit-and-explore ratios.**

EvoRL is an open-source evolutionary reinforcement learning framework that leverages Hydra for declarative experiment configuration. When you configure Population-Based Training (PBT) with Hydra configuration files in EvoRL, you are essentially orchestrating a meta-learning loop where a population of agents evolves hyperparameters while training on a base algorithm like PPO or SAC.

## Understanding PBT Architecture in EvoRL

EvoRL separates concerns between the meta-level PBT controller and the underlying RL training loop. In [`evorl/algorithms/meta/pbt_workflow.py`](https://github.com/emi-group/evorl/blob/main/evorl/algorithms/meta/pbt_workflow.py), the `PBTWorkflow` class (and its templates `PBTWorkflowTemplate` and `PBTOffpolicyWorkflowTemplate`) implements the core logic for population management, exploit-and-explore operations, and metric aggregation.

The configuration system requires two distinct components:

- **PBT Workflow**: The meta-controller that manages the population (configured via `workflow_cls`)
- **Target Workflow**: The base RL algorithm configuration (imported via Hydra's `defaults` mechanism)

## Key Configuration Components

### Hydra Defaults and Composition

Every PBT configuration file starts with a `defaults` block that composes the target workflow into the configuration namespace. This creates a nested `target_workflow` key containing all parameters from the base algorithm.

```yaml
defaults:
  - /agent@target_workflow: ppo   # Imports configs/agent/ppo.yaml as target_workflow

  - _self_

```

The `/agent@target_workflow` syntax instructs Hydra to load the PPO configuration and place it under the `target_workflow` key, allowing the PBT meta-controller to access and modify the base algorithm's hyperparameters.

### Population Hyperparameters

The PBT-specific parameters control the evolutionary process:

- **`pop_size`**: Total number of agents in the population (must be divisible by the number of devices; the workflow automatically rescales if needed)
- **`bottom_ratio`**: Fraction of population identified as underperforming (exploited and replaced)
- **`top_ratio`**: Fraction of population identified as high-performing (copied to replace bottom individuals)
- **`perturb_factor`**: Multiplicative noise applied during exploration (e.g., `0.2` for ±20% perturbation)
- **`search_space`**: Bounds for each tunable hyperparameter with `low` and `high` values

### Warm-up Phase

The `warmup_steps` parameter defines an initial training period where standard RL updates occur without any exploit-and-explore operations. The workflow checks `state.metrics.iterations + 1 <= ceil(warmup_steps / workflow_steps_per_iter)` to determine when to begin PBT operations.

## Step-by-Step Configuration Guide

To create a custom PBT experiment, follow these steps:

1. **Copy an existing PBT configuration** from [`configs/agent/pbt-ppo.yaml`](https://github.com/emi-group/evorl/blob/main/configs/agent/pbt-ppo.yaml) or [`configs/agent/meta/pbt-paramppo.yaml`](https://github.com/emi-group/evorl/blob/main/configs/agent/meta/pbt-paramppo.yaml) to a new file (e.g., [`my_pbt.yaml`](https://github.com/emi-group/evorl/blob/main/my_pbt.yaml)).

2. **Select your target algorithm** by modifying the defaults block:
   ```yaml
   defaults:
     - /agent@target_workflow: sac   # Switch from PPO to SAC

     - _self_
   ```

3. **Adjust population parameters** based on your computational resources:
   ```yaml
   pop_size: 16
   bottom_ratio: 0.25
   top_ratio: 0.25
   perturb_factor:
     lr: 0.3
     entropy_coeff: 0.2
   ```

4. **Define the search space** for hyperparameters you want to evolve:
   ```yaml
   search_space:
     lr:
       low: 1e-5
       high: 5e-3
     entropy_coeff:
       low: 0.0
       high: 0.1
   ```

5. **Configure warm-up and training duration**:
   ```yaml
   warmup_steps: 20
   num_iters: 300
   workflow_steps_per_iter: 10
   ```

## Running a PBT Experiment

Execute your configured PBT experiment using the Hydra CLI from the repository root:

```bash
python -m scripts.train \
    workflow_cls=evorl.algorithms.meta.pbt.PBTWorkflow \
    +agent=my_pbt.yaml \
    env=env/brax/walker2d \
    seed=123 \
    recorders=[log]

```

**Key command components:**

- `workflow_cls=evorl.algorithms.meta.pbt.PBTWorkflow` – Instantiates the PBT meta-controller from [`evorl/algorithms/meta/pbt_workflow.py`](https://github.com/emi-group/evorl/blob/main/evorl/algorithms/meta/pbt_workflow.py)
- `+agent=my_pbt.yaml` – Loads your custom PBT configuration (the `+` prefix adds this to the default config group)
- `env=env/brax/walker2d` – Selects the Brax Walker2D environment (config files located in `configs/env/brax/`)
- `recorders=[log]` – Restricts logging to console output (omit or modify to enable WandB)

The [`scripts/train.py`](https://github.com/emi-group/evorl/blob/main/scripts/train.py) entry point resolves the composed configuration through Hydra, builds the workflow via `workflow_cls.build_from_config`, and executes the `init → learn → close` lifecycle while automatically handling multi-device sharding for the population.

## Summary

- **EvoRL implements PBT** through the `PBTWorkflow` class in [`evorl/algorithms/meta/pbt_workflow.py`](https://github.com/emi-group/evorl/blob/main/evorl/algorithms/meta/pbt_workflow.py), which orchestrates populations of RL agents.
- **Hydra composition** allows you to combine a PBT meta-configuration with any target workflow (PPO, SAC, etc.) using the `defaults` block and `/agent@target_workflow` syntax.
- **Key parameters** include `pop_size`, `bottom_ratio`, `top_ratio`, `perturb_factor`, and `search_space` to control the evolutionary process.
- **Warm-up steps** allow initial convergence before PBT operations begin, configured via `warmup_steps`.
- **Execution** requires specifying `workflow_cls=evorl.algorithms.meta.pbt.PBTWorkflow` and your custom agent config when running `python -m scripts.train`.

## Frequently Asked Questions

### What is the difference between `PBTWorkflowTemplate` and `PBTOffpolicyWorkflowTemplate`?

`PBTWorkflowTemplate` in [`evorl/algorithms/meta/pbt_workflow.py`](https://github.com/emi-group/evorl/blob/main/evorl/algorithms/meta/pbt_workflow.py) is designed for on-policy algorithms like PPO, while `PBTOffpolicyWorkflowTemplate` handles off-policy algorithms such as SAC or TD3. Both inherit from `PBTWorkflowBase` and implement the `exploit_and_explore` method, but they differ in how they manage replay buffers and update frequencies to accommodate the underlying algorithm's sample efficiency characteristics.

### How do I tune the `perturb_factor` for different hyperparameters?

The `perturb_factor` dictionary in your PBT config controls the relative magnitude of multiplicative noise applied during exploration. Set higher values (e.g., `0.3` to `0.5`) for hyperparameters where you expect high sensitivity or need aggressive exploration, such as learning rate or entropy coefficient. Use lower values (e.g., `0.1` to `0.2`) for more stable parameters like discount factor. The perturbation is applied as `new_value = old_value * (1 + perturb_factor * random_uniform(-1, 1))` within the bounds defined in `search_space`.

### Can I use PBT with custom RL algorithms not included in EvoRL?

Yes, you can configure PBT to work with custom workflows by ensuring your custom algorithm implements the EvoRL workflow interface and can be instantiated through Hydra. Create a configuration file for your custom agent in `configs/agent/`, then reference it in your PBT config's `defaults` block using `/agent@target_workflow: your_custom_agent`. The `PBTWorkflow` class will treat your custom algorithm as the target workflow, evolving its hyperparameters according to the PBT schedule while your custom workflow handles the underlying RL training logic.