How to Integrate Custom Evolutionary Computation (EC) Components with RL Workflows in EvoRL

You integrate custom EC components into EvoRL by implementing the EvoOptimizer abstract base class with init, ask, and tell methods, then injecting the optimizer instance into a hybrid ERL workflow or pure EC workflow template.

EvoRL is a high-performance JAX framework that modularizes Evolutionary Computation (EC) and Reinforcement Learning (RL) into swappable optimizers and workflows. This guide explains how to integrate custom Evolutionary Computation components with RL workflows in EvoRL using the framework's standard interfaces and dependency injection mechanisms.

Architecture and Integration Points

EvoRL separates concerns into distinct architectural layers. The EC Optimizer API in evorl/ec/optimizers/ec_optimizer.py defines the contract that all evolutionary algorithms must implement through three core methods: init for initialization, ask for generating candidate populations, and tell for updating strategy state based on fitness evaluations.

The Hybrid ERL Workflow in evorl/algorithms/erl/erl_workflow.py orchestrates the interaction between RL gradient updates and EC population-based search. This workflow maintains both an RL optimizer (typically Optax) and an EC optimizer, calling self.ec_optimizer.ask() to generate populations and self.ec_optimizer.tell() to process fitness results within the training loop.

Parameter handling relies on ParamVectorSpec from evorl/utils/ec_utils.py, which provides to_vector() and to_tree() methods for flattening JAX pytrees into parameter vectors suitable for evolutionary operations and reconstructing them for agent evaluation.

Implementing the EvoOptimizer Interface

To create a custom EC component, subclass EvoOptimizer and implement the required methods to generate and update populations.

Required Methods

Every custom optimizer must implement three specific methods:

• init(self, pop_size: int, param_spec: ParamVectorSpec, **kwargs) -> ECState: Initialize the evolutionary strategy state, including RNG keys, population statistics, and the ParamVectorSpec for shape handling.
• ask(self, state: ECState) -> tuple[chex.ArrayTree, ECState]: Generate a population of candidate parameter vectors using the current strategy state. Return the population (as a pytree matching the agent's parameter structure) and the updated state.
• tell(self, state: ECState, fitnesses: chex.ArrayTree) -> tuple[dict, ECState]: Update the strategy state based on the evaluated fitnesses of the population. Return a dictionary of metrics and the updated state.

Custom Optimizer Example

Here is a complete implementation of a simple (μ, λ) evolution strategy:

# evorl/ec/optimizers/my_optimizer.py

from evorl.ec.optimizers import EvoOptimizer, ECState
from evorl.utils.ec_utils import ParamVectorSpec
import chex
import jax
import jax.numpy as jnp

class MyCustomOptimizer(EvoOptimizer):
    """Simple (μ, λ) evolution strategy for EvoRL."""
    
    def init(self, pop_size: int, param_spec: ParamVectorSpec, sigma: float = 0.1) -> ECState:
        rng = jax.random.PRNGKey(0)
        mean = jnp.zeros(param_spec.vec_size)
        cov = sigma * jnp.eye(param_spec.vec_size)
        return {
            "mean": mean,
            "cov": cov,
            "rng": rng,
            "pop_size": pop_size,
            "spec": param_spec
        }
    
    def ask(self, state: ECState) -> tuple[chex.ArrayTree, ECState]:
        rng, subkey = jax.random.split(state["rng"])
        # Sample population from multivariate Gaussian

        pop = jax.random.multivariate_normal(
            subkey,
            state["mean"],
            state["cov"],
            (state["pop_size"],)
        )
        # Convert flat vectors back to pytree structure

        pop = state["spec"].to_tree(pop)
        new_state = {**state, "rng": rng, "last_pop": pop}
        return pop, new_state
    
    def tell(self, state: ECState, fitnesses: chex.ArrayTree) -> tuple[dict, ECState]:
        # Select top μ individuals

        mu = state["pop_size"] // 2
        top_idx = jnp.argsort(fitnesses)[-mu:]
        elite = jax.tree_util.tree_map(lambda x: x[top_idx], state["last_pop"])
        # Recompute mean from elites

        new_mean = jax.tree_util.tree_map(lambda x: jnp.mean(x, axis=0), elite)
        new_state = {**state, "mean": new_mean}
        metrics = {"mean_norm": jnp.linalg.norm(new_mean)}
        return metrics, new_state

Registering the Optimizer

Add your optimizer to the package exports to enable configuration-based instantiation:

# evorl/ec/optimizers/__init__.py

from .ec_optimizer import EvoOptimizer, ECState
from .my_optimizer import MyCustomOptimizer

__all__ = ["EvoOptimizer", "ECState", "MyCustomOptimizer"]

Wiring into RL Workflows

Once implemented, inject your optimizer into either hybrid ERL workflows or pure EC workflows.

Hybrid ERL Integration

The ERLWorkflowBase class in evorl/algorithms/erl/erl_workflow.py expects an ec_optimizer attribute. You can inject your custom implementation after building the workflow:

# train_custom_erl.py

import jax
from omegaconf import OmegaConf
from evorl.workflows import ERLTD3WorkflowTemplate
from evorl.ec.optimizers import MyCustomOptimizer
from evorl.utils.ec_utils import ParamVectorSpec

# Initialize agent and environment (details omitted)

agent = MyAgent(...)
dummy_params = agent.init(obs_space, action_space, jax.random.PRNGKey(0)).params
param_spec = ParamVectorSpec(dummy_params)

# Create custom EC optimizer instance

ec_opt = MyCustomOptimizer().init(
    pop_size=64,
    param_spec=param_spec,
    sigma=0.05
)

# Build workflow and inject optimizer

workflow = ERLTD3WorkflowTemplate.build_from_config(
    config=OmegaConf.load("configs/agent/erl/erl-td3.yaml"),
    enable_jit=True
)
workflow.ec_optimizer = ec_opt  # Injection point

# Run training

state = workflow.setup(jax.random.PRNGKey(42))
for step in range(total_steps):
    metrics, state = workflow.step(state)

The ERLWorkflowBase.step method automatically calls self.ec_optimizer.ask() to generate populations and self.ec_optimizer.tell() to process fitness results within the hybrid training loop.

Pure EC Workflows

For evolutionary algorithms without RL components, subclass ECWorkflowTemplate from evorl/workflows/ec_workflow.py:

# my_ec_workflow.py

from evorl.workflows.ec_workflow import ECWorkflowTemplate
from evorl.utils.ec_utils import ParamVectorSpec
from evorl.ec.optimizers import MyCustomOptimizer

class MyECWorkflow(ECWorkflowTemplate):
    def _setup_agent_and_optimizer(self, key):
        agent_state = self.agent.init(self.obs_space, self.action_space, key)
        param_spec = ParamVectorSpec(agent_state.params)
        ec_state = MyCustomOptimizer().init(pop_size=128, param_spec=param_spec)
        return agent_state, ec_state
    
    def _replace_actor_params(self, agent_state, params):
        return agent_state.replace(params=params)

# Build and run

workflow = MyECWorkflow.build_from_config(cfg)
state = workflow.setup(jax.random.PRNGKey(0))
for i in range(500):
    metrics, state = workflow.step(state)

Summary

• Implement the EvoOptimizer interface with init, ask, and tell methods in evorl/ec/optimizers/.
• Use ParamVectorSpec from evorl/utils/ec_utils.py to handle flattening and unflattening of JAX pytrees.
• Register your optimizer in evorl/ec/optimizers/__init__.py for config-based loading.
• Inject the optimizer into ERLWorkflowBase.ec_optimizer for hybrid RL+EC training, or subclass ECWorkflowTemplate for pure evolution.
• Leverage the existing workflow infrastructure to handle population evaluation and agent parameter replacement automatically.

Frequently Asked Questions

What interface must custom EC optimizers implement?

Custom optimizers must inherit from EvoOptimizer and implement three methods: init() to create the initial strategy state, ask() to generate candidate populations, and tell() to update the strategy based on fitness evaluations. These methods are defined in evorl/ec/optimizers/ec_optimizer.py.

How does EvoRL convert between agent parameters and EC vectors?

EvoRL uses ParamVectorSpec from evorl/utils/ec_utils.py to convert between nested JAX pytrees (agent parameters) and flat vectors (required by most EC algorithms). The to_vector() method flattens parameters, while to_tree() reconstructs the original structure.

Can I use custom EC components without RL?

Yes. You can create pure EC workflows by subclassing ECWorkflowTemplate from evorl/workflows/ec_workflow.py and implementing _setup_agent_and_optimizer() and _replace_actor_params(). This runs the evolutionary loop without any RL gradient updates.

Where do I register new optimizers for configuration-based loading?

Register your optimizer in evorl/ec/optimizers/__init__.py by importing the class and adding it to the __all__ list. This enables instantiation via Hydra/OmegaConf configurations using the standard EvoRL config system.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}