# How to Integrate Custom Evolutionary Computation (EC) Components with RL Workflows in EvoRL

> Integrate custom EC components into EvoRL by implementing the EvoOptimizer abstract base class with init ask and tell methods. Learn how to inject your optimizer into hybrid ERL or pure EC workflows in this guide.

- Repository: [EMI-Group/evorl](https://github.com/emi-group/evorl)
- Tags: how-to-guide
- Published: 2026-03-01

---

**You integrate custom EC components into EvoRL by implementing the `EvoOptimizer` abstract base class with `init`, `ask`, and `tell` methods, then injecting the optimizer instance into a hybrid ERL workflow or pure EC workflow template.**

EvoRL is a high-performance JAX framework that modularizes Evolutionary Computation (EC) and Reinforcement Learning (RL) into swappable optimizers and workflows. This guide explains how to integrate custom Evolutionary Computation components with RL workflows in EvoRL using the framework's standard interfaces and dependency injection mechanisms.

## Architecture and Integration Points

EvoRL separates concerns into distinct architectural layers. The **EC Optimizer API** in [`evorl/ec/optimizers/ec_optimizer.py`](https://github.com/emi-group/evorl/blob/main/evorl/ec/optimizers/ec_optimizer.py) defines the contract that all evolutionary algorithms must implement through three core methods: `init` for initialization, `ask` for generating candidate populations, and `tell` for updating strategy state based on fitness evaluations.

The **Hybrid ERL Workflow** in [`evorl/algorithms/erl/erl_workflow.py`](https://github.com/emi-group/evorl/blob/main/evorl/algorithms/erl/erl_workflow.py) orchestrates the interaction between RL gradient updates and EC population-based search. This workflow maintains both an RL optimizer (typically Optax) and an EC optimizer, calling `self.ec_optimizer.ask()` to generate populations and `self.ec_optimizer.tell()` to process fitness results within the training loop.

Parameter handling relies on **ParamVectorSpec** from [`evorl/utils/ec_utils.py`](https://github.com/emi-group/evorl/blob/main/evorl/utils/ec_utils.py), which provides `to_vector()` and `to_tree()` methods for flattening JAX pytrees into parameter vectors suitable for evolutionary operations and reconstructing them for agent evaluation.

## Implementing the EvoOptimizer Interface

To create a custom EC component, subclass `EvoOptimizer` and implement the required methods to generate and update populations.

### Required Methods

Every custom optimizer must implement three specific methods:

- **`init(self, pop_size: int, param_spec: ParamVectorSpec, **kwargs) -> ECState`**: Initialize the evolutionary strategy state, including RNG keys, population statistics, and the `ParamVectorSpec` for shape handling.
- **`ask(self, state: ECState) -> tuple[chex.ArrayTree, ECState]`**: Generate a population of candidate parameter vectors using the current strategy state. Return the population (as a pytree matching the agent's parameter structure) and the updated state.
- **`tell(self, state: ECState, fitnesses: chex.ArrayTree) -> tuple[dict, ECState]`**: Update the strategy state based on the evaluated fitnesses of the population. Return a dictionary of metrics and the updated state.

### Custom Optimizer Example

Here is a complete implementation of a simple (μ, λ) evolution strategy:

```python

# evorl/ec/optimizers/my_optimizer.py

from evorl.ec.optimizers import EvoOptimizer, ECState
from evorl.utils.ec_utils import ParamVectorSpec
import chex
import jax
import jax.numpy as jnp

class MyCustomOptimizer(EvoOptimizer):
    """Simple (μ, λ) evolution strategy for EvoRL."""
    
    def init(self, pop_size: int, param_spec: ParamVectorSpec, sigma: float = 0.1) -> ECState:
        rng = jax.random.PRNGKey(0)
        mean = jnp.zeros(param_spec.vec_size)
        cov = sigma * jnp.eye(param_spec.vec_size)
        return {
            "mean": mean,
            "cov": cov,
            "rng": rng,
            "pop_size": pop_size,
            "spec": param_spec
        }
    
    def ask(self, state: ECState) -> tuple[chex.ArrayTree, ECState]:
        rng, subkey = jax.random.split(state["rng"])
        # Sample population from multivariate Gaussian

        pop = jax.random.multivariate_normal(
            subkey,
            state["mean"],
            state["cov"],
            (state["pop_size"],)
        )
        # Convert flat vectors back to pytree structure

        pop = state["spec"].to_tree(pop)
        new_state = {**state, "rng": rng, "last_pop": pop}
        return pop, new_state
    
    def tell(self, state: ECState, fitnesses: chex.ArrayTree) -> tuple[dict, ECState]:
        # Select top μ individuals

        mu = state["pop_size"] // 2
        top_idx = jnp.argsort(fitnesses)[-mu:]
        elite = jax.tree_util.tree_map(lambda x: x[top_idx], state["last_pop"])
        # Recompute mean from elites

        new_mean = jax.tree_util.tree_map(lambda x: jnp.mean(x, axis=0), elite)
        new_state = {**state, "mean": new_mean}
        metrics = {"mean_norm": jnp.linalg.norm(new_mean)}
        return metrics, new_state

```

### Registering the Optimizer

Add your optimizer to the package exports to enable configuration-based instantiation:

```python

# evorl/ec/optimizers/__init__.py

from .ec_optimizer import EvoOptimizer, ECState
from .my_optimizer import MyCustomOptimizer

__all__ = ["EvoOptimizer", "ECState", "MyCustomOptimizer"]

```

## Wiring into RL Workflows

Once implemented, inject your optimizer into either hybrid ERL workflows or pure EC workflows.

### Hybrid ERL Integration

The `ERLWorkflowBase` class in [`evorl/algorithms/erl/erl_workflow.py`](https://github.com/emi-group/evorl/blob/main/evorl/algorithms/erl/erl_workflow.py) expects an `ec_optimizer` attribute. You can inject your custom implementation after building the workflow:

```python

# train_custom_erl.py

import jax
from omegaconf import OmegaConf
from evorl.workflows import ERLTD3WorkflowTemplate
from evorl.ec.optimizers import MyCustomOptimizer
from evorl.utils.ec_utils import ParamVectorSpec

# Initialize agent and environment (details omitted)

agent = MyAgent(...)
dummy_params = agent.init(obs_space, action_space, jax.random.PRNGKey(0)).params
param_spec = ParamVectorSpec(dummy_params)

# Create custom EC optimizer instance

ec_opt = MyCustomOptimizer().init(
    pop_size=64,
    param_spec=param_spec,
    sigma=0.05
)

# Build workflow and inject optimizer

workflow = ERLTD3WorkflowTemplate.build_from_config(
    config=OmegaConf.load("configs/agent/erl/erl-td3.yaml"),
    enable_jit=True
)
workflow.ec_optimizer = ec_opt  # Injection point

# Run training

state = workflow.setup(jax.random.PRNGKey(42))
for step in range(total_steps):
    metrics, state = workflow.step(state)

```

The `ERLWorkflowBase.step` method automatically calls `self.ec_optimizer.ask()` to generate populations and `self.ec_optimizer.tell()` to process fitness results within the hybrid training loop.

### Pure EC Workflows

For evolutionary algorithms without RL components, subclass `ECWorkflowTemplate` from [`evorl/workflows/ec_workflow.py`](https://github.com/emi-group/evorl/blob/main/evorl/workflows/ec_workflow.py):

```python

# my_ec_workflow.py

from evorl.workflows.ec_workflow import ECWorkflowTemplate
from evorl.utils.ec_utils import ParamVectorSpec
from evorl.ec.optimizers import MyCustomOptimizer

class MyECWorkflow(ECWorkflowTemplate):
    def _setup_agent_and_optimizer(self, key):
        agent_state = self.agent.init(self.obs_space, self.action_space, key)
        param_spec = ParamVectorSpec(agent_state.params)
        ec_state = MyCustomOptimizer().init(pop_size=128, param_spec=param_spec)
        return agent_state, ec_state
    
    def _replace_actor_params(self, agent_state, params):
        return agent_state.replace(params=params)

# Build and run

workflow = MyECWorkflow.build_from_config(cfg)
state = workflow.setup(jax.random.PRNGKey(0))
for i in range(500):
    metrics, state = workflow.step(state)

```

## Summary

- **Implement** the `EvoOptimizer` interface with `init`, `ask`, and `tell` methods in `evorl/ec/optimizers/`.
- **Use** `ParamVectorSpec` from [`evorl/utils/ec_utils.py`](https://github.com/emi-group/evorl/blob/main/evorl/utils/ec_utils.py) to handle flattening and unflattening of JAX pytrees.
- **Register** your optimizer in [`evorl/ec/optimizers/__init__.py`](https://github.com/emi-group/evorl/blob/main/evorl/ec/optimizers/__init__.py) for config-based loading.
- **Inject** the optimizer into `ERLWorkflowBase.ec_optimizer` for hybrid RL+EC training, or subclass `ECWorkflowTemplate` for pure evolution.
- **Leverage** the existing workflow infrastructure to handle population evaluation and agent parameter replacement automatically.

## Frequently Asked Questions

### What interface must custom EC optimizers implement?

Custom optimizers must inherit from `EvoOptimizer` and implement three methods: `init()` to create the initial strategy state, `ask()` to generate candidate populations, and `tell()` to update the strategy based on fitness evaluations. These methods are defined in [`evorl/ec/optimizers/ec_optimizer.py`](https://github.com/emi-group/evorl/blob/main/evorl/ec/optimizers/ec_optimizer.py).

### How does EvoRL convert between agent parameters and EC vectors?

EvoRL uses `ParamVectorSpec` from [`evorl/utils/ec_utils.py`](https://github.com/emi-group/evorl/blob/main/evorl/utils/ec_utils.py) to convert between nested JAX pytrees (agent parameters) and flat vectors (required by most EC algorithms). The `to_vector()` method flattens parameters, while `to_tree()` reconstructs the original structure.

### Can I use custom EC components without RL?

Yes. You can create pure EC workflows by subclassing `ECWorkflowTemplate` from [`evorl/workflows/ec_workflow.py`](https://github.com/emi-group/evorl/blob/main/evorl/workflows/ec_workflow.py) and implementing `_setup_agent_and_optimizer()` and `_replace_actor_params()`. This runs the evolutionary loop without any RL gradient updates.

### Where do I register new optimizers for configuration-based loading?

Register your optimizer in [`evorl/ec/optimizers/__init__.py`](https://github.com/emi-group/evorl/blob/main/evorl/ec/optimizers/__init__.py) by importing the class and adding it to the `__all__` list. This enables instantiation via Hydra/OmegaConf configurations using the standard EvoRL config system.