# How to Configure GRPO Hyperparameters in AReaL: A Complete Guide

> Master GRPO hyperparameters in AReaL with this guide. Learn to configure clipping bounds, reward normalization, and advantage estimation using YAML or CLI overrides for optimal performance.

- Repository: [inclusionAI/areal](https://github.com/inclusionai/areal)
- Tags: how-to-guide
- Published: 2026-03-04

---

**GRPO hyperparameters in AReaL are controlled through three dataclasses—`GRPOConfig`, `PPOActorConfig`, and `NormConfig`—located in [`areal/api/cli_args.py`](https://github.com/inclusionai/areal/blob/main/areal/api/cli_args.py), allowing you to control clipping bounds, reward normalization, and advantage estimation via YAML files or CLI overrides.**

Setting up **GRPO (Group Relative Policy Optimization) algorithm hyperparameters** in the AReaL framework requires understanding its inheritance from standard PPO configurations. This guide walks through the specific dataclass structures, critical parameter locations, and practical configuration methods based on the actual source code in the `inclusionai/areal` repository.

## Understanding GRPO Configuration Architecture

AReaL implements GRPO as a specialized wrapper around the generic PPO configuration system. All hyperparameters are defined in three hierarchical dataclasses in [`areal/api/cli_args.py`](https://github.com/inclusionai/areal/blob/main/areal/api/cli_args.py):

- **`GRPOConfig`** (lines 2155–2158): Acts as the top-level experiment configuration. It inherits from `PPOConfig` and does not add new fields; all GRPO-specific knobs reside in the nested `actor` configuration.

- **`PPOActorConfig`** (lines 979–1005): Contains the core policy-gradient settings that directly affect GRPO behavior, including clipping bounds, normalization flags, and importance-sampling levels.

- **`NormConfig`** (lines 33–70): Defines fine-grained control for mean and standard deviation normalization of rewards and advantages, supporting both batch-level and group-level aggregation.

## Core GRPO Hyperparameters

All tunable knobs for the GRPO algorithm are exposed through the `actor` configuration namespace. These parameters control clipping bounds, normalization strategies, and importance-sampling behavior.

### Policy Gradient Clipping

- **`actor.eps_clip`** — Sets the symmetric clipping bound ε for importance sampling ratios, constraining the ratio to `[1-ε, 1+ε]`. The default value in `PPOActorConfig` is `0.2`, but GRPO configurations typically override this to `0.4` for greater exploration. Adjust via CLI: `--actor.eps_clip 0.3`.

- **`actor.eps_clip_higher`** — Configures asymmetric clipping by specifying a distinct upper bound. This parameter is `null` by default and requires `actor.use_decoupled_loss=true` to activate. Set with `--actor.eps_clip_higher 0.5`.

- **`actor.use_decoupled_loss`** — Boolean flag enabling the decoupled PPO loss formulation necessary for asymmetric clipping. The default is `true` in GRPO examples.

### Reward and Advantage Normalization

Normalization behavior is governed by nested `NormConfig` objects referenced as `actor.reward_norm` and `actor.adv_norm`:

- **`actor.reward_norm.mean_level`** — Controls mean subtraction aggregation. Use `batch` for standard GRPO or `group` for Dr.GRPO variants. CLI: `--actor.reward_norm.mean_level=group`.

- **`actor.reward_norm.std_level`** — Controls standard deviation normalization. Options include `batch`, `group`, or `null` (disabled). For Dr.GRPO, set to `null` to disable std normalization.

- **`actor.reward_norm.group_size`** — Defines the number of trajectories per group when using group-level normalization. Typically set to `${gconfig.n_samples}` to match the number of sampled completions.

- **`actor.adv_norm.mean_leave1out`** — Enables leave-one-out averaging for advantages when set to `true`, implementing the RLOO (Remove Likelihood One Out) algorithm.

### Importance Sampling Configuration

- **`actor.importance_sampling_level`** — Determines reward aggregation granularity. Set to `"token"` for standard GRPO (per-token rewards) or `"sequence"` for GSPO (Group Sequence Policy Optimization) which uses sequence-level rewards.

## Default Configuration and Example Files

The repository ships a ready-to-run GRPO configuration for the GSM8K benchmark at [`examples/math/gsm8k_grpo.yaml`](https://github.com/inclusionai/areal/blob/main/examples/math/gsm8k_grpo.yaml). This file illustrates the default GRPO hyperparameters:

```yaml
actor:
  eps_clip: 0.4                     # larger than PPO default

  temperature: ${gconfig.temperature}
  reward_scaling: 10.0
  reward_bias: -0.5
  kl_ctl: 0.0
  ppo_n_minibatches: 1
  recompute_logprob: true
  use_decoupled_loss: true
  behave_imp_weight_cap: 5.0
  reward_norm:
    mean_level: group
    std_level: group
    group_size: ${gconfig.n_samples}
  adv_norm:
    mean_level: batch
    std_level: batch
  weight_update_mode: xccl

```

These defaults can be overridden on the command line without editing the YAML. The documentation in [`docs/algorithms/grpo_series.md`](https://github.com/inclusionai/areal/blob/main/docs/algorithms/grpo_series.md) also summarizes how to switch algorithms using CLI overrides.

## Setting Up a Custom GRPO Experiment

Follow this workflow to configure GRPO hyperparameters for your specific use case:

1. **Start from the template** ([`gsm8k_grpo.yaml`](https://github.com/inclusionai/areal/blob/main/gsm8k_grpo.yaml)) or create a new YAML file.

2. **Adjust the `actor` block**: Modify `eps_clip`, `reward_norm`, `adv_norm`, and other fields according to your target variant.

3. **Launch the experiment** with your desired scheduler. For a local run:

   ```bash
   python3 examples/math/gsm8k_rl.py \
       --config examples/math/gsm8k_grpo.yaml \
       scheduler.type=local \
       --actor.eps_clip 0.3 \
       --actor.reward_norm.mean_level=batch \
       --actor.reward_norm.std_level=batch
   ```

### Programmatic Configuration

You can also build configurations programmatically using the dataclasses directly:

```python
from areal.api.cli_args import GRPOConfig, load_expr_config

# Load the default YAML and modify fields in Python

cfg, _ = load_expr_config(
    ["--config", "examples/math/gsm8k_grpo.yaml"], GRPOConfig
)

# Change clipping and normalisation

cfg.actor.eps_clip = 0.3
cfg.actor.reward_norm.mean_level = "batch"
cfg.actor.reward_norm.std_level = "batch"
cfg.actor.adv_norm.mean_leave1out = True   # RLOO style

print(cfg)   # ready to pass to the trainer

```

## Advanced Variant: Configuring Dr.GRPO

To switch from standard GRPO to the Dr.GRPO variant (which uses group-level advantage normalization), override the default batch-level settings:

```bash
python3 examples/math/gsm8k_rl.py \
    --config examples/math/gsm8k_grpo.yaml \
    scheduler.type=local \
    actor.adv_norm.mean_level=group \
    actor.adv_norm.std_level=null

```

This configuration matches the algorithmic description in [`docs/algorithms/grpo_series.md`](https://github.com/inclusionai/areal/blob/main/docs/algorithms/grpo_series.md), implementing group-relative advantage estimation without standard deviation normalization.

## Summary

- **GRPO hyperparameters** in AReaL are defined in [`areal/api/cli_args.py`](https://github.com/inclusionai/areal/blob/main/areal/api/cli_args.py) within three dataclasses: `GRPOConfig`, `PPOActorConfig`, and `NormConfig`.
- **Key parameters** include `eps_clip` for ratio clipping, `reward_norm` and `adv_norm` for controlling normalization levels (`batch` vs `group`), and `importance_sampling_level` for token vs sequence aggregation.
- **Default configurations** are provided in [`examples/math/gsm8k_grpo.yaml`](https://github.com/inclusionai/areal/blob/main/examples/math/gsm8k_grpo.yaml), which can be customized via CLI overrides without file modification.
- **Advanced variants** like Dr.GRPO are achieved by switching `adv_norm.mean_level` to `group` and disabling standard deviation normalization.

## Frequently Asked Questions

### What is the difference between GRPO and Dr.GRPO hyperparameters in AReaL?

The primary difference lies in the **advantage normalization** settings. Standard GRPO uses `actor.adv_norm.mean_level=batch` and `actor.adv_norm.std_level=batch`, normalizing across the entire batch. Dr.GRPO switches to `actor.adv_norm.mean_level=group` and typically sets `actor.adv_norm.std_level=null`, normalizing advantages only within each group of samples and removing standard deviation scaling.

### How do I change the clipping bound for GRPO without editing the YAML file?

Use the command-line interface to override the `eps_clip` parameter. Append `--actor.eps_clip 0.3` to your launch command to change the symmetric clipping bound from its default value. For asymmetric clipping, additionally specify `--actor.eps_clip_higher 0.5` and ensure `--actor.use_decoupled_loss=true` is set.

### Where are the default GRPO hyperparameters defined in the source code?

Default values are defined in [`areal/api/cli_args.py`](https://github.com/inclusionai/areal/blob/main/areal/api/cli_args.py) within two specific dataclasses. The `PPOActorConfig` class (around lines 979–1005) defines clipping bounds and importance sampling levels, while the `NormConfig` class (around lines 33–70) defines normalization parameters. The `GRPOConfig` class (lines 2155–2158) inherits from `PPOConfig` and serves as the top-level container.

### Can I configure GRPO hyperparameters programmatically instead of using YAML?

Yes, you can instantiate and modify the configuration dataclasses directly in Python. Import `GRPOConfig` and `load_expr_config` from `areal.api.cli_args`, load your base configuration using `load_expr_config`, then modify fields such as `cfg.actor.eps_clip` or `cfg.actor.reward_norm.mean_level` before passing the configuration to the trainer. This approach is particularly useful for automated hyperparameter sweeps.