How to Configure GRPO Hyperparameters in AReaL: A Complete Guide

GRPO hyperparameters in AReaL are controlled through three dataclasses—GRPOConfig, PPOActorConfig, and NormConfig—located in areal/api/cli_args.py, allowing you to control clipping bounds, reward normalization, and advantage estimation via YAML files or CLI overrides.

Setting up GRPO (Group Relative Policy Optimization) algorithm hyperparameters in the AReaL framework requires understanding its inheritance from standard PPO configurations. This guide walks through the specific dataclass structures, critical parameter locations, and practical configuration methods based on the actual source code in the inclusionai/areal repository.

Understanding GRPO Configuration Architecture

AReaL implements GRPO as a specialized wrapper around the generic PPO configuration system. All hyperparameters are defined in three hierarchical dataclasses in areal/api/cli_args.py:

• GRPOConfig (lines 2155–2158): Acts as the top-level experiment configuration. It inherits from PPOConfig and does not add new fields; all GRPO-specific knobs reside in the nested actor configuration.

• PPOActorConfig (lines 979–1005): Contains the core policy-gradient settings that directly affect GRPO behavior, including clipping bounds, normalization flags, and importance-sampling levels.

• NormConfig (lines 33–70): Defines fine-grained control for mean and standard deviation normalization of rewards and advantages, supporting both batch-level and group-level aggregation.

Core GRPO Hyperparameters

All tunable knobs for the GRPO algorithm are exposed through the actor configuration namespace. These parameters control clipping bounds, normalization strategies, and importance-sampling behavior.

Policy Gradient Clipping

• actor.eps_clip — Sets the symmetric clipping bound ε for importance sampling ratios, constraining the ratio to [1-ε, 1+ε]. The default value in PPOActorConfig is 0.2, but GRPO configurations typically override this to 0.4 for greater exploration. Adjust via CLI: --actor.eps_clip 0.3.

• actor.eps_clip_higher — Configures asymmetric clipping by specifying a distinct upper bound. This parameter is null by default and requires actor.use_decoupled_loss=true to activate. Set with --actor.eps_clip_higher 0.5.

• actor.use_decoupled_loss — Boolean flag enabling the decoupled PPO loss formulation necessary for asymmetric clipping. The default is true in GRPO examples.

Reward and Advantage Normalization

Normalization behavior is governed by nested NormConfig objects referenced as actor.reward_norm and actor.adv_norm:

• actor.reward_norm.mean_level — Controls mean subtraction aggregation. Use batch for standard GRPO or group for Dr.GRPO variants. CLI: --actor.reward_norm.mean_level=group.

• actor.reward_norm.std_level — Controls standard deviation normalization. Options include batch, group, or null (disabled). For Dr.GRPO, set to null to disable std normalization.

• actor.reward_norm.group_size — Defines the number of trajectories per group when using group-level normalization. Typically set to ${gconfig.n_samples} to match the number of sampled completions.

• actor.adv_norm.mean_leave1out — Enables leave-one-out averaging for advantages when set to true, implementing the RLOO (Remove Likelihood One Out) algorithm.

Importance Sampling Configuration

• actor.importance_sampling_level — Determines reward aggregation granularity. Set to "token" for standard GRPO (per-token rewards) or "sequence" for GSPO (Group Sequence Policy Optimization) which uses sequence-level rewards.

Default Configuration and Example Files

The repository ships a ready-to-run GRPO configuration for the GSM8K benchmark at examples/math/gsm8k_grpo.yaml. This file illustrates the default GRPO hyperparameters:

actor:
  eps_clip: 0.4                     # larger than PPO default

  temperature: ${gconfig.temperature}
  reward_scaling: 10.0
  reward_bias: -0.5
  kl_ctl: 0.0
  ppo_n_minibatches: 1
  recompute_logprob: true
  use_decoupled_loss: true
  behave_imp_weight_cap: 5.0
  reward_norm:
    mean_level: group
    std_level: group
    group_size: ${gconfig.n_samples}
  adv_norm:
    mean_level: batch
    std_level: batch
  weight_update_mode: xccl

These defaults can be overridden on the command line without editing the YAML. The documentation in docs/algorithms/grpo_series.md also summarizes how to switch algorithms using CLI overrides.

Setting Up a Custom GRPO Experiment

Follow this workflow to configure GRPO hyperparameters for your specific use case:

1. Start from the template (gsm8k_grpo.yaml) or create a new YAML file.

2. Adjust the actor block: Modify eps_clip, reward_norm, adv_norm, and other fields according to your target variant.

3. Launch the experiment with your desired scheduler. For a local run:

   python3 examples/math/gsm8k_rl.py \
       --config examples/math/gsm8k_grpo.yaml \
       scheduler.type=local \
       --actor.eps_clip 0.3 \
       --actor.reward_norm.mean_level=batch \
       --actor.reward_norm.std_level=batch

Programmatic Configuration

You can also build configurations programmatically using the dataclasses directly:

from areal.api.cli_args import GRPOConfig, load_expr_config

# Load the default YAML and modify fields in Python

cfg, _ = load_expr_config(
    ["--config", "examples/math/gsm8k_grpo.yaml"], GRPOConfig
)

# Change clipping and normalisation

cfg.actor.eps_clip = 0.3
cfg.actor.reward_norm.mean_level = "batch"
cfg.actor.reward_norm.std_level = "batch"
cfg.actor.adv_norm.mean_leave1out = True   # RLOO style

print(cfg)   # ready to pass to the trainer

Advanced Variant: Configuring Dr.GRPO

To switch from standard GRPO to the Dr.GRPO variant (which uses group-level advantage normalization), override the default batch-level settings:

python3 examples/math/gsm8k_rl.py \
    --config examples/math/gsm8k_grpo.yaml \
    scheduler.type=local \
    actor.adv_norm.mean_level=group \
    actor.adv_norm.std_level=null

This configuration matches the algorithmic description in docs/algorithms/grpo_series.md, implementing group-relative advantage estimation without standard deviation normalization.

Summary

• GRPO hyperparameters in AReaL are defined in areal/api/cli_args.py within three dataclasses: GRPOConfig, PPOActorConfig, and NormConfig.
• Key parameters include eps_clip for ratio clipping, reward_norm and adv_norm for controlling normalization levels (batch vs group), and importance_sampling_level for token vs sequence aggregation.
• Default configurations are provided in examples/math/gsm8k_grpo.yaml, which can be customized via CLI overrides without file modification.
• Advanced variants like Dr.GRPO are achieved by switching adv_norm.mean_level to group and disabling standard deviation normalization.

Frequently Asked Questions

What is the difference between GRPO and Dr.GRPO hyperparameters in AReaL?

The primary difference lies in the advantage normalization settings. Standard GRPO uses actor.adv_norm.mean_level=batch and actor.adv_norm.std_level=batch, normalizing across the entire batch. Dr.GRPO switches to actor.adv_norm.mean_level=group and typically sets actor.adv_norm.std_level=null, normalizing advantages only within each group of samples and removing standard deviation scaling.

How do I change the clipping bound for GRPO without editing the YAML file?

Use the command-line interface to override the eps_clip parameter. Append --actor.eps_clip 0.3 to your launch command to change the symmetric clipping bound from its default value. For asymmetric clipping, additionally specify --actor.eps_clip_higher 0.5 and ensure --actor.use_decoupled_loss=true is set.

Where are the default GRPO hyperparameters defined in the source code?

Default values are defined in areal/api/cli_args.py within two specific dataclasses. The PPOActorConfig class (around lines 979–1005) defines clipping bounds and importance sampling levels, while the NormConfig class (around lines 33–70) defines normalization parameters. The GRPOConfig class (lines 2155–2158) inherits from PPOConfig and serves as the top-level container.

Can I configure GRPO hyperparameters programmatically instead of using YAML?

Yes, you can instantiate and modify the configuration dataclasses directly in Python. Import GRPOConfig and load_expr_config from areal.api.cli_args, load your base configuration using load_expr_config, then modify fields such as cfg.actor.eps_clip or cfg.actor.reward_norm.mean_level before passing the configuration to the trainer. This approach is particularly useful for automated hyperparameter sweeps.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}