# Implementing M2PO (Mixture of Memory Policy Optimization) in AReaL

> Implement M2PO (Mixture of Memory Policy Optimization) in AReaL. Stabilize policy updates by masking high-variance tokens using second-momentum statistics. Learn how M2PO integrates with PPO trainer.

- Repository: [inclusionAI/areal](https://github.com/inclusionai/areal)
- Tags: how-to-guide
- Published: 2026-03-04

---

**M2PO is built into AReaL’s PPO trainer and can be activated by setting the `m2_threshold` hyper‑parameter, which masks out high‑variance tokens based on second‑momentum statistics to stabilize policy updates.**

The AReaL repository (`inclusionai/areal`) ships with a production‑ready implementation of **Mixture of Memory Policy Optimization (M2PO)**, a variance‑reduction technique designed for stable RLHF training. Rather than requiring a separate algorithm implementation, M2PO is exposed as a configuration option within the standard PPO actor, allowing you to filter noisy gradient signals by suppressing tokens with anomalously large second‑momentum values.

## How M2PO Works in AReaL

AReaL’s M2PO implementation operates as a post‑processing mask applied to the PPO loss computation. After calculating proximal log‑probabilities, the trainer evaluates the squared difference between behavior‑policy and proximal‑policy log‑probabilities to identify and remove high‑variance tokens before back‑propagation.

### Configuration Interface

The entry point for M2PO is defined in [`areal/api/cli_args.py`](https://github.com/inclusionai/areal/blob/main/areal/api/cli_args.py), where the `ActorConfig` dataclass exposes the `m2_threshold` field:

```python

# areal/api/cli_args.py (lines 1001-1004)

@dataclass
class ActorConfig:
    # ... other fields ...

    m2_threshold: Optional[float] = None  # Second-momentum threshold for M2PO

```

Setting this value to a float (e.g., `0.05`) enables the algorithm; leaving it as `None` disables masking entirely.

### Core Masking Algorithm

The actual filtering logic resides in [`areal/trainer/ppo/actor.py`](https://github.com/inclusionai/areal/blob/main/areal/trainer/ppo/actor.py). When `m2_threshold` is configured, the trainer invokes `_apply_m2po_masking` immediately after resolving proximal log‑probabilities:

```python

# areal/trainer/ppo/actor.py (lines 96-99)

if self.cfg.m2_threshold is not None:
    loss_mask = self._apply_m2po_masking(
        old_logp, prox_logp, loss_mask, self.cfg.m2_threshold
    )

```

The `_apply_m2po_masking` function computes the per‑token second‑momentum as `m2 = (old_logp - prox_logp)²`, then iteratively removes the highest‑momentum tokens until the mean second‑momentum across the remaining active tokens falls below the threshold. A helper function `_get_m2po_loss_mask` implements the "keep‑largest‑until‑threshold" logic with proper handling for empty inputs.

## Enabling M2PO in Your Training Pipeline

You can activate M2PO through YAML configuration, JSON configuration, or direct CLI arguments without modifying source code.

### YAML Configuration

Add the `m2_threshold` key under the actor configuration block:

```yaml
actor:
  use_sapo_loss: false          # Optional; M2PO works with standard PPO loss

  m2_threshold: 0.05            # Enable M2PO with threshold 0.05

  eps_clip: 0.2
  # ... other PPO hyper-parameters ...

```

### Command‑Line Interface

Pass the parameter directly when launching training:

```bash
python -m areal.train \
  --actor.use_sapo_loss=false \
  --actor.m2_threshold=0.05 \
  --actor.eps_clip=0.2

```

### Programmatic Configuration

When building configurations in Python, instantiate `ActorConfig` with the threshold defined:

```python
from areal.api.cli_args import ActorConfig

cfg = ActorConfig(
    use_sapo_loss=False,
    m2_threshold=0.05,           # Enable M2PO filtering

    eps_clip=0.2,
    # ... other params ...

)

```

The `PPOActor` trainer automatically applies the mask during `train_step()` when this configuration is present.

## Understanding the Algorithm Implementation

M2PO reduces gradient noise by targeting tokens where the policy update magnitude (measured by second‑moment) is statistically anomalous. The algorithm proceeds as follows:

1. **Compute Second‑Momentum**: For each token, calculate `m2 = (old_logp - prox_logp)²`
2. **Sort and Filter**: Sort `m2` values of active tokens in descending order
3. **Threshold Truncation**: Remove the highest‑momentum tokens until `mean(m2) < m2_threshold`
4. **Apply Mask**: Return the filtered `loss_mask` to the loss function (SAPO or standard PPO)

This approach is particularly effective in long‑sequence or high‑entropy environments where individual tokens can inject destabilizing variance into policy gradients.

### Visualizing the Mask Effect

To inspect how M2PO affects your batch, you can simulate the masking logic:

```python
import torch
from areal.trainer.ppo.actor import _apply_m2po_masking  # Internal API

old_logp = torch.randn(2, 8)           # Behavior log‑probs (batch_size, seq_len)

prox_logp = torch.randn(2, 8)          # Proximal log‑probs after clipping

loss_mask = torch.ones_like(old_logp).bool()

filtered_mask = _apply_m2po_masking(
    old_logp, prox_logp, loss_mask, m2_threshold=0.05
)

print(f"Original token count: {loss_mask.sum().item()}")
print(f"Filtered token count: {filtered_mask.sum().item()}")

```

The output will show a reduced token count, reflecting the removal of high‑variance positions that exceeded the second‑momentum threshold.

## Summary

- **M2PO is native to AReaL**: The algorithm is pre‑implemented in [`areal/trainer/ppo/actor.py`](https://github.com/inclusionai/areal/blob/main/areal/trainer/ppo/actor.py) and requires no external dependencies.
- **Activation is configuration‑only**: Set `actor.m2_threshold` to a float value (e.g., `0.05`) via CLI or config file.
- **Variance reduction mechanism**: The algorithm masks tokens where `(old_logp - prox_logp)²` is anomalously high, reducing gradient noise during RLHF training.
- **Compatible with standard losses**: M2PO works with both SAPO and vanilla PPO loss functions, applying the mask before the final loss aggregation.

## Frequently Asked Questions

### What is the recommended value for `m2_threshold`?

Typical values range between **0.01 and 0.1**, depending on your task's entropy and sequence length. Start with `0.05` and monitor the ratio of masked tokens; if too many tokens are being filtered (more than 20%), increase the threshold. Conversely, if gradient instability persists, decrease the threshold to be more aggressive.

### Does M2PO work with the SAPO loss function?

Yes. The masking step occurs before the loss computation branches into SAPO or standard PPO paths. In [`areal/trainer/ppo/actor.py`](https://github.com/inclusionai/areal/blob/main/areal/trainer/ppo/actor.py), the `_apply_m2po_masking` function returns a filtered `loss_mask` that is passed to whichever loss implementation is configured via `use_sapo_loss`.

### Can I use M2PO with custom loss implementations?

While `_apply_m2po_masking` is an internal method, you can replicate its logic in custom trainers by importing the helper from [`areal/trainer/ppo/actor.py`](https://github.com/inclusionai/areal/blob/main/areal/trainer/ppo/actor.py). The function signature accepts `old_logp`, `prox_logp`, `loss_mask`, and `m2_threshold`, returning a boolean tensor of the same shape as your input mask.

### How does M2PO differ from standard PPO clipping?

Standard PPO uses `eps_clip` to bound the probability ratio, but it still computes gradients over all tokens. **M2PO augments this** by completely zeroing out the loss contribution from high‑variance tokens based on second‑moment statistics, providing an additional layer of gradient filtering beyond ratio clipping.