# Implementing DAPO (Direct Advantage Policy Optimization) Filtering Strategies in AReaL

> Implement DAPO filtering strategies in AReaL using configurable over-long sequence penalties. Inject negative reward shaping into PPO loss for optimized policy performance.

- Repository: [inclusionAI/areal](https://github.com/inclusionai/areal)
- Tags: how-to-guide
- Published: 2026-03-04

---

**AReaL implements Direct Advantage Policy Optimization (DAPO) filtering through a configurable over-long sequence penalty that injects negative reward shaping into the PPO loss function via the `reward_overlong_penalty` utility and `CliArgs` configuration flags.**

AReaL (Advanced Reinforcement Learning) provides a production-grade PPO pipeline that supports **Direct Advantage Policy Optimization (DAPO)** filtering strategies to control response length. By integrating reward shaping directly into the policy gradient step, AReaL enables fine-grained control over token generation budgets without modifying core loss computation logic.

## Core Architecture of DAPO Filtering

AReaL’s DAPO implementation follows a three-layer architecture: configuration via CLI arguments, reward shaping through pure functions, and automatic integration within the PPO actor.

### Configuration Layer in [`areal/api/cli_args.py`](https://github.com/inclusionai/areal/blob/main/areal/api/cli_args.py)

The DAPO configuration interface resides in [`areal/api/cli_args.py`](https://github.com/inclusionai/areal/blob/main/areal/api/cli_args.py) (line 1017), where the `CliArgs` dataclass exposes three critical fields:

```python
overlong_reward_penalty: bool = field(
    default=False,
    metadata={"help": "Penalty for overlong sequences. Used within DAPO."},
)
overlong_tokens: int | None = field(default=None, metadata={"help": "Maximum allowed token length."})
overlong_penalty_factor: float | None = field(
    default=None,
    metadata={"help": "Strength of the penalty (higher → stronger discouragement)."},
)

```

When `overlong_reward_penalty` is set to `True`, the trainer activates the penalty routine during each PPO update.

### Reward Shaping Implementation in [`areal/utils/functional/functional.py`](https://github.com/inclusionai/areal/blob/main/areal/utils/functional/functional.py)

The core penalty computation lives in [`areal/utils/functional/functional.py`](https://github.com/inclusionai/areal/blob/main/areal/utils/functional/functional.py) (line 474) within the `reward_overlong_penalty` function. This pure-Python helper calculates a negative reward proportional to sequence overflow:

```python
def reward_overlong_penalty(
    logprobs: torch.Tensor,
    seq_lengths: torch.Tensor,
    max_response_length: int,
    overlong_tokens: int,
    overlong_penalty_factor: float,
) -> torch.Tensor:
    """
    Apply a penalty for each token that pushes the output beyond
    `max_response_length`. The penalty grows linearly with the
    overflow amount and is scaled by `overlong_penalty_factor`.
    """
    # Compute how many tokens are over the limit.

    exceed_len = torch.clamp(seq_lengths - max_response_length, min=0)
    # Avoid division-by-zero when `overlong_tokens` is zero.

    overlong_reward = -exceed_len / overlong_tokens * overlong_penalty_factor
    # Clamp to ≤ 0 so the penalty never becomes a bonus.

    overlong_reward = torch.clamp(overlong_reward, max=0.0)
    # Add to the per-token reward score.

    reward_score_cur = logprobs.clone()
    reward_score_cur += overlong_reward.unsqueeze(-1)
    return reward_score_cur

```

The function returns a tensor of the same shape as `logprobs` with penalties incorporated, ready for downstream loss computation in SAPO or GRPO pipelines.

### Trainer Integration in [`areal/trainer/ppo/actor.py`](https://github.com/inclusionai/areal/blob/main/areal/trainer/ppo/actor.py)

[`areal/trainer/ppo/actor.py`](https://github.com/inclusionai/areal/blob/main/areal/trainer/ppo/actor.py) (line 136) contains the integration logic inside `PPOActor._step`. When DAPO is enabled, the actor applies the penalty before computing the policy-gradient loss:

```python
if self.config.overlong_reward_penalty:
    overlong_tokens = self.config.overlong_tokens
    overlong_penalty_factor = self.config.overlong_penalty_factor
    assert overlong_tokens is not None
    assert overlong_penalty_factor is not None
    data = reward_overlong_penalty(
        logprobs=data["logprobs"],
        seq_lengths=data["seq_lengths"],
        max_response_length=self.config.max_response_length,
        overlong_tokens=overlong_tokens,
        overlong_penalty_factor=overlong_penalty_factor,
    )
    # The `data` dict now contains the penalised rewards.

```

The modified `data` dictionary flows directly into existing loss pipelines, and the actor logs `overlong_reward` statistics for monitoring.

## Enabling DAPO Filtering via CLI

Activate DAPO filtering by passing the appropriate flags when launching training scripts:

```bash
uv run python examples/math/gsm8k_grpo.py \
    --overlong-reward-penalty \
    --overlong-tokens 1024 \
    --overlong-penalty-factor 0.7 \
    --max-response-length 1024

```

The `--max-response-length` parameter defines the hard limit that triggers the penalty calculation.

## Programmatic Usage and Customization

### Direct Function Usage

Import `reward_overlong_penalty` for custom reward manipulation outside the standard trainer flow:

```python
from areal.utils.functional import reward_overlong_penalty
import torch

logprobs = torch.randn(8, 1024)          # [batch, tokens]

seq_lengths = torch.tensor([900, 1025, 980, 1024, 1010, 950, 1030, 995])

penalised_rewards = reward_overlong_penalty(
    logprobs=logprobs,
    seq_lengths=seq_lengths,
    max_response_length=1024,
    overlong_tokens=1024,
    overlong_penalty_factor=0.5,
)

```

### Extending with Custom Penalty Schedules

For non-linear penalty schedules, subclass `PPOActor` and override the penalty application:

```python
from areal.trainer.ppo.actor import PPOActor
import torch

class ExponentialDapoActor(PPOActor):
    def _apply_overlong_penalty(self, data):
        overflow = torch.clamp(data["seq_lengths"] - self.config.max_response_length, min=0)
        factor = self.config.overlong_penalty_factor
        # Custom exponential scaling

        penalty = -(overflow ** 2) / self.config.overlong_tokens * factor
        data["logprobs"] += penalty.unsqueeze(-1)
        return data

    def _step(self, batch):
        data = super()._step(batch)
        if self.config.overlong_reward_penalty:
            data = self._apply_overlong_penalty(data)
        return data

```

## Summary

- **Configuration**: DAPO filtering is controlled via `CliArgs` in [`areal/api/cli_args.py`](https://github.com/inclusionai/areal/blob/main/areal/api/cli_args.py) using three fields: `overlong_reward_penalty`, `overlong_tokens`, and `overlong_penalty_factor`.
- **Implementation**: The `reward_overlong_penalty` function in [`areal/utils/functional/functional.py`](https://github.com/inclusionai/areal/blob/main/areal/utils/functional/functional.py) computes linear penalties proportional to sequence overflow.
- **Integration**: `PPOActor._step` in [`areal/trainer/ppo/actor.py`](https://github.com/inclusionai/areal/blob/main/areal/trainer/ppo/actor.py) automatically applies penalties when enabled, injecting statistics into training logs.
- **Flexibility**: The modular design allows custom penalty schedules through subclassing without modifying core PPO logic.

## Frequently Asked Questions

### What is DAPO filtering in AReaL?

Direct Advantage Policy Optimization (DAPO) filtering in AReaL refers to a reward-shaping technique that applies penalties to over-long sequences during PPO training. According to the AReaL source code, this is implemented as a negative reward term added to the per-token log probabilities before advantage estimation in the PPO actor.

### How does the overlong penalty function calculate negative rewards?

The `reward_overlong_penalty` function in [`areal/utils/functional/functional.py`](https://github.com/inclusionai/areal/blob/main/areal/utils/functional/functional.py) calculates penalties by clamping the difference between actual sequence lengths and `max_response_length` to positive values, then scaling by `-exceed_len / overlong_tokens * overlong_penalty_factor`. The result is clamped to a maximum of 0.0 to prevent positive rewards, ensuring the penalty never becomes a bonus.

### Can DAPO filtering be used with GRPO and other algorithms?

Yes. Because AReaL applies the penalty inside `PPOActor._step` before the loss computation, the modified rewards flow into any downstream loss function including GRPO, SAPO, or standard PPO objectives. The penalty acts as a pre-processing step on the reward tensor, making it compatible with all AReaL training pipelines.

### Which configuration flags control DAPO behavior?

Three CLI flags defined in [`areal/api/cli_args.py`](https://github.com/inclusionai/areal/blob/main/areal/api/cli_args.py) control DAPO: `--overlong-reward-penalty` enables the feature, `--overlong-tokens` sets the normalization denominator for penalty magnitude, and `--overlong-penalty-factor` controls the scaling strength. All three must be set for the penalty to activate during training.