# Understanding Asynchronous vs Synchronous RL Training Modes in AReaL

> Explore synchronous vs asynchronous RL training in AReaL. Learn how to ensure deterministic training or maximize GPU utilization with our flexible modes.

- Repository: [inclusionAI/areal](https://github.com/inclusionai/areal)
- Tags: deep-dive
- Published: 2026-03-04

---

**AReaL supports both synchronous (blocking) and asynchronous (decoupled) RL training modes, where synchronous mode ensures deterministic training by blocking until all workers complete rollouts with identical policy versions, while asynchronous mode maximizes GPU utilization by overlapping rollout generation, reward computation, and checkpointing across distributed workers.**

AReaL (inclusionai/areal) is an open-source framework designed for large-scale reinforcement learning training. Understanding the distinction between asynchronous vs synchronous RL training modes in AReaL is essential for optimizing throughput and resource utilization, whether you are running small-scale experiments on a single node or distributed training across hundreds of GPUs.

## Core Architectural Differences

The primary distinction between the two modes lies in how rollouts are generated relative to policy updates and how checkpoints are persisted.

| Aspect | Synchronous Mode | Asynchronous (Decoupled) Mode |
|--------|------------------|-------------------------------|
| **Rollout Generation** | The training loop blocks until a full batch of rollouts completes. All workers use the same policy version simultaneously. | Rollouts generate concurrently with policy updates. The `AsyncRewardWrapper` and `async_task_runner` enable reward functions, environment steps, and checkpoint saves to run in separate event loops. |
| **Loss Calculation** | PPO/GRPO loss computes on synchronous policy-to-reference log-probabilities. The prox-log-p is either recomputed or taken from the rollout. | With `use_decoupled_loss=True`, the decoupled loss allows the policy that generated the trajectory (`π_old`) to differ from the current policy (`π_new`). Importance-weight correction (`behave_imp_weight_*`) can be applied per-token or per-sequence. |
| **Checkpointing** | `Saver` calls `torch.distributed.checkpoint.save` synchronously; training stalls until I/O completes. | `Saver` delegates to `AsyncCheckpointManager`. Checkpoints stage in pinned GPU memory, transfer to CPU in the background, and training continues immediately. |
| **Configuration** | `SaverConfig.mode = "sync"`, `MegatronEngineConfig.async_save = False`, `PPOActorConfig.use_decoupled_loss = False` | `SaverConfig.mode = "async"`, `MegatronEngineConfig.async_save = True`, `PPOActorConfig.use_decoupled_loss = True` |
| **Staleness Handling** | Not required; every worker sees identical weight versions. | `StalenessManager` tracks weight versions per rollout and discards overly stale samples before `ppo_update`. |

## How Synchronous Mode Works in AReaL

In synchronous mode, the training loop follows a strict sequential pattern: generate rollouts, compute rewards, calculate losses, update weights, and save checkpoints. Each step blocks until completion.

### Rollout Generation and Loss Calculation

The workflow in [`areal/workflow/rlvr.py`](https://github.com/inclusionai/areal/blob/main/areal/workflow/rlvr.py) calls `arun_episode` directly, which invokes the engine and reward function synchronously via `self.reward_fn`. All workers participate in a collective operation, ensuring they use identical policy weights.

Loss computation occurs on synchronous policy-to-reference log-probabilities. The `prox_logp` is either recomputed during training (`recompute_logprob=True`) or retrieved from the rollout buffer (`use_decoupled_loss=False`).

### Checkpointing Behavior

The `Saver` class in [`areal/utils/saver.py`](https://github.com/inclusionai/areal/blob/main/areal/utils/saver.py) triggers `torch.distributed.checkpoint.save` directly. Training stalls until the distributed checkpoint write completes across all ranks, ensuring deterministic persistence but introducing I/O bottlenecks.

## How Asynchronous Mode Works in AReaL

Asynchronous mode decouples rollout generation from policy updates, enabling parallel execution of reward computation, checkpointing, and training steps.

### Decoupled Rollouts and AsyncRewardWrapper

In [`areal/workflow/rlvr.py`](https://github.com/inclusionai/areal/blob/main/areal/workflow/rlvr.py), the workflow creates an `AsyncRewardWrapper` around the reward function:

```python
self.async_reward_fn = AsyncRewardWrapper(self.reward_fn)

```

The workflow then awaits reward computation asynchronously:

```python
reward = await self.async_reward_fn(prompt, **kwargs)

```

This allows the training loop to continue processing other rollouts while rewards compute in the background, either in a thread pool or via native async I/O.

### Decoupled Loss and Importance Weighting

When `PPOActorConfig.use_decoupled_loss=True`, the trainer uses **decoupled loss** calculation. The policy that generated the trajectory (`π_old`) may differ from the current policy (`π_new`), requiring importance-weight correction.

The `behave_imp_weight_mode` and `behave_imp_weight_cap` parameters control whether correction applies per-token or per-sequence, adjusting gradients to account for the probability ratio between the behavior policy and the current policy.

### Asynchronous Checkpointing

The `AsyncCheckpointManager` in [`areal/utils/async_checkpoint.py`](https://github.com/inclusionai/areal/blob/main/areal/utils/async_checkpoint.py) stages checkpoints in pinned GPU memory, then transfers them to CPU and storage in the background. Training continues immediately after the async save initiates, with consolidation across ranks handled on a separate thread.

### Handling Staleness with StalenessManager

The `StalenessManager` in [`areal/infra/staleness_manager.py`](https://github.com/inclusionai/areal/blob/main/areal/infra/staleness_manager.py) tracks the weight version used to generate each rollout. Before the PPO update in [`areal/trainer/ppo/actor.py`](https://github.com/inclusionai/areal/blob/main/areal/trainer/ppo/actor.py), it discards samples that exceed the configured staleness threshold, ensuring training stability despite asynchronous execution.

## Configuration Guide: Enabling Each Mode

### Synchronous Mode (Default)

No special configuration is required. The default settings enforce synchronous execution:

```yaml

# config.yaml

saver:
  mode: sync  # default

megatron:
  async_save: false  # default

actor:
  use_decoupled_loss: false  # default

```

### Asynchronous Mode

Enable full asynchronous training by setting the following configuration options defined in [`areal/api/cli_args.py`](https://github.com/inclusionai/areal/blob/main/areal/api/cli_args.py):

```yaml

# config.yaml

saver:
  mode: async  # or "auto" to let AReaL detect support

megatron:
  async_save: true

actor:
  use_decoupled_loss: true
  prox_logp_method: loglinear  # skips forward pass for speed

  behave_imp_weight_mode: token_mask
  behave_imp_weight_cap: 5.0

```

The `AsyncMode` enum and `PPOActorConfig` dataclass in [`areal/api/cli_args.py`](https://github.com/inclusionai/areal/blob/main/areal/api/cli_args.py) expose these fields, allowing fine-grained control over the decoupled training pipeline.

## Key Implementation Files

| File | Purpose |
|------|---------|
| [`areal/utils/async_checkpoint.py`](https://github.com/inclusionai/areal/blob/main/areal/utils/async_checkpoint.py) | `AsyncMode` enum and `AsyncCheckpointManager` for non-blocking saves |
| [`areal/utils/saver.py`](https://github.com/inclusionai/areal/blob/main/areal/utils/saver.py) | High-level `Saver` API that routes to sync or async backends |
| [`areal/api/cli_args.py`](https://github.com/inclusionai/areal/blob/main/areal/api/cli_args.py) | Configuration dataclasses (`SaverConfig`, `MegatronEngineConfig`, `PPOActorConfig`) |
| [`areal/workflow/rlvr.py`](https://github.com/inclusionai/areal/blob/main/areal/workflow/rlvr.py) | Core rollout workflow with `AsyncRewardWrapper` integration |
| [`areal/infra/async_task_runner.py`](https://github.com/inclusionai/areal/blob/main/areal/infra/async_task_runner.py) | Generic asyncio task queue for scheduler operations |
| [`areal/infra/staleness_manager.py`](https://github.com/inclusionai/areal/blob/main/areal/infra/staleness_manager.py) | Weight version tracking and stale sample filtering |
| [`areal/infra/scheduler/local.py`](https://github.com/inclusionai/areal/blob/main/areal/infra/scheduler/local.py) | `LocalScheduler` with `async_call_engine` method |
| [`areal/infra/scheduler/slurm.py`](https://github.com/inclusionai/areal/blob/main/areal/infra/scheduler/slurm.py) | `SlurmScheduler` with `async_call_engine` method |
| [`areal/trainer/ppo/actor.py`](https://github.com/inclusionai/areal/blob/main/areal/trainer/ppo/actor.py) | PPO update logic with `should_compute_prox_logp` |
| [`tests/test_prox_approx.py`](https://github.com/inclusionai/areal/blob/main/tests/test_prox_approx.py) | Unit tests for async vs sync prox-log-p behavior |

## Practical Code Examples

### Enabling Async Checkpointing in a Training Script

```python

# train.py

from areal.api.cli_args import SaverConfig, MegatronEngineConfig, PPOActorConfig
from areal.trainer.ppo.actor import PPOTrainer

cfg = {
    "saver": SaverConfig(mode="async", freq_epochs=0, freq_steps=1000, freq_secs=0),
    "megatron": MegatronEngineConfig(async_save=True),
    "actor": PPOActorConfig(
        use_decoupled_loss=True,
        prox_logp_method="loglinear",
        behave_imp_weight_mode="token_mask",
        behave_imp_weight_cap=5.0,
    ),
    # … other required fields …

}

trainer = PPOTrainer.from_config(cfg)
trainer.run()  # training loop now runs with async rollouts & checkpoints

```

The `AsyncCheckpointManager` creates separate Gloo process groups (`self._pg` and `self._consolidation_pg`) to avoid interfering with main training collectives, as implemented in [`areal/utils/async_checkpoint.py`](https://github.com/inclusionai/areal/blob/main/areal/utils/async_checkpoint.py) (lines 45-55).

### Writing an Async Reward Function

```python

# my_reward.py

import aiohttp
from areal.utils.async_reward_wrapper import AsyncRewardWrapper

async def fetch_score(prompt: str) -> float:
    async with aiohttp.ClientSession() as session:
        async with session.post("https://my-reward/api", json={"prompt": prompt}) as r:
            data = await r.json()
            return data["score"]

# Wrap it so the workflow can treat it uniformly

reward_fn = AsyncRewardWrapper(fetch_score)

```

In [`areal/workflow/rlvr.py`](https://github.com/inclusionai/areal/blob/main/areal/workflow/rlvr.py) (lines 78-95), the workflow creates `self.async_reward_fn = AsyncRewardWrapper(self.reward_fn)` and awaits it via `reward = await self.async_reward_fn(prompt, **kwargs)`.

### Using the Async Task Runner Directly

```python
from areal.infra.async_task_runner import AsyncTaskRunner

runner = AsyncTaskRunner(max_concurrency=4)

@runner.task
async def heavy_compute(x):
    # Simulate GPU work

    await asyncio.sleep(0.2)
    return x * x

# Schedule many tasks without blocking the trainer

futs = [runner.submit(heavy_compute, i) for i in range(100)]
results = await asyncio.gather(*futs)  # runs concurrently

```

The `AsyncTaskRunner` in [`areal/infra/async_task_runner.py`](https://github.com/inclusionai/areal/blob/main/areal/infra/async_task_runner.py) (lines 1-70) provides queue-based execution with pause/resume capabilities and configurable concurrency limits.

### Switching the Prox-Log-P Method

```python
actor_cfg = PPOActorConfig(
    use_decoupled_loss=True,
    prox_logp_method="loglinear",  # skips a forward pass on the new policy

    recompute_logprob=False,
)

# Inside PPOActor.update()

if actor_cfg.should_compute_prox_logp():
    # will return False -> no forward pass

    pass

```

The `should_compute_prox_logp` method in [`areal/api/cli_args.py`](https://github.com/inclusionai/areal/blob/main/areal/api/cli_args.py) (lines 1150-1156) determines whether a forward pass is necessary based on the `ProxLogpMethod` enum defined in [`areal/utils/constants.py`](https://github.com/inclusionai/areal/blob/main/areal/utils/constants.py).

## Summary

- **Synchronous mode** blocks the training loop until all workers complete rollouts, compute rewards, and finish checkpointing, ensuring deterministic behavior but potentially leaving GPUs idle during I/O operations.
- **Asynchronous mode** decouples rollout generation from policy updates using `AsyncRewardWrapper` and `AsyncTaskRunner`, allowing concurrent reward computation, background checkpointing via `AsyncCheckpointManager`, and improved throughput for large-scale clusters.
- **Decoupled loss** (`use_decoupled_loss=True`) enables training on rollouts generated by older policy versions (`π_old`) while updating the current policy (`π_new`), with importance-weight correction handled by `behave_imp_weight_mode` and `behave_imp_weight_cap`.
- **Staleness management** is critical in async mode; the `StalenessManager` tracks weight versions per rollout and filters overly stale samples before the PPO update to maintain training stability.

## Frequently Asked Questions

### What is the main performance benefit of using asynchronous RL training in AReaL?

Asynchronous mode eliminates idle GPU time by overlapping rollout generation, reward computation, and checkpoint I/O. While synchronous training stalls during checkpoint writes and reward calculations, the async pipeline uses `AsyncCheckpointManager` and `AsyncRewardWrapper` to perform these operations in background threads, keeping the training loop continuously occupied.

### How does AReaL prevent training instability from stale rollouts in asynchronous mode?

The `StalenessManager` in [`areal/infra/staleness_manager.py`](https://github.com/inclusionai/areal/blob/main/areal/infra/staleness_manager.py) tracks the weight version used to generate each rollout. Before the PPO update in [`areal/trainer/ppo/actor.py`](https://github.com/inclusionai/areal/blob/main/areal/trainer/ppo/actor.py), it discards samples that exceed the configured staleness threshold. Additionally, importance-weight correction via `behave_imp_weight_mode` and `behave_imp_weight_cap` adjusts gradients to account for the probability ratio between the behavior policy (`π_old`) and the current policy (`π_new`).

### Can I mix synchronous rollouts with asynchronous checkpointing?

Yes. AReaL allows granular configuration through [`areal/api/cli_args.py`](https://github.com/inclusionai/areal/blob/main/areal/api/cli_args.py). You can set `saver.mode="async"` and `megatron.async_save=true` to enable background checkpointing while keeping `actor.use_decoupled_loss=false` to maintain synchronous rollout generation. This hybrid approach reduces I/O blocking without introducing the complexity of decoupled loss calculation.

### What configuration changes are required to switch from synchronous to fully asynchronous training?

To enable fully asynchronous training, modify your YAML configuration or Python dataclasses as follows: set `saver.mode` to `"async"` (or `"auto"` to auto-detect support), set `megatron.async_save` to `true`, and set `actor.use_decoupled_loss` to `true`. Optionally, set `actor.prox_logp_method` to `"loglinear"` to skip unnecessary forward passes, and configure `actor.behave_imp_weight_mode` and `actor.behave_imp_weight_cap` to manage importance weighting for stale rollouts.