Understanding Asynchronous vs Synchronous RL Training Modes in AReaL

AReaL supports both synchronous (blocking) and asynchronous (decoupled) RL training modes, where synchronous mode ensures deterministic training by blocking until all workers complete rollouts with identical policy versions, while asynchronous mode maximizes GPU utilization by overlapping rollout generation, reward computation, and checkpointing across distributed workers.

AReaL (inclusionai/areal) is an open-source framework designed for large-scale reinforcement learning training. Understanding the distinction between asynchronous vs synchronous RL training modes in AReaL is essential for optimizing throughput and resource utilization, whether you are running small-scale experiments on a single node or distributed training across hundreds of GPUs.

Core Architectural Differences

The primary distinction between the two modes lies in how rollouts are generated relative to policy updates and how checkpoints are persisted.

Aspect	Synchronous Mode	Asynchronous (Decoupled) Mode
Rollout Generation	The training loop blocks until a full batch of rollouts completes. All workers use the same policy version simultaneously.	Rollouts generate concurrently with policy updates. The AsyncRewardWrapper and async_task_runner enable reward functions, environment steps, and checkpoint saves to run in separate event loops.
Loss Calculation	PPO/GRPO loss computes on synchronous policy-to-reference log-probabilities. The prox-log-p is either recomputed or taken from the rollout.	With use_decoupled_loss=True, the decoupled loss allows the policy that generated the trajectory (π_old) to differ from the current policy (π_new). Importance-weight correction (behave_imp_weight_*) can be applied per-token or per-sequence.
Checkpointing	Saver calls torch.distributed.checkpoint.save synchronously; training stalls until I/O completes.	Saver delegates to AsyncCheckpointManager. Checkpoints stage in pinned GPU memory, transfer to CPU in the background, and training continues immediately.
Configuration	SaverConfig.mode = "sync", MegatronEngineConfig.async_save = False, PPOActorConfig.use_decoupled_loss = False	SaverConfig.mode = "async", MegatronEngineConfig.async_save = True, PPOActorConfig.use_decoupled_loss = True
Staleness Handling	Not required; every worker sees identical weight versions.	StalenessManager tracks weight versions per rollout and discards overly stale samples before ppo_update.

How Synchronous Mode Works in AReaL

In synchronous mode, the training loop follows a strict sequential pattern: generate rollouts, compute rewards, calculate losses, update weights, and save checkpoints. Each step blocks until completion.

Rollout Generation and Loss Calculation

The workflow in areal/workflow/rlvr.py calls arun_episode directly, which invokes the engine and reward function synchronously via self.reward_fn. All workers participate in a collective operation, ensuring they use identical policy weights.

Loss computation occurs on synchronous policy-to-reference log-probabilities. The prox_logp is either recomputed during training (recompute_logprob=True) or retrieved from the rollout buffer (use_decoupled_loss=False).

Checkpointing Behavior

The Saver class in areal/utils/saver.py triggers torch.distributed.checkpoint.save directly. Training stalls until the distributed checkpoint write completes across all ranks, ensuring deterministic persistence but introducing I/O bottlenecks.

How Asynchronous Mode Works in AReaL

Asynchronous mode decouples rollout generation from policy updates, enabling parallel execution of reward computation, checkpointing, and training steps.

Decoupled Rollouts and AsyncRewardWrapper

In areal/workflow/rlvr.py, the workflow creates an AsyncRewardWrapper around the reward function:

self.async_reward_fn = AsyncRewardWrapper(self.reward_fn)

The workflow then awaits reward computation asynchronously:

reward = await self.async_reward_fn(prompt, **kwargs)

This allows the training loop to continue processing other rollouts while rewards compute in the background, either in a thread pool or via native async I/O.

Decoupled Loss and Importance Weighting

When PPOActorConfig.use_decoupled_loss=True, the trainer uses decoupled loss calculation. The policy that generated the trajectory (π_old) may differ from the current policy (π_new), requiring importance-weight correction.

The behave_imp_weight_mode and behave_imp_weight_cap parameters control whether correction applies per-token or per-sequence, adjusting gradients to account for the probability ratio between the behavior policy and the current policy.

Asynchronous Checkpointing

The AsyncCheckpointManager in areal/utils/async_checkpoint.py stages checkpoints in pinned GPU memory, then transfers them to CPU and storage in the background. Training continues immediately after the async save initiates, with consolidation across ranks handled on a separate thread.

Handling Staleness with StalenessManager

The StalenessManager in areal/infra/staleness_manager.py tracks the weight version used to generate each rollout. Before the PPO update in areal/trainer/ppo/actor.py, it discards samples that exceed the configured staleness threshold, ensuring training stability despite asynchronous execution.

Configuration Guide: Enabling Each Mode

Synchronous Mode (Default)

No special configuration is required. The default settings enforce synchronous execution:

# config.yaml

saver:
  mode: sync  # default

megatron:
  async_save: false  # default

actor:
  use_decoupled_loss: false  # default

Asynchronous Mode

Enable full asynchronous training by setting the following configuration options defined in areal/api/cli_args.py:

# config.yaml

saver:
  mode: async  # or "auto" to let AReaL detect support

megatron:
  async_save: true

actor:
  use_decoupled_loss: true
  prox_logp_method: loglinear  # skips forward pass for speed

  behave_imp_weight_mode: token_mask
  behave_imp_weight_cap: 5.0

The AsyncMode enum and PPOActorConfig dataclass in areal/api/cli_args.py expose these fields, allowing fine-grained control over the decoupled training pipeline.

Key Implementation Files

File	Purpose
areal/utils/async_checkpoint.py	AsyncMode enum and AsyncCheckpointManager for non-blocking saves
areal/utils/saver.py	High-level Saver API that routes to sync or async backends
areal/api/cli_args.py	Configuration dataclasses (SaverConfig, MegatronEngineConfig, PPOActorConfig)
areal/workflow/rlvr.py	Core rollout workflow with AsyncRewardWrapper integration
areal/infra/async_task_runner.py	Generic asyncio task queue for scheduler operations
areal/infra/staleness_manager.py	Weight version tracking and stale sample filtering
areal/infra/scheduler/local.py	LocalScheduler with async_call_engine method
areal/infra/scheduler/slurm.py	SlurmScheduler with async_call_engine method
areal/trainer/ppo/actor.py	PPO update logic with should_compute_prox_logp
tests/test_prox_approx.py	Unit tests for async vs sync prox-log-p behavior

Practical Code Examples

Enabling Async Checkpointing in a Training Script

# train.py

from areal.api.cli_args import SaverConfig, MegatronEngineConfig, PPOActorConfig
from areal.trainer.ppo.actor import PPOTrainer

cfg = {
    "saver": SaverConfig(mode="async", freq_epochs=0, freq_steps=1000, freq_secs=0),
    "megatron": MegatronEngineConfig(async_save=True),
    "actor": PPOActorConfig(
        use_decoupled_loss=True,
        prox_logp_method="loglinear",
        behave_imp_weight_mode="token_mask",
        behave_imp_weight_cap=5.0,
    ),
    # … other required fields …

}

trainer = PPOTrainer.from_config(cfg)
trainer.run()  # training loop now runs with async rollouts & checkpoints

The AsyncCheckpointManager creates separate Gloo process groups (self._pg and self._consolidation_pg) to avoid interfering with main training collectives, as implemented in areal/utils/async_checkpoint.py (lines 45-55).

Writing an Async Reward Function

# my_reward.py

import aiohttp
from areal.utils.async_reward_wrapper import AsyncRewardWrapper

async def fetch_score(prompt: str) -> float:
    async with aiohttp.ClientSession() as session:
        async with session.post("https://my-reward/api", json={"prompt": prompt}) as r:
            data = await r.json()
            return data["score"]

# Wrap it so the workflow can treat it uniformly

reward_fn = AsyncRewardWrapper(fetch_score)

In areal/workflow/rlvr.py (lines 78-95), the workflow creates self.async_reward_fn = AsyncRewardWrapper(self.reward_fn) and awaits it via reward = await self.async_reward_fn(prompt, **kwargs).

Using the Async Task Runner Directly

from areal.infra.async_task_runner import AsyncTaskRunner

runner = AsyncTaskRunner(max_concurrency=4)

@runner.task
async def heavy_compute(x):
    # Simulate GPU work

    await asyncio.sleep(0.2)
    return x * x

# Schedule many tasks without blocking the trainer

futs = [runner.submit(heavy_compute, i) for i in range(100)]
results = await asyncio.gather(*futs)  # runs concurrently

The AsyncTaskRunner in areal/infra/async_task_runner.py (lines 1-70) provides queue-based execution with pause/resume capabilities and configurable concurrency limits.

Switching the Prox-Log-P Method

actor_cfg = PPOActorConfig(
    use_decoupled_loss=True,
    prox_logp_method="loglinear",  # skips a forward pass on the new policy

    recompute_logprob=False,
)

# Inside PPOActor.update()

if actor_cfg.should_compute_prox_logp():
    # will return False -> no forward pass

    pass

The should_compute_prox_logp method in areal/api/cli_args.py (lines 1150-1156) determines whether a forward pass is necessary based on the ProxLogpMethod enum defined in areal/utils/constants.py.

Summary

• Synchronous mode blocks the training loop until all workers complete rollouts, compute rewards, and finish checkpointing, ensuring deterministic behavior but potentially leaving GPUs idle during I/O operations.
• Asynchronous mode decouples rollout generation from policy updates using AsyncRewardWrapper and AsyncTaskRunner, allowing concurrent reward computation, background checkpointing via AsyncCheckpointManager, and improved throughput for large-scale clusters.
• Decoupled loss (use_decoupled_loss=True) enables training on rollouts generated by older policy versions (π_old) while updating the current policy (π_new), with importance-weight correction handled by behave_imp_weight_mode and behave_imp_weight_cap.
• Staleness management is critical in async mode; the StalenessManager tracks weight versions per rollout and filters overly stale samples before the PPO update to maintain training stability.

Frequently Asked Questions

What is the main performance benefit of using asynchronous RL training in AReaL?

Asynchronous mode eliminates idle GPU time by overlapping rollout generation, reward computation, and checkpoint I/O. While synchronous training stalls during checkpoint writes and reward calculations, the async pipeline uses AsyncCheckpointManager and AsyncRewardWrapper to perform these operations in background threads, keeping the training loop continuously occupied.

How does AReaL prevent training instability from stale rollouts in asynchronous mode?

The StalenessManager in areal/infra/staleness_manager.py tracks the weight version used to generate each rollout. Before the PPO update in areal/trainer/ppo/actor.py, it discards samples that exceed the configured staleness threshold. Additionally, importance-weight correction via behave_imp_weight_mode and behave_imp_weight_cap adjusts gradients to account for the probability ratio between the behavior policy (π_old) and the current policy (π_new).

Can I mix synchronous rollouts with asynchronous checkpointing?

Yes. AReaL allows granular configuration through areal/api/cli_args.py. You can set saver.mode="async" and megatron.async_save=true to enable background checkpointing while keeping actor.use_decoupled_loss=false to maintain synchronous rollout generation. This hybrid approach reduces I/O blocking without introducing the complexity of decoupled loss calculation.

What configuration changes are required to switch from synchronous to fully asynchronous training?

To enable fully asynchronous training, modify your YAML configuration or Python dataclasses as follows: set saver.mode to "async" (or "auto" to auto-detect support), set megatron.async_save to true, and set actor.use_decoupled_loss to true. Optionally, set actor.prox_logp_method to "loglinear" to skip unnecessary forward passes, and configure actor.behave_imp_weight_mode and actor.behave_imp_weight_cap to manage importance weighting for stale rollouts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}