Configuring Multi-Turn Agentic Workflows with Reward Discounting in AReaL

AReaL enables multi-turn agentic workflows through the MultiTurnWorkflow class and OpenAI client wrapper, applying geometric reward discounting via the turn_discount parameter to propagate terminal rewards backward through conversation turns.

Configuring multi-turn agentic workflows with reward discounting is essential for training language models on complex, multi-step tasks. The AReaL framework (inclusionai/areal) provides native support for this pattern through configurable geometric decay factors that incentivize high-quality intermediate reasoning steps. This guide explains the architectural implementation and practical configuration patterns found in the source code.

Understanding the MultiTurnWorkflow Architecture

Core Components

The foundation of multi-turn support resides in the MultiTurnWorkflow class, which inherits from RolloutWorkflow and orchestrates conversational episodes. Located in [areal/workflow/multi_turn.py](https://github.com/inclusionai/areal/blob/main/areal/workflow/multi_turn.py), this class manages a sequence of turns where an LLM interacts with an environment before receiving a final reward.

The workflow accepts a turn_discount parameter in its constructor (line 28), representing the geometric discount factor γ where 0 < γ ≤ 1. The implementation validates this value immediately (lines 32-33) and stores it internally as self.turn_discount (line 39) for use during reward aggregation.

Geometric Discount Implementation

When an episode concludes, the workflow applies backward reward propagation through a geometric decay loop. The algorithm walks the turn sequence in reverse order, multiplying a running discount factor by self.turn_discount at each step:

discount = 1.0
for turn in reversed(self.turns):
    discount *= self.turn_discount          # geometric decay

    reward = float(turn.reward * discount)   # scaled reward for this turn

    turn.reward = reward

This logic appears in the reward aggregation loop (lines 76, 121-123), with final discounted values attached to each Interaction object returned by the workflow. For example, with turn_discount = 0.9 and a terminal reward of 2.0, the preceding turn receives 1.8 (0.9 × 2.0), the one before that receives 1.62 (0.9² × 2.0), and so on.

Configuring Discounting in the OpenAI Client

The apply_reward_discount Method

When using the OpenAI-based rollout client (areal.experimental.openai.client.OpenAIClient), reward discounting is applied explicitly via the apply_reward_discount method. Defined in [areal/experimental/openai/client.py](https://github.com/inclusionai/areal/blob/main/areal/experimental/openai/client.py) (lines 1100-1108), this method walks cached interactions in insertion order and applies the same geometric backward discount used by the workflow class.

The underlying cache implementation in [areal/experimental/openai/cache.py](https://github.com/inclusionai/areal/blob/main/areal/experimental/openai/cache.py) enforces a single-use policy (lines 68-70) to prevent double-counting, with the actual reward propagation occurring at line 85. The client method accepts a turn_discount: float = 1.0 argument and protects against multiple invocations via an internal flag.

Step-by-Step Configuration Workflow

Configuring a multi-turn episode with reward discounting follows this pattern:

1. Instantiate the workflow or client with the desired discount factor. For direct workflow usage, pass turn_discount to the constructor; for the OpenAI client, prepare to call the application method after generation.

2. Execute the episode by running the workflow or generating completions through the client. Each turn may invoke the LLM, collect observations, and eventually receive a terminal reward signal.

3. Apply discounting if using the OpenAI client wrapper by calling client.apply_reward_discount(turn_discount=0.9) after all completions are cached but before export.

4. Export the episode to retrieve interactions containing the discounted rewards, ready for training or evaluation.

Practical Code Examples

Example 1: Direct MultiTurnWorkflow Usage

from areal.workflow.multi_turn import MultiTurnWorkflow

workflow = MultiTurnWorkflow(
    env=my_env,
    policy=my_policy,
    max_turns=5,
    turn_discount=0.9,          # geometric discount factor

)

episode = workflow.run()

# episode.interactions contain backward-discounted rewards

Example 2: OpenAI Client Wrapper

from areal.experimental.openai.client import OpenAIClient

client = OpenAIClient(model="gpt-4o-mini")

# ... generate completions, automatically cached ...

client.apply_reward_discount(turn_discount=0.85)   # apply once

exported = client.export_interactions(style="individual", reward_discount=0.85)

Example 3: Experimental V2 Workflow

from areal.experimental.workflow.multi_turn_v2 import MultiTurnWorkflow

workflow = MultiTurnWorkflow(
    env=my_env,
    policy=my_policy,
    max_turns=3,
    turn_discount=0.95,
)
episode = workflow.run()

Summary

• Multi-turn workflows in AReaL are orchestrated by MultiTurnWorkflow in areal/workflow/multi_turn.py, which inherits from RolloutWorkflow.
• The turn_discount parameter (0 < γ ≤ 1) controls geometric decay applied when propagating terminal rewards backward through turn sequences.
• Discounting occurs automatically within the workflow class or explicitly via OpenAIClient.apply_reward_discount when using the experimental OpenAI client wrapper.
• The cache implementation enforces single-use protections to prevent duplicate discount applications during distributed training runs.
• Configuration requires instantiating the workflow with the discount factor or invoking the client method post-generation before exporting episodes.

Frequently Asked Questions

What is the valid range for the turn_discount parameter?

The turn_discount parameter accepts float values where 0 < γ ≤ 1. The constructor in areal/workflow/multi_turn.py validates this range (lines 32-33) and raises a ValueError if the provided value falls outside these bounds, ensuring geometric decay remains mathematically valid.

How does geometric discounting work in multi-turn episodes?

Geometric discounting propagates the final reward backward through the turn sequence by multiplying a running discount factor by turn_discount at each preceding step. According to the implementation in lines 121-123 of multi_turn.py, earlier turns receive exponentially smaller reward contributions, incentivizing the model to reach successful outcomes faster while still receiving credit for productive intermediate steps.

Can I apply reward discounting multiple times to the same episode?

No. The OpenAIClient implementation prevents multiple discount applications through a single-use flag enforced by the underlying Cache class (areal/experimental/openai/cache.py, lines 68-70). Attempting to call apply_reward_discount more than once will fail, protecting against accidental double-scaling of rewards during data preparation.

Where is the discount logic implemented for the OpenAI client?

The discount logic for the OpenAI client resides in two files: the application interface is in areal/experimental/openai/client.py (method definition at lines 1100-1108), while the actual propagation algorithm executes in areal/experimental/openai/cache.py (line 85). This separation allows the cache to maintain state integrity while the client provides the configuration API.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}