# Configuring Multi-Turn Agentic Workflows with Reward Discounting in AReaL

> Learn to configure multi-turn agentic workflows in AReaL. Apply geometric reward discounting for effective backward propagation of terminal rewards across conversation turns.

- Repository: [inclusionAI/areal](https://github.com/inclusionai/areal)
- Tags: how-to-guide
- Published: 2026-03-04

---

**AReaL enables multi-turn agentic workflows through the `MultiTurnWorkflow` class and OpenAI client wrapper, applying geometric reward discounting via the `turn_discount` parameter to propagate terminal rewards backward through conversation turns.**

Configuring multi-turn agentic workflows with reward discounting is essential for training language models on complex, multi-step tasks. The AReaL framework (`inclusionai/areal`) provides native support for this pattern through configurable geometric decay factors that incentivize high-quality intermediate reasoning steps. This guide explains the architectural implementation and practical configuration patterns found in the source code.

## Understanding the MultiTurnWorkflow Architecture

### Core Components

The foundation of multi-turn support resides in the **`MultiTurnWorkflow`** class, which inherits from `RolloutWorkflow` and orchestrates conversational episodes. Located in [[`areal/workflow/multi_turn.py`](https://github.com/inclusionai/areal/blob/main/areal/workflow/multi_turn.py)](https://github.com/inclusionai/areal/blob/main/areal/workflow/multi_turn.py), this class manages a sequence of turns where an LLM interacts with an environment before receiving a final reward.

The workflow accepts a **`turn_discount`** parameter in its constructor (line 28), representing the geometric discount factor γ where 0 < γ ≤ 1. The implementation validates this value immediately (lines 32-33) and stores it internally as `self.turn_discount` (line 39) for use during reward aggregation.

### Geometric Discount Implementation

When an episode concludes, the workflow applies backward reward propagation through a geometric decay loop. The algorithm walks the turn sequence in reverse order, multiplying a running discount factor by `self.turn_discount` at each step:

```python
discount = 1.0
for turn in reversed(self.turns):
    discount *= self.turn_discount          # geometric decay

    reward = float(turn.reward * discount)   # scaled reward for this turn

    turn.reward = reward

```

This logic appears in the reward aggregation loop (lines 76, 121-123), with final discounted values attached to each `Interaction` object returned by the workflow. For example, with `turn_discount = 0.9` and a terminal reward of 2.0, the preceding turn receives 1.8 (0.9 × 2.0), the one before that receives 1.62 (0.9² × 2.0), and so on.

## Configuring Discounting in the OpenAI Client

### The apply_reward_discount Method

When using the OpenAI-based rollout client (`areal.experimental.openai.client.OpenAIClient`), reward discounting is applied explicitly via the **`apply_reward_discount`** method. Defined in [[`areal/experimental/openai/client.py`](https://github.com/inclusionai/areal/blob/main/areal/experimental/openai/client.py)](https://github.com/inclusionai/areal/blob/main/areal/experimental/openai/client.py) (lines 1100-1108), this method walks cached interactions in insertion order and applies the same geometric backward discount used by the workflow class.

The underlying cache implementation in [[`areal/experimental/openai/cache.py`](https://github.com/inclusionai/areal/blob/main/areal/experimental/openai/cache.py)](https://github.com/inclusionai/areal/blob/main/areal/experimental/openai/cache.py) enforces a single-use policy (lines 68-70) to prevent double-counting, with the actual reward propagation occurring at line 85. The client method accepts a `turn_discount: float = 1.0` argument and protects against multiple invocations via an internal flag.

## Step-by-Step Configuration Workflow

Configuring a multi-turn episode with reward discounting follows this pattern:

1. **Instantiate the workflow** or client with the desired discount factor. For direct workflow usage, pass `turn_discount` to the constructor; for the OpenAI client, prepare to call the application method after generation.

2. **Execute the episode** by running the workflow or generating completions through the client. Each turn may invoke the LLM, collect observations, and eventually receive a terminal reward signal.

3. **Apply discounting** if using the OpenAI client wrapper by calling `client.apply_reward_discount(turn_discount=0.9)` after all completions are cached but before export.

4. **Export the episode** to retrieve interactions containing the discounted rewards, ready for training or evaluation.

## Practical Code Examples

### Example 1: Direct MultiTurnWorkflow Usage

```python
from areal.workflow.multi_turn import MultiTurnWorkflow

workflow = MultiTurnWorkflow(
    env=my_env,
    policy=my_policy,
    max_turns=5,
    turn_discount=0.9,          # geometric discount factor

)

episode = workflow.run()

# episode.interactions contain backward-discounted rewards

```

### Example 2: OpenAI Client Wrapper

```python
from areal.experimental.openai.client import OpenAIClient

client = OpenAIClient(model="gpt-4o-mini")

# ... generate completions, automatically cached ...

client.apply_reward_discount(turn_discount=0.85)   # apply once

exported = client.export_interactions(style="individual", reward_discount=0.85)

```

### Example 3: Experimental V2 Workflow

```python
from areal.experimental.workflow.multi_turn_v2 import MultiTurnWorkflow

workflow = MultiTurnWorkflow(
    env=my_env,
    policy=my_policy,
    max_turns=3,
    turn_discount=0.95,
)
episode = workflow.run()

```

## Summary

- **Multi-turn workflows** in AReaL are orchestrated by `MultiTurnWorkflow` in [`areal/workflow/multi_turn.py`](https://github.com/inclusionai/areal/blob/main/areal/workflow/multi_turn.py), which inherits from `RolloutWorkflow`.
- The **`turn_discount`** parameter (0 < γ ≤ 1) controls geometric decay applied when propagating terminal rewards backward through turn sequences.
- Discounting occurs automatically within the workflow class or explicitly via `OpenAIClient.apply_reward_discount` when using the experimental OpenAI client wrapper.
- The cache implementation enforces single-use protections to prevent duplicate discount applications during distributed training runs.
- Configuration requires instantiating the workflow with the discount factor or invoking the client method post-generation before exporting episodes.

## Frequently Asked Questions

### What is the valid range for the turn_discount parameter?

The `turn_discount` parameter accepts float values where 0 < γ ≤ 1. The constructor in [`areal/workflow/multi_turn.py`](https://github.com/inclusionai/areal/blob/main/areal/workflow/multi_turn.py) validates this range (lines 32-33) and raises a `ValueError` if the provided value falls outside these bounds, ensuring geometric decay remains mathematically valid.

### How does geometric discounting work in multi-turn episodes?

Geometric discounting propagates the final reward backward through the turn sequence by multiplying a running discount factor by `turn_discount` at each preceding step. According to the implementation in lines 121-123 of [`multi_turn.py`](https://github.com/inclusionai/areal/blob/main/multi_turn.py), earlier turns receive exponentially smaller reward contributions, incentivizing the model to reach successful outcomes faster while still receiving credit for productive intermediate steps.

### Can I apply reward discounting multiple times to the same episode?

No. The `OpenAIClient` implementation prevents multiple discount applications through a single-use flag enforced by the underlying `Cache` class ([`areal/experimental/openai/cache.py`](https://github.com/inclusionai/areal/blob/main/areal/experimental/openai/cache.py), lines 68-70). Attempting to call `apply_reward_discount` more than once will fail, protecting against accidental double-scaling of rewards during data preparation.

### Where is the discount logic implemented for the OpenAI client?

The discount logic for the OpenAI client resides in two files: the application interface is in [`areal/experimental/openai/client.py`](https://github.com/inclusionai/areal/blob/main/areal/experimental/openai/client.py) (method definition at lines 1100-1108), while the actual propagation algorithm executes in [`areal/experimental/openai/cache.py`](https://github.com/inclusionai/areal/blob/main/areal/experimental/openai/cache.py) (line 85). This separation allows the cache to maintain state integrity while the client provides the configuration API.