# How AgentScope Supports Model Finetuning with Reinforcement Learning: A Complete Guide to the Tuner Sub‑Package

> AgentScope enables model finetuning with reinforcement learning using its tuner sub-package. Learn how to leverage its components and the Trinity-RFT engine for efficient RL.

- Repository: [AgentScope-AI/agentscope](https://github.com/agentscope-ai/agentscope)
- Tags: how-to-guide
- Published: 2026-03-09

---

**AgentScope supports model finetuning with reinforcement learning through a dedicated `tuner` sub‑package that decomposes training into three pluggable components—workflow functions, judge functions, and the `tune` driver—while leveraging the Trinity‑RFT engine for distributed RL updates.**

The `agentscope-ai/agentscope` repository provides a Python‑native framework for turning any agent into a trainable reinforcement learning (RL) component. Instead of monolithic training scripts, the library separates concerns into async, composable functions that connect to the Trinity‑RFT backend for scalable GRPO and PPO updates.

## The Three‑Component Architecture

The design philosophy behind AgentScope’s RL finetuning capability is the strict separation of three responsibilities. This modularity allows developers to swap out evaluation logic, agent architectures, or optimization algorithms without rewriting boilerplate infrastructure code.

### Workflow Function

The **workflow function** executes your agent on a specific task and returns a `WorkflowOutput` dataclass. Located in [`src/agentscope/tuner/_workflow.py`](https://github.com/agentscope-ai/agentscope/blob/main/src/agentscope/tuner/_workflow.py), this signature guarantees a consistent async interface:

```python
Callable[
    [Dict, ChatModelBase, Optional[Dict[str, ChatModelBase]], Optional[Logger]],
    Awaitable[WorkflowOutput]
]

```

The `WorkflowOutput` class captures the raw `response`, an optional scalar `reward`, and auxiliary `metrics`. If the workflow itself computes a reward (e.g., environment feedback), the judge step can be bypassed.

### Judge Function

The **judge function** evaluates the quality of the workflow’s response and computes a scalar reward signal. Defined in [`src/agentscope/tuner/_judge.py`](https://github.com/agentscope-ai/agentscope/blob/main/src/agentscope/tuner/_judge.py), the `JudgeType` signature accepts the task dictionary, the workflow’s response, and optional auxiliary models:

```python
Callable[
    [Dict, Any, Optional[Dict[str, ChatModelBase]]],
    Awaitable[JudgeOutput]
]

```

The `JudgeOutput` dataclass wraps a single `reward: float` field. This design supports arbitrary evaluation logic—ranging from exact string matching to LLM‑as‑Judge patterns—without coupling the scoring logic to the training loop.

### Tuning Driver (`tune`)

The **`tune` function** in [`src/agentscope/tuner/_tune.py`](https://github.com/agentscope-ai/agentscope/blob/main/src/agentscope/tuner/_tune.py) serves as the entry point. It orchestrates dataset loading, model initialization, and algorithm selection, then translates these components into a YAML configuration consumed by Trinity‑RFT. The driver invokes `run_stage` from `trinity.cli.launcher` to handle distributed training, checkpointing, and experiment tracking (TensorBoard, WandB).

## Core Types and Configuration Classes

AgentScope exposes several configuration dataclasses in the `tuner` sub‑package to declaratively define the training environment.

### WorkflowOutput and WorkflowType

`WorkflowOutput` (from [`_workflow.py`](https://github.com/agentscope-ai/agentscope/blob/main/_workflow.py)) is the mandatory return type for all workflow functions. It standardizes how responses and optional rewards flow back to the RL optimizer.

### JudgeOutput and JudgeType

`JudgeOutput` (from [`_judge.py`](https://github.com/agentscope-ai/agentscope/blob/main/_judge.py)) is a lightweight container ensuring the reward signal is explicitly typed as a float. The `JudgeType` protocol allows the tuner to accept both simple heuristics and complex multi‑model evaluation pipelines.

### AlgorithmConfig

`AlgorithmConfig` (in [`src/agentscope/tuner/_algorithm.py`](https://github.com/agentscope-ai/agentscope/blob/main/src/agentscope/tuner/_algorithm.py)) specifies the RL algorithm and hyperparameters. Supported values for `algorithm_type` include `"multi_step_grpo"` and variants of **PPO**, along with fields for `learning_rate`, `batch_size`, and `group_size`.

### TunerModelConfig and DatasetConfig

`TunerModelConfig` (from [`_model.py`](https://github.com/agentscope-ai/agentscope/blob/main/_model.py)) encapsulates model‑specific settings such as `model_path`, LoRA adapter locations, and `max_model_len`. `DatasetConfig` (from [`_dataset.py`](https://github.com/agentscope-ai/agentscope/blob/main/_dataset.py)) wraps HuggingFace datasets, providing a `preview()` method for sanity checks before launching distributed jobs.

## The RL Training Loop

Under the hood, the `tune` driver constructs a training loop that bridges AgentScope’s high‑level abstractions to Trinity‑RFT’s optimized kernels:

1. **Sample** a task batch from the `DatasetConfig`.
2. **Execute** the workflow function to generate agent responses.
3. **Evaluate** via the judge function to produce a scalar reward.
4. **Update** the model using the configured RL algorithm (GRPO or PPO) based on the reward signal.

Because both workflow and judge functions are async, they can perform I/O‑bound operations—such as calling external APIs or running multi‑agent simulations—without blocking the training orchestrator.

## Complete Implementation Example

The following example mirrors the official tutorial at [`docs/tutorial/en/src/task_tuner.py`](https://github.com/agentscope-ai/agentscope/blob/main/docs/tutorial/en/src/task_tuner.py), demonstrating a ReAct agent workflow, a string‑match judge, and a GRPO training configuration.

### Creating the Workflow

Define an async workflow that instantiates a `ReActAgent` and returns a `WorkflowOutput`:

```python

# workflow.py

from typing import Dict, Optional
from agentscope.agent import ReActAgent
from agentscope.formatter import OpenAIChatFormatter
from agentscope.message import Msg
from agentscope.model import ChatModelBase
from agentscope.tuner import WorkflowOutput

async def my_workflow(
    task: Dict,
    model: ChatModelBase,
    auxiliary_models: Optional[Dict[str, ChatModelBase]] = None,
) -> WorkflowOutput:
    """Run a ReAct agent on a math question."""
    agent = ReActAgent(
        name="react_agent",
        sys_prompt="You are a helpful math problem‑solving agent.",
        model=model,
        formatter=OpenAIChatFormatter(),
    )
    response = await agent.reply(
        msg=Msg("user", task["question"], role="user")
    )
    return WorkflowOutput(response=response)

```

### Defining the Judge

Implement a judge that rewards exact matches against ground‑truth answers:

```python

# judge.py

from typing import Any, Dict, Optional
from agentscope.tuner import JudgeOutput

async def my_judge(
    task: Dict,
    response: Any,
    auxiliary_models: Optional[Dict[str, ChatModelBase]] = None,
) -> JudgeOutput:
    """Reward 1.0 if the ground‑truth answer appears in the response."""
    reward = 1.0 if task["answer"] in response.get_text_content() else 0.0
    return JudgeOutput(reward=reward)

```

### Launching the Tuning Job

Wire everything together using `tune`, `DatasetConfig`, `TunerModelConfig`, and `AlgorithmConfig`:

```python

# tune.py

import asyncio
from agentscope.tuner import (
    tune,
    DatasetConfig,
    TunerModelConfig,
    AlgorithmConfig,
)

if __name__ == "__main__":
    # 1️⃣ Load dataset (HuggingFace format)

    dataset = DatasetConfig(path="my_dataset", split="train")

    # 2️⃣ Configure the trainable model (e.g., Qwen‑3 LoRA)

    model_cfg = TunerModelConfig(
        model_path="Qwen/Qwen3-0.6B",
        max_model_len=16384,
    )

    # 3️⃣ Choose RL algorithm (GRPO in this case)

    algo_cfg = AlgorithmConfig(
        algorithm_type="multi_step_grpo",
        group_size=8,
        batch_size=32,
        learning_rate=1e-6,
    )

    # 4️⃣ Launch the tuning job

    tune(
        workflow_func=__import__("workflow").my_workflow,
        judge_func=__import__("judge").my_judge,
        train_dataset=dataset,
        model=model_cfg,
        algorithm=algo_cfg,
        project_name="my_rl_finetune",
        experiment_name="math_react",
    )

```

**Prerequisites:** Ensure `trinity-rft` is installed (`pip install trinity-rft`) and a Ray cluster is active (`ray start --head`). Environment variables for underlying LLM API keys must be exported before execution.

## Integration with Trinity‑RFT

AgentScope does not reimplement RL kernels from scratch. Instead, the `tune` function in [`src/agentscope/tuner/_tune.py`](https://github.com/agentscope-ai/agentscope/blob/main/src/agentscope/tuner/_tune.py) generates a YAML configuration understood by **Trinity‑RFT**, a specialized library for large‑scale RL finetuning. This delegation provides:

- **Distributed training** across multiple GPUs via Ray.
- **Automatic checkpointing** and model versioning.
- **Built‑in monitoring** integrations for TensorBoard and Weights & Biases.

The decoupling allows AgentScope to focus on agent semantics while Trinity‑RFT handles gradient accumulation, advantage estimation, and parameter updates for algorithms like GRPO and PPO.

## Summary

- AgentScope enables **model finetuning with reinforcement learning** through the `tuner` sub‑package, separating workflow execution, reward judgment, and training orchestration.
- The **`WorkflowOutput`** and **`JudgeOutput`** dataclasses in [`_workflow.py`](https://github.com/agentscope-ai/agentscope/blob/main/_workflow.py) and [`_judge.py`](https://github.com/agentscope-ai/agentscope/blob/main/_judge.py) standardize how agent responses and rewards flow into the RL loop.
- **`AlgorithmConfig`** supports modern algorithms including **GRPO** and **PPO**, configured via the `algorithm_type` field.
- The **`tune`** function bridges AgentScope to Trinity‑RFT, automatically handling distributed training and experiment tracking.
- All components use async signatures, allowing complex evaluation logic and multi‑agent simulations without blocking the optimizer.

## Frequently Asked Questions

### What reinforcement learning algorithms does AgentScope support?

AgentScope supports **GRPO** (Group Relative Policy Optimization) and **PPO** (Proximal Policy Optimization) through the `AlgorithmConfig` class. The `algorithm_type` field accepts values such as `"multi_step_grpo"` to specify which optimizer Trinity‑RFT will use during the training loop.

### Do I need to implement both a workflow and a judge function?

You must always provide a **workflow function**, but the **judge function** is optional if your workflow returns a reward directly in the `WorkflowOutput`. However, separating the judge is recommended for complex evaluation logic or when using LLM‑as‑Judge patterns that require auxiliary models.

### Can I use custom models or LoRA adapters with the tuner?

Yes. The `TunerModelConfig` class in [`src/agentscope/tuner/_model.py`](https://github.com/agentscope-ai/agentscope/blob/main/src/agentscope/tuner/_model.py) accepts parameters like `model_path` and LoRA configuration options. You can point to local checkpoints or HuggingFace model IDs, and the tuner will load them via Trinity‑RFT for the RL update steps.

### What infrastructure is required to run distributed RL training?

AgentScope’s tuner requires **Trinity‑RFT** and a **Ray** cluster. You must install `trinity-rft` via pip and initialize Ray with `ray start --head` (or connect to an existing cluster). The `tune` function then delegates distributed training, checkpointing, and logging to these underlying services.