How AgentScope Supports Model Finetuning with Reinforcement Learning: A Complete Guide to the Tuner Sub‑Package

AgentScope supports model finetuning with reinforcement learning through a dedicated tuner sub‑package that decomposes training into three pluggable components—workflow functions, judge functions, and the tune driver—while leveraging the Trinity‑RFT engine for distributed RL updates.

The agentscope-ai/agentscope repository provides a Python‑native framework for turning any agent into a trainable reinforcement learning (RL) component. Instead of monolithic training scripts, the library separates concerns into async, composable functions that connect to the Trinity‑RFT backend for scalable GRPO and PPO updates.

The Three‑Component Architecture

The design philosophy behind AgentScope’s RL finetuning capability is the strict separation of three responsibilities. This modularity allows developers to swap out evaluation logic, agent architectures, or optimization algorithms without rewriting boilerplate infrastructure code.

Workflow Function

The workflow function executes your agent on a specific task and returns a WorkflowOutput dataclass. Located in src/agentscope/tuner/_workflow.py, this signature guarantees a consistent async interface:

Callable[
    [Dict, ChatModelBase, Optional[Dict[str, ChatModelBase]], Optional[Logger]],
    Awaitable[WorkflowOutput]
]

The WorkflowOutput class captures the raw response, an optional scalar reward, and auxiliary metrics. If the workflow itself computes a reward (e.g., environment feedback), the judge step can be bypassed.

Judge Function

The judge function evaluates the quality of the workflow’s response and computes a scalar reward signal. Defined in src/agentscope/tuner/_judge.py, the JudgeType signature accepts the task dictionary, the workflow’s response, and optional auxiliary models:

Callable[
    [Dict, Any, Optional[Dict[str, ChatModelBase]]],
    Awaitable[JudgeOutput]
]

The JudgeOutput dataclass wraps a single reward: float field. This design supports arbitrary evaluation logic—ranging from exact string matching to LLM‑as‑Judge patterns—without coupling the scoring logic to the training loop.

Tuning Driver (tune)

The tune function in src/agentscope/tuner/_tune.py serves as the entry point. It orchestrates dataset loading, model initialization, and algorithm selection, then translates these components into a YAML configuration consumed by Trinity‑RFT. The driver invokes run_stage from trinity.cli.launcher to handle distributed training, checkpointing, and experiment tracking (TensorBoard, WandB).

Core Types and Configuration Classes

AgentScope exposes several configuration dataclasses in the tuner sub‑package to declaratively define the training environment.

WorkflowOutput and WorkflowType

WorkflowOutput (from _workflow.py) is the mandatory return type for all workflow functions. It standardizes how responses and optional rewards flow back to the RL optimizer.

JudgeOutput and JudgeType

JudgeOutput (from _judge.py) is a lightweight container ensuring the reward signal is explicitly typed as a float. The JudgeType protocol allows the tuner to accept both simple heuristics and complex multi‑model evaluation pipelines.

AlgorithmConfig

AlgorithmConfig (in src/agentscope/tuner/_algorithm.py) specifies the RL algorithm and hyperparameters. Supported values for algorithm_type include "multi_step_grpo" and variants of PPO, along with fields for learning_rate, batch_size, and group_size.

TunerModelConfig and DatasetConfig

TunerModelConfig (from _model.py) encapsulates model‑specific settings such as model_path, LoRA adapter locations, and max_model_len. DatasetConfig (from _dataset.py) wraps HuggingFace datasets, providing a preview() method for sanity checks before launching distributed jobs.

The RL Training Loop

Under the hood, the tune driver constructs a training loop that bridges AgentScope’s high‑level abstractions to Trinity‑RFT’s optimized kernels:

1. Sample a task batch from the DatasetConfig.
2. Execute the workflow function to generate agent responses.
3. Evaluate via the judge function to produce a scalar reward.
4. Update the model using the configured RL algorithm (GRPO or PPO) based on the reward signal.

Because both workflow and judge functions are async, they can perform I/O‑bound operations—such as calling external APIs or running multi‑agent simulations—without blocking the training orchestrator.

Complete Implementation Example

The following example mirrors the official tutorial at docs/tutorial/en/src/task_tuner.py, demonstrating a ReAct agent workflow, a string‑match judge, and a GRPO training configuration.

Creating the Workflow

Define an async workflow that instantiates a ReActAgent and returns a WorkflowOutput:

# workflow.py

from typing import Dict, Optional
from agentscope.agent import ReActAgent
from agentscope.formatter import OpenAIChatFormatter
from agentscope.message import Msg
from agentscope.model import ChatModelBase
from agentscope.tuner import WorkflowOutput

async def my_workflow(
    task: Dict,
    model: ChatModelBase,
    auxiliary_models: Optional[Dict[str, ChatModelBase]] = None,
) -> WorkflowOutput:
    """Run a ReAct agent on a math question."""
    agent = ReActAgent(
        name="react_agent",
        sys_prompt="You are a helpful math problem‑solving agent.",
        model=model,
        formatter=OpenAIChatFormatter(),
    )
    response = await agent.reply(
        msg=Msg("user", task["question"], role="user")
    )
    return WorkflowOutput(response=response)

Defining the Judge

Implement a judge that rewards exact matches against ground‑truth answers:

# judge.py

from typing import Any, Dict, Optional
from agentscope.tuner import JudgeOutput

async def my_judge(
    task: Dict,
    response: Any,
    auxiliary_models: Optional[Dict[str, ChatModelBase]] = None,
) -> JudgeOutput:
    """Reward 1.0 if the ground‑truth answer appears in the response."""
    reward = 1.0 if task["answer"] in response.get_text_content() else 0.0
    return JudgeOutput(reward=reward)

Launching the Tuning Job

Wire everything together using tune, DatasetConfig, TunerModelConfig, and AlgorithmConfig:

# tune.py

import asyncio
from agentscope.tuner import (
    tune,
    DatasetConfig,
    TunerModelConfig,
    AlgorithmConfig,
)

if __name__ == "__main__":
    # 1️⃣ Load dataset (HuggingFace format)

    dataset = DatasetConfig(path="my_dataset", split="train")

    # 2️⃣ Configure the trainable model (e.g., Qwen‑3 LoRA)

    model_cfg = TunerModelConfig(
        model_path="Qwen/Qwen3-0.6B",
        max_model_len=16384,
    )

    # 3️⃣ Choose RL algorithm (GRPO in this case)

    algo_cfg = AlgorithmConfig(
        algorithm_type="multi_step_grpo",
        group_size=8,
        batch_size=32,
        learning_rate=1e-6,
    )

    # 4️⃣ Launch the tuning job

    tune(
        workflow_func=__import__("workflow").my_workflow,
        judge_func=__import__("judge").my_judge,
        train_dataset=dataset,
        model=model_cfg,
        algorithm=algo_cfg,
        project_name="my_rl_finetune",
        experiment_name="math_react",
    )

Prerequisites: Ensure trinity-rft is installed (pip install trinity-rft) and a Ray cluster is active (ray start --head). Environment variables for underlying LLM API keys must be exported before execution.

Integration with Trinity‑RFT

AgentScope does not reimplement RL kernels from scratch. Instead, the tune function in src/agentscope/tuner/_tune.py generates a YAML configuration understood by Trinity‑RFT, a specialized library for large‑scale RL finetuning. This delegation provides:

• Distributed training across multiple GPUs via Ray.
• Automatic checkpointing and model versioning.
• Built‑in monitoring integrations for TensorBoard and Weights & Biases.

The decoupling allows AgentScope to focus on agent semantics while Trinity‑RFT handles gradient accumulation, advantage estimation, and parameter updates for algorithms like GRPO and PPO.

Summary

• AgentScope enables model finetuning with reinforcement learning through the tuner sub‑package, separating workflow execution, reward judgment, and training orchestration.
• The WorkflowOutput and JudgeOutput dataclasses in _workflow.py and _judge.py standardize how agent responses and rewards flow into the RL loop.
• AlgorithmConfig supports modern algorithms including GRPO and PPO, configured via the algorithm_type field.
• The tune function bridges AgentScope to Trinity‑RFT, automatically handling distributed training and experiment tracking.
• All components use async signatures, allowing complex evaluation logic and multi‑agent simulations without blocking the optimizer.

Frequently Asked Questions

What reinforcement learning algorithms does AgentScope support?

AgentScope supports GRPO (Group Relative Policy Optimization) and PPO (Proximal Policy Optimization) through the AlgorithmConfig class. The algorithm_type field accepts values such as "multi_step_grpo" to specify which optimizer Trinity‑RFT will use during the training loop.

Do I need to implement both a workflow and a judge function?

You must always provide a workflow function, but the judge function is optional if your workflow returns a reward directly in the WorkflowOutput. However, separating the judge is recommended for complex evaluation logic or when using LLM‑as‑Judge patterns that require auxiliary models.

Can I use custom models or LoRA adapters with the tuner?

Yes. The TunerModelConfig class in src/agentscope/tuner/_model.py accepts parameters like model_path and LoRA configuration options. You can point to local checkpoints or HuggingFace model IDs, and the tuner will load them via Trinity‑RFT for the RL update steps.

What infrastructure is required to run distributed RL training?

AgentScope’s tuner requires Trinity‑RFT and a Ray cluster. You must install trinity-rft via pip and initialize Ray with ray start --head (or connect to an existing cluster). The tune function then delegates distributed training, checkpointing, and logging to these underlying services.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}