AgentScope Evaluation Frameworks: Built-in Benchmarking and Custom Metrics

AgentScope provides a modular evaluation framework that supports parallel benchmarking via RayEvaluator, sequential debugging via GeneralEvaluator, and includes the built-in ACEBench dataset alongside customizable metrics and persistent storage.

AgentScope (agentscope-ai/agentscope) ships with a comprehensive evaluation framework designed for rigorous agent benchmarking. The framework follows a hierarchical architecture separating benchmarks, tasks, metrics, and evaluators in src/agentscope/evaluate/. This modular design allows researchers to evaluate agents on built-in datasets or implement custom evaluation scenarios.

Core Components of the AgentScope Evaluation Framework

The framework organizes evaluation logic into five distinct layers, each implemented as an abstract base class.

Benchmarks and Tasks

The Benchmark acts as a container for evaluation tasks. In src/agentscope/evaluate/_benchmark_base.py, the BenchmarkBase class defines the interface for datasets, while src/agentscope/evaluate/_task.py implements the Task class—a single evaluation unit containing the input prompt, ground truth, and associated metrics.

Metrics and Solutions

Metrics quantify agent performance. The MetricBase class in src/agentscope/evaluate/_metric_base.py provides the interface for scoring functions, while SolutionOutput in src/agentscope/evaluate/_solution.py wraps the agent's generated response. Each metric callable receives a SolutionOutput object and returns a MetricResult containing the score and diagnostic metadata.

Evaluators: Parallel vs Sequential Execution

AgentScope provides two evaluator implementations to accommodate different computational needs.

RayEvaluator (defined in src/agentscope/evaluate/_evaluator/_ray_evaluator.py) enables distributed evaluation using Ray actors. Set the max_workers parameter to control parallelization across CPU or GPU resources.

GeneralEvaluator (defined in src/agentscope/evaluate/_evaluator/_general_evaluator.py) executes tasks sequentially. This evaluator is ideal for debugging agent behavior or running lightweight benchmarks where parallel overhead would be inefficient.

Persistent Storage with Resume Capability

The FileEvaluatorStorage class in src/agentscope/evaluate/_evaluator_storage/_file_evaluator_storage.py provides fault-tolerant persistence. Evaluation runs automatically save intermediate states to disk, allowing interrupted benchmarks to resume without recomputing completed tasks.

Built-in Benchmarks: ACEBench and GAIA

AgentScope includes production-ready benchmark implementations.

ACEBench (src/agentscope/evaluate/_ace_benchmark/_ace_benchmark.py) is a comprehensive evaluation suite covering reasoning, planning, and tool-use scenarios. Instantiate it directly via ACEBenchmark() to evaluate agents on complex multi-step tasks.

GAIA is currently listed as a coming benchmark in the documentation, with placeholder infrastructure prepared for future integration.

Implementing Parallel Evaluation with RayEvaluator

For large-scale benchmarking, the RayEvaluator distributes tasks across multiple workers. The following example demonstrates evaluating a ReAct agent on ACEBench:

import os
import asyncio
from agentscope.evaluate import (
    RayEvaluator,
    FileEvaluatorStorage,
    ACEBenchmark,
)
from agentscope.agent import ReActAgent
from agentscope.model import DashScopeChatModel
from agentscope.formatter import DashScopeChatFormatter
from agentscope.tool import Toolkit, execute_python_code, execute_shell_command

async def run():
    # Configure agent with tool capabilities

    toolkit = Toolkit()
    toolkit.register_tool_function(execute_python_code)
    toolkit.register_tool_function(execute_shell_command)

    agent = ReActAgent(
        name="Frank",
        sys_prompt="You are a helpful assistant.",
        model=DashScopeChatModel(
            model_name="qwen-max",
            api_key=os.getenv("DASHSCOPE_API_KEY"),
            stream=False,
        ),
        formatter=DashScopeChatFormatter(),
        toolkit=toolkit,
    )

    # Define solution function mapping tasks to agent outputs

    async def solve(task):
        msg = await agent.run(task.input)
        return msg.content

    # Initialize distributed evaluator

    evaluator = RayEvaluator(
        benchmark=ACEBenchmark(),
        solution_func=solve,
        storage=FileEvaluatorStorage(),
        max_workers=4,
    )

    await evaluator.run()

if __name__ == "__main__":
    asyncio.run(run())

Debugging with GeneralEvaluator

For development and troubleshooting, use GeneralEvaluator to execute tasks sequentially with detailed logging:

from agentscope.evaluate import GeneralEvaluator, FileEvaluatorStorage, ACEBenchmark

evaluator = GeneralEvaluator(
    benchmark=ACEBenchmark(),
    solution_func=solve,
    storage=FileEvaluatorStorage(),
)

await evaluator.run()

Creating Custom Benchmarks and Metrics

You can extend BenchmarkBase and MetricBase to evaluate domain-specific capabilities. The following example implements a numerical exact-match benchmark:

from agentscope.evaluate import (
    BenchmarkBase,
    Task,
    MetricBase,
    MetricResult,
    MetricType,
    SolutionOutput,
)

class CheckEqual(MetricBase):
    def __init__(self, gt: float):
        super().__init__(
            name="exact-match",
            metric_type=MetricType.NUMERICAL,
            description="Check if output equals ground truth",
            categories=[],
        )
        self.gt = gt

    async def __call__(self, sol: SolutionOutput) -> MetricResult:
        is_correct = sol.output == self.gt
        return MetricResult(
            name=self.name,
            result=1.0 if is_correct else 0.0,
            message="correct" if is_correct else "incorrect",
        )

class ToyBenchmark(BenchmarkBase):
    def __init__(self):
        super().__init__(name="Toy", description="Demo benchmark")
        self.tasks = [
            Task(
                id="t1",
                input="What is 2+2?",
                ground_truth=4,
                metrics=[CheckEqual(4)],
                metadata={},
            ),
            Task(
                id="t2",
                input="What is 3*3?",
                ground_truth=9,
                metrics=[CheckEqual(9)],
                metadata={},
            ),
        ]

    def __iter__(self):
        yield from self.tasks

Run custom benchmarks using either RayEvaluator or GeneralEvaluator by passing the benchmark instance to the benchmark parameter.

Summary

• AgentScope's evaluation framework separates concerns into Benchmarks, Tasks, Metrics, Solutions, and Evaluators.
• RayEvaluator (_ray_evaluator.py) enables distributed evaluation across multiple workers, while GeneralEvaluator (_general_evaluator.py) provides sequential execution for debugging.
• FileEvaluatorStorage (_file_evaluator_storage.py) persists results automatically, supporting resume functionality for interrupted runs.
• The framework includes ACEBench (_ace_benchmark.py) for testing reasoning and tool-use capabilities.
• Developers can subclass BenchmarkBase and MetricBase to implement domain-specific evaluation logic.

Frequently Asked Questions

What is the difference between RayEvaluator and GeneralEvaluator in AgentScope?

RayEvaluator distributes tasks across Ray actors for parallel execution, configured via the max_workers parameter, making it suitable for large-scale benchmarks. GeneralEvaluator runs tasks sequentially in a single process, providing immediate feedback and easier debugging for agent development.

How does AgentScope handle interrupted evaluation runs?

The framework uses FileEvaluatorStorage to persist intermediate results to disk after each task completion. If a run is interrupted, the evaluator detects existing progress in the storage backend and resumes from the last completed task without reprocessing previous results.

Can I use custom metrics with the built-in ACEBench benchmark?

Yes. While ACEBench includes predefined metrics, you can subclass MetricBase in src/agentscope/evaluate/_metric_base.py to implement custom scoring logic. Pass your metric instances when constructing Task objects, or subclass ACEBenchmark to modify the default metric configuration.

Does AgentScope support the GAIA benchmark?

GAIA is currently listed as a coming benchmark in the AgentScope documentation. The framework infrastructure supports its addition, but the implementation is not yet available in the current release. Users can currently use ACEBench or implement custom benchmarks following the BenchmarkBase interface.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}