# AgentScope Evaluation Frameworks: Built-in Benchmarking and Custom Metrics

> Explore AgentScope evaluation frameworks for agent benchmarking. Discover parallel benchmarking, sequential debugging, ACEBench, and custom metrics for robust AI agent evaluation.

- Repository: [AgentScope-AI/agentscope](https://github.com/agentscope-ai/agentscope)
- Tags: deep-dive
- Published: 2026-03-09

---

**AgentScope provides a modular evaluation framework that supports parallel benchmarking via RayEvaluator, sequential debugging via GeneralEvaluator, and includes the built-in ACEBench dataset alongside customizable metrics and persistent storage.**

AgentScope (agentscope-ai/agentscope) ships with a comprehensive evaluation framework designed for rigorous agent benchmarking. The framework follows a hierarchical architecture separating benchmarks, tasks, metrics, and evaluators in `src/agentscope/evaluate/`. This modular design allows researchers to evaluate agents on built-in datasets or implement custom evaluation scenarios.

## Core Components of the AgentScope Evaluation Framework

The framework organizes evaluation logic into five distinct layers, each implemented as an abstract base class.

### Benchmarks and Tasks

The **Benchmark** acts as a container for evaluation tasks. In [`src/agentscope/evaluate/_benchmark_base.py`](https://github.com/agentscope-ai/agentscope/blob/main/src/agentscope/evaluate/_benchmark_base.py), the `BenchmarkBase` class defines the interface for datasets, while [`src/agentscope/evaluate/_task.py`](https://github.com/agentscope-ai/agentscope/blob/main/src/agentscope/evaluate/_task.py) implements the **Task** class—a single evaluation unit containing the input prompt, ground truth, and associated metrics.

### Metrics and Solutions

Metrics quantify agent performance. The `MetricBase` class in [`src/agentscope/evaluate/_metric_base.py`](https://github.com/agentscope-ai/agentscope/blob/main/src/agentscope/evaluate/_metric_base.py) provides the interface for scoring functions, while `SolutionOutput` in [`src/agentscope/evaluate/_solution.py`](https://github.com/agentscope-ai/agentscope/blob/main/src/agentscope/evaluate/_solution.py) wraps the agent's generated response. Each metric callable receives a `SolutionOutput` object and returns a `MetricResult` containing the score and diagnostic metadata.

### Evaluators: Parallel vs Sequential Execution

AgentScope provides two evaluator implementations to accommodate different computational needs.

**RayEvaluator** (defined in [`src/agentscope/evaluate/_evaluator/_ray_evaluator.py`](https://github.com/agentscope-ai/agentscope/blob/main/src/agentscope/evaluate/_evaluator/_ray_evaluator.py)) enables distributed evaluation using Ray actors. Set the `max_workers` parameter to control parallelization across CPU or GPU resources.

**GeneralEvaluator** (defined in [`src/agentscope/evaluate/_evaluator/_general_evaluator.py`](https://github.com/agentscope-ai/agentscope/blob/main/src/agentscope/evaluate/_evaluator/_general_evaluator.py)) executes tasks sequentially. This evaluator is ideal for debugging agent behavior or running lightweight benchmarks where parallel overhead would be inefficient.

### Persistent Storage with Resume Capability

The **FileEvaluatorStorage** class in [`src/agentscope/evaluate/_evaluator_storage/_file_evaluator_storage.py`](https://github.com/agentscope-ai/agentscope/blob/main/src/agentscope/evaluate/_evaluator_storage/_file_evaluator_storage.py) provides fault-tolerant persistence. Evaluation runs automatically save intermediate states to disk, allowing interrupted benchmarks to resume without recomputing completed tasks.

## Built-in Benchmarks: ACEBench and GAIA

AgentScope includes production-ready benchmark implementations.

**ACEBench** ([`src/agentscope/evaluate/_ace_benchmark/_ace_benchmark.py`](https://github.com/agentscope-ai/agentscope/blob/main/src/agentscope/evaluate/_ace_benchmark/_ace_benchmark.py)) is a comprehensive evaluation suite covering reasoning, planning, and tool-use scenarios. Instantiate it directly via `ACEBenchmark()` to evaluate agents on complex multi-step tasks.

**GAIA** is currently listed as a coming benchmark in the documentation, with placeholder infrastructure prepared for future integration.

## Implementing Parallel Evaluation with RayEvaluator

For large-scale benchmarking, the `RayEvaluator` distributes tasks across multiple workers. The following example demonstrates evaluating a ReAct agent on ACEBench:

```python
import os
import asyncio
from agentscope.evaluate import (
    RayEvaluator,
    FileEvaluatorStorage,
    ACEBenchmark,
)
from agentscope.agent import ReActAgent
from agentscope.model import DashScopeChatModel
from agentscope.formatter import DashScopeChatFormatter
from agentscope.tool import Toolkit, execute_python_code, execute_shell_command

async def run():
    # Configure agent with tool capabilities

    toolkit = Toolkit()
    toolkit.register_tool_function(execute_python_code)
    toolkit.register_tool_function(execute_shell_command)

    agent = ReActAgent(
        name="Frank",
        sys_prompt="You are a helpful assistant.",
        model=DashScopeChatModel(
            model_name="qwen-max",
            api_key=os.getenv("DASHSCOPE_API_KEY"),
            stream=False,
        ),
        formatter=DashScopeChatFormatter(),
        toolkit=toolkit,
    )

    # Define solution function mapping tasks to agent outputs

    async def solve(task):
        msg = await agent.run(task.input)
        return msg.content

    # Initialize distributed evaluator

    evaluator = RayEvaluator(
        benchmark=ACEBenchmark(),
        solution_func=solve,
        storage=FileEvaluatorStorage(),
        max_workers=4,
    )

    await evaluator.run()

if __name__ == "__main__":
    asyncio.run(run())

```

## Debugging with GeneralEvaluator

For development and troubleshooting, use `GeneralEvaluator` to execute tasks sequentially with detailed logging:

```python
from agentscope.evaluate import GeneralEvaluator, FileEvaluatorStorage, ACEBenchmark

evaluator = GeneralEvaluator(
    benchmark=ACEBenchmark(),
    solution_func=solve,
    storage=FileEvaluatorStorage(),
)

await evaluator.run()

```

## Creating Custom Benchmarks and Metrics

You can extend `BenchmarkBase` and `MetricBase` to evaluate domain-specific capabilities. The following example implements a numerical exact-match benchmark:

```python
from agentscope.evaluate import (
    BenchmarkBase,
    Task,
    MetricBase,
    MetricResult,
    MetricType,
    SolutionOutput,
)

class CheckEqual(MetricBase):
    def __init__(self, gt: float):
        super().__init__(
            name="exact-match",
            metric_type=MetricType.NUMERICAL,
            description="Check if output equals ground truth",
            categories=[],
        )
        self.gt = gt

    async def __call__(self, sol: SolutionOutput) -> MetricResult:
        is_correct = sol.output == self.gt
        return MetricResult(
            name=self.name,
            result=1.0 if is_correct else 0.0,
            message="correct" if is_correct else "incorrect",
        )

class ToyBenchmark(BenchmarkBase):
    def __init__(self):
        super().__init__(name="Toy", description="Demo benchmark")
        self.tasks = [
            Task(
                id="t1",
                input="What is 2+2?",
                ground_truth=4,
                metrics=[CheckEqual(4)],
                metadata={},
            ),
            Task(
                id="t2",
                input="What is 3*3?",
                ground_truth=9,
                metrics=[CheckEqual(9)],
                metadata={},
            ),
        ]

    def __iter__(self):
        yield from self.tasks

```

Run custom benchmarks using either `RayEvaluator` or `GeneralEvaluator` by passing the benchmark instance to the `benchmark` parameter.

## Summary

- AgentScope's evaluation framework separates concerns into Benchmarks, Tasks, Metrics, Solutions, and Evaluators.
- **RayEvaluator** ([`_ray_evaluator.py`](https://github.com/agentscope-ai/agentscope/blob/main/_ray_evaluator.py)) enables distributed evaluation across multiple workers, while **GeneralEvaluator** ([`_general_evaluator.py`](https://github.com/agentscope-ai/agentscope/blob/main/_general_evaluator.py)) provides sequential execution for debugging.
- **FileEvaluatorStorage** ([`_file_evaluator_storage.py`](https://github.com/agentscope-ai/agentscope/blob/main/_file_evaluator_storage.py)) persists results automatically, supporting resume functionality for interrupted runs.
- The framework includes **ACEBench** ([`_ace_benchmark.py`](https://github.com/agentscope-ai/agentscope/blob/main/_ace_benchmark.py)) for testing reasoning and tool-use capabilities.
- Developers can subclass `BenchmarkBase` and `MetricBase` to implement domain-specific evaluation logic.

## Frequently Asked Questions

### What is the difference between RayEvaluator and GeneralEvaluator in AgentScope?

**RayEvaluator** distributes tasks across Ray actors for parallel execution, configured via the `max_workers` parameter, making it suitable for large-scale benchmarks. **GeneralEvaluator** runs tasks sequentially in a single process, providing immediate feedback and easier debugging for agent development.

### How does AgentScope handle interrupted evaluation runs?

The framework uses **FileEvaluatorStorage** to persist intermediate results to disk after each task completion. If a run is interrupted, the evaluator detects existing progress in the storage backend and resumes from the last completed task without reprocessing previous results.

### Can I use custom metrics with the built-in ACEBench benchmark?

Yes. While ACEBench includes predefined metrics, you can subclass `MetricBase` in [`src/agentscope/evaluate/_metric_base.py`](https://github.com/agentscope-ai/agentscope/blob/main/src/agentscope/evaluate/_metric_base.py) to implement custom scoring logic. Pass your metric instances when constructing Task objects, or subclass `ACEBenchmark` to modify the default metric configuration.

### Does AgentScope support the GAIA benchmark?

GAIA is currently listed as a coming benchmark in the AgentScope documentation. The framework infrastructure supports its addition, but the implementation is not yet available in the current release. Users can currently use ACEBench or implement custom benchmarks following the `BenchmarkBase` interface.