# How to Evaluate AI Agent Performance Using τ-Bench, GAIA, or OSWorld

> Learn to evaluate AI agent performance with τ-Bench, GAIA, or OSWorld. Configure frameworks and run evaluations to measure pass@k and exact-match metrics for AI agent capabilities.

- Repository: [Bojie Li/ai-agent-book](https://github.com/bojieli/ai-agent-book)
- Tags: how-to-guide
- Published: 2026-07-31

---

**You can evaluate AI agent performance by configuring the τ-Bench framework's `RunConfig` to select GAIA for knowledge-intensive reasoning tasks or OSWorld for desktop automation scenarios, then executing the evaluation loop via [`run.py`](https://github.com/bojieli/ai-agent-book/blob/main/run.py) or OSWorld-specific scripts that aggregate per-step results into pass@k and exact-match metrics.**

The `bojieli/ai-agent-book` repository provides a production-ready evaluation pipeline for benchmarking large language model agents against industry standards. This guide demonstrates how to evaluate AI agent performance using τ-Bench, GAIA, and OSWorld with executable code examples derived directly from the source.

## Understanding the τ-Bench Evaluation Framework

The **τ-Bench** package lives under `chapter2/prompt-engineering/tau_bench` and provides a model-agnostic architecture for agent evaluation. The framework separates concerns into three distinct layers that communicate through well-defined Python interfaces.

### Core Components: Environments, Agents, and Configuration

The evaluation pipeline revolves around three primary components defined in the source:

- **Environments** (`tau_bench.envs`): Supplies sandboxed scenarios via `get_env` (e.g., simulated users or web-search tools) that the agent interacts with. Defined in [`envs/__init__.py`](https://github.com/bojieli/ai-agent-book/blob/main/envs/__init__.py).
- **Agents** (`tau_bench.agents`): Implements decision-making logic such as few-shot, ReAct, or tool-calling patterns. The base class resides in [`agents/base.py`](https://github.com/bojieli/ai-agent-book/blob/main/agents/base.py).
- **Run Configuration** (`tau_bench.types`): The `RunConfig` dataclass specifies which environment, agent, and evaluation metrics to use for a single run.

In [`chapter2/prompt-engineering/tau_bench/run.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter2/prompt-engineering/tau_bench/run.py), the `main()` function parses a `RunConfig`, instantiates the chosen environment, builds the agent class (e.g., `ChatReActAgent`), and executes an interaction loop that collects `EnvRunResult` objects. After the loop finishes, the framework aggregates per-step scores into the final benchmark score.

### Model-Agnostic Architecture

τ-Bench is deliberately **model-agnostic**: any LLM implementing the required `generate` interface can be plugged in via the `model_utils` sub-package. Adapters for OpenAI, Mistral, and Claude reside in [`model_utils/model/openai.py`](https://github.com/bojieli/ai-agent-book/blob/main/model_utils/model/openai.py), [`model_utils/model/mistral.py`](https://github.com/bojieli/ai-agent-book/blob/main/model_utils/model/mistral.py), and similar files.

## Evaluating Agents on the GAIA Benchmark

**GAIA** is a general-purpose AI benchmark testing reasoning, multimodal perception, web-search, and tool-use capabilities. The repository references GAIA from Chapter 6 and fetches data from HuggingFace:

```bash
git clone https://huggingface.co/datasets/gaia-benchmark/GAIA chapter6/GAIA

```

### GAIA Integration via User Simulation

The τ-Bench GAIA integration lives in `chapter2/prompt-engineering/tau_bench`. The environment `tau_bench.envs.user` provides a **user simulation** via `LLMUserSimulationEnv` that reads a GAIA problem, forwards the prompt to the agent, and expects a JSON answer containing `final_answer` and `reason`.

**Implementation highlights**:
- **Problem loading**: [`tau_bench/envs/user.py`](https://github.com/bojieli/ai-agent-book/blob/main/tau_bench/envs/user.py) reads the GAIA JSONL file and feeds individual questions to the agent.
- **Agent interaction**: The agent processes the query and returns a structured response that the environment validates against ground truth.

### Scoring and Validation Logic

Each trajectory is compared against the ground-truth answer using the `evaluate` method inside [`datapoint.py`](https://github.com/bojieli/ai-agent-book/blob/main/datapoint.py). This returns a binary correctness flag and a confidence score, enabling calculation of **exact-match** accuracy across the dataset.

## Testing Real-World Desktop Automation with OSWorld

**OSWorld** evaluates agents on full-OS tasks including file management, desktop-application automation, and system configuration. The repository contains a thin wrapper under `chapter8/gaia-experience/AWorld/examples/osworld` demonstrating how to run agents against the OSWorld-Verified tasks.

### OSWorld-Verified Task Suite

The OSWorld-Verified suite ships with **134 independent evaluation functions** that inspect the OS state after each step (filesystem, process list, network sockets, etc.). The agent receives observations via the OSWorld API, decides on an action (e.g., `click`, `type`, `run_command`), and the environment validates the result automatically.

### The AWorld Agent Interface

Key implementation files for OSWorld evaluation include:

- [`run_multienv_aworldAgent.py`](https://github.com/bojieli/ai-agent-book/blob/main/run_multienv_aworldAgent.py): Launches the AWorld agent, selects an OSWorld task, and executes it in a VM.
- [`grounding.py`](https://github.com/bojieli/ai-agent-book/blob/main/grounding.py): Implements `OSWorldACI`, a concrete **Agent-Core-Interface** that translates high-level actions into OSWorld API calls.
- [`README.md`](https://github.com/bojieli/ai-agent-book/blob/main/README.md) (OSWorld example): Documents the required OSWorld setup, including VM launch commands and checkpoint handling.

## Step-by-Step Implementation Examples

### Running τ-Bench on GAIA

Create a Python script that configures the `RunConfig` for GAIA evaluation:

```python

# example_run_gaia.py

from pathlib import Path
import sys

# Add repository root to PYTHONPATH

repo_root = Path(__file__).parents[1]
sys.path.append(str(repo_root))

from tau_bench.run import main as tau_main
from tau_bench.types import RunConfig, EnvName, AgentName, ModelName

if __name__ == "__main__":
    cfg = RunConfig(
        env=EnvName.USER,                      # GAIA-style user simulation

        agent=AgentName.CHAT_REACT,           # ReAct-style LLM agent

        model=ModelName.OPENAI_GPT4,          # Any model obeying the API

        num_episodes=10,                      # Sample 10 GAIA problems

        max_steps=50,
        seed=42,
    )
    results = tau_main(cfg)
    print("Overall GAIA score:", results.summary())

```

**Key details**:
- `EnvName.USER` points to [`tau_bench/envs/user.py`](https://github.com/bojieli/ai-agent-book/blob/main/tau_bench/envs/user.py) which loads the GAIA problem set.
- `AgentName.CHAT_REACT` loads [`agents/chat_react_agent.py`](https://github.com/bojieli/ai-agent-book/blob/main/agents/chat_react_agent.py).
- The framework logs each trajectory in `tau_bench/logs/`.

### Executing OSWorld Evaluations

For desktop automation benchmarking, use the AWorld wrapper:

```python

# run_osworld_example.py

import os
from pathlib import Path
import sys

repo_root = Path(__file__).parents[1]
sys.path.append(str(repo_root))

from chapter8.gaia_experience.AWorld.examples.osworld.run_multienv_aworldAgent import (
    main as run_osworld,
)

if __name__ == "__main__":
    # OSWorld expects a pre-configured VM; set the path to the VMX file

    vmx_path = "/path/to/your/osworld_vm.vmx"
    # Choose a task from the OSWorld-Verified set (e.g., "file_manager_copy")

    task_id = "osworld-verified:file_manager_copy"

    results = run_osworld(
        vmx_path=vmx_path,
        task_id=task_id,
        model_name="openai/gpt-4o",       # Model used by the AWorld agent

        temperature=1.0,
        max_steps=50,
    )
    print("OSWorld task result:", results)

```

**Implementation notes**:
- [`run_multienv_aworldAgent.py`](https://github.com/bojieli/ai-agent-book/blob/main/run_multienv_aworldAgent.py) creates an instance of `OSWorldACI` (defined in [`grounding.py`](https://github.com/bojieli/ai-agent-book/blob/main/grounding.py)).
- The agent issues high-level commands translated into OSWorld API calls.
- After the episode finishes, OSWorld's built-in evaluator returns a **pass@1** score (approximately 58% for the reference implementation).

### Command-Line Interface Options

You can also invoke evaluations directly from the terminal:

```bash

# Evaluate GAIA

python -m tau_bench.run --env user --agent chat_react --model openai/gpt-4o --episodes 20

# Evaluate OSWorld

python -m chapter8.gaia_experience.AWorld.examples.osworld.run_multienv_aworldAgent \
    --vmx /path/to/vm.vmx --task osworld-verified:file_manager_copy \
    --model openai/gpt-4o --temp 1.0

```

## Key Source Files in the Repository

Understanding these files is essential for customizing evaluations:

- **[`chapter2/prompt-engineering/tau_bench/run.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter2/prompt-engineering/tau_bench/run.py)**: Main driver parsing `RunConfig`, building environment and agent instances, and running episodes.
- **[`chapter2/prompt-engineering/tau_bench/envs/user.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter2/prompt-engineering/tau_bench/envs/user.py)**: GAIA-style user simulation environment loading GAIA JSONL problems.
- **[`chapter2/prompt-engineering/tau_bench/agents/chat_react_agent.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter2/prompt-engineering/tau_bench/agents/chat_react_agent.py)**: ReAct-style LLM agent supporting tool invocation and step-by-step reasoning.
- **`chapter2/prompt-engineering/tau_bench/model_utils/`**: Adapter layer for OpenAI, Mistral, Claude, and other LLM providers.
- **[`chapter6/README.md`](https://github.com/bojieli/ai-agent-book/blob/main/chapter6/README.md)**: Overview of all evaluation benchmarks including GAIA, OSWorld, and τ-Bench.
- **[`chapter8/gaia-experience/AWorld/examples/osworld/run_multienv_aworldAgent.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter8/gaia-experience/AWorld/examples/osworld/run_multienv_aworldAgent.py)**: Wrapper launching the AWorld agent against OSWorld tasks.
- **[`chapter8/gaia-experience/AWorld/examples/osworld/grounding.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter8/gaia-experience/AWorld/examples/osworld/grounding.py)**: Implements `OSWorldACI` for translating agent actions to OSWorld API calls.
- **[`chapter8/gaia-experience/AWorld/examples/osworld/README.md`](https://github.com/bojieli/ai-agent-book/blob/main/chapter8/gaia-experience/AWorld/examples/osworld/README.md)**: Instructions for OSWorld VM setup and benchmark reproduction.

## Summary

- **τ-Bench** provides a unified framework for agent evaluation through the `RunConfig` abstraction, separating environments from agent logic.
- **GAIA** integration uses the `user` environment in τ-Bench to test knowledge-intensive tasks, scoring based on exact-match validation against ground truth.
- **OSWorld** requires the AWorld wrapper and `OSWorldACI` interface to evaluate desktop automation capabilities across 134 verified OS tasks.
- All three benchmarks support **model-agnostic** evaluation via the `model_utils` adapter pattern, allowing you to swap LLM providers without changing evaluation logic.
- Evaluation metrics include **pass@k**, **exact-match**, and per-step correctness scores aggregated automatically by the framework.

## Frequently Asked Questions

### How do I switch between GAIA and OSWorld in the same evaluation pipeline?

You can swap benchmarks by changing the `env` parameter in your `RunConfig`. For GAIA, use `EnvName.USER`; for OSWorld, configure the AWorld wrapper with the specific task ID. Both use the same τ-Bench infrastructure, ensuring consistent metrics and logging formats.

### What hardware requirements are needed for OSWorld evaluation?

OSWorld requires a virtual machine (VM) running a full operating system. You must provide a VMX file path (`vmx_path`) pointing to a pre-configured VM image. The [`run_multienv_aworldAgent.py`](https://github.com/bojieli/ai-agent-book/blob/main/run_multienv_aworldAgent.py) script interacts with this VM via the OSWorld API, executing real desktop actions and verifying state changes through filesystem and process inspection.

### Can I use custom models with τ-Bench instead of OpenAI GPT-4?

Yes. τ-Bench is model-agnostic. Create a new adapter in `tau_bench/model_utils/model/` implementing the required `generate` interface, then reference your model in `RunConfig` (e.g., `model=ModelName.YOUR_MODEL`). The framework supports Mistral, Claude, and local models through this adapter pattern.

### How is the final score calculated in GAIA evaluations?

The `evaluate` method in [`datapoint.py`](https://github.com/bojieli/ai-agent-book/blob/main/datapoint.py) compares the agent's `final_answer` against the ground-truth answer from the GAIA dataset. It returns a binary correctness flag (1 for correct, 0 for incorrect) and an optional confidence score. The framework aggregates these flags across all episodes to compute the overall **exact-match** accuracy percentage.