# How to Evaluate AI Agent Performance Using GAIA, OSWorld, and SWE-bench

> Evaluate AI agent performance using benchmarks like GAIA, OSWorld, and SWE-bench. Learn how to use companion tools in the ai-agent-book repository for comprehensive testing.

- Repository: [Bojie Li/ai-agent-book](https://github.com/bojieli/ai-agent-book)
- Tags: how-to-guide
- Published: 2026-07-30

---

**You can evaluate AI Agent performance by running specialized benchmarks—SWE-bench for code patching, GAIA for tool-augmented reasoning, and OSWorld for operating-system tasks—using the companion tools and wrapper scripts provided in Chapter 6 of the ai-agent-book repository.**

The **ai-agent-book** repository (bojieli/ai-agent-book) provides a complete teaching platform that combines theoretical foundations with hands-on evaluation code. Chapter 6 is specifically dedicated to **Agent Evaluation** and integrates three industry-standard benchmarks that measure distinct capabilities: generating software patches, executing multi-step reasoning with tools, and manipulating full desktop environments. According to the repository's design philosophy, the evaluation environment serves as the *stage* and the dataset acts as the *script*—the quality of your evaluation setup directly determines the signal quality you obtain from agent runs【/cache/repos/github.com/bojieli/ai-agent-book/main/book-en/chapter6.md†L155-L167】.

## Understanding the Three Core AI Agent Benchmarks

The repository structures evaluation around three external benchmarks, each targeting a specific dimension of agent capability.

### SWE-bench: Real-World Code Generation

**SWE-bench** measures an agent's ability to generate correct patches for real GitHub issues. It includes multiple variants—Lite, Verified, and Multimodal—to accommodate different resource constraints and testing scenarios. In the repository, this benchmark lives in `chapter6/SWE-bench/` and serves as the standard for assessing software engineering agents【/cache/repos/github.com/bojieli/ai-agent-book/main/chapter6/README.en.md】.

### GAIA: Tool-Augmented Reasoning

**GAIA** evaluates tool-augmented, search-enabled autonomous reasoning across over 450 non-trivial questions distributed across three difficulty levels. This benchmark tests whether agents can effectively use external tools and search capabilities to solve complex problems. The wrapper code resides in `chapter6/GAIA/`【/cache/repos/github.com/bojieli/ai-agent-book/main/chapter6/README.en.md】.

### OSWorld: Full Operating System Tasks

**OSWorld** assesses agent performance on full-operating-system tasks including file management, application interaction, and system configuration within a realistic desktop environment. Located in `chapter6/OSWorld/`, this benchmark is essential for evaluating agents designed for general computer use【/cache/repos/github.com/bojieli/ai-agent-book/main/chapter6/README.en.md】.

## Setting Up the Evaluation Environment

The benchmarks are external repositories that must be cloned into the Chapter 6 workspace. The root README provides a "一键克隆脚本" (one-click clone script) section with the specific commands【/cache/repos/github.com/bojieli/ai-agent-book/main/README.md†L76-L88】.

Clone the three evaluation benchmarks from the repository root:

```bash
git clone https://github.com/SWE-bench/SWE-bench.git   chapter6/SWE-bench
git clone https://github.com/xlang-ai/GAIA.git       chapter6/GAIA
git clone https://github.com/xlang-ai/OSWorld.git    chapter6/OSWorld

```

Each cloned directory contains the original benchmark suite plus repository-specific wrapper scripts that standardize the evaluation interface.

## Running Evaluations with Companion Tools

Beyond the raw benchmarks, the repository provides three specialized tools in `chapter6/` that facilitate systematic evaluation: an ELO leaderboard for comparative analysis, a latency benchmark for API performance, and a cost analyzer for economic assessment.

### ELO Leaderboard Comparisons

The `elo-leaderboard` tool allows you to compare multiple agents across benchmarks using the ELO rating system. This is implemented in [`chapter6/elo-leaderboard/main.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter6/elo-leaderboard/main.py) and supports head-to-head agent comparison【/cache/repos/github.com/bojieli/ai-agent-book/main/chapter6/elo-leaderboard/README.md†L1-L12】.

Install requirements and run a comparison between two agents on SWE-bench:

```bash
pip install -r chapter6/elo-leaderboard/requirements.txt

python chapter6/elo-leaderboard/main.py \
    --benchmark SWE-bench \
    --agents agent_a.py agent_b.py \
    --output results.json

```

The script loads benchmark tasks, executes each agent to produce patches or answers, and calculates relative performance scores using the ELO algorithm.

### API Latency Benchmarking

The `model-benchmark` utility measures LLM API performance characteristics including Time To First Token (TTFT), percentile latencies (p50/p95), and success rates. This is critical when evaluating agent performance, as response latency directly impacts multi-step task completion times【/cache/repos/github.com/bojieli/ai-agent-book/main/chapter6/model-benchmark/README.md†L1-L9】.

Benchmark a specific model's latency profile:

```bash
pip install -r chapter6/model-benchmark/requirements.txt

python chapter6/model-benchmark/benchmark.py \
    --model openai/gpt-4o \
    --prompt "Explain the difference between SWE-bench and GAIA." \
    --concurrency 8 \
    --output latency.csv

```

### Cost Analysis for Multi-turn Agents

The `agent-cost-analysis` framework traces token-level usage across multi-turn agent workflows to calculate precise economic costs. This tool runs in [`chapter6/agent-cost-analysis/demo.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter6/agent-cost-analysis/demo.py) and breaks down expenses per task step【/cache/repos/github.com/bojieli/ai-agent-book/main/chapter6/agent-cost-analysis/README.md†L1-L7】.

Analyze costs for a customer-service agent task:

```bash
pip install -r chapter6/agent-cost-analysis/requirements.txt

python chapter6/agent-cost-analysis/demo.py \
    --task refund_customer \
    --model openai/gpt-4o-mini

```

The demo executes a realistic multi-turn dialogue, tracks token consumption at each step, and outputs a detailed cost table enabling data-driven decisions about model selection and prompt optimization.

## Design Philosophy and Interpretation

The theoretical foundation for these evaluations is documented in [`book-en/chapter6.md`](https://github.com/bojieli/ai-agent-book/blob/main/book-en/chapter6.md), which explains benchmark design principles, data contamination challenges, and evaluation-driven agent selection workflows. The repository treats evaluation not merely as a testing phase but as an integral component of agent architecture design—what the documentation describes as the interplay between the evaluation *stage* and the dataset *script*【/cache/repos/github.com/bojieli/ai-agent-book/main/book-en/chapter6.md†L155-L167】.

When interpreting results across **SWE-bench**, **GAIA**, and **OSWorld**, consider that each benchmark captures a distinct capability vector: SWE-bench validates code correctness against ground-truth patches, GAIA measures information synthesis and tool use accuracy, and OSWorld evaluates UI interaction and system-level task completion rates.

## Summary

- **Three benchmark types**: Use SWE-bench for code generation, GAIA for reasoning with tools, and OSWorld for desktop automation tasks.
- **Repository location**: All evaluation code resides in `chapter6/`, with benchmark wrappers in `chapter6/SWE-bench/`, `chapter6/GAIA/`, and `chapter6/OSWorld/`.
- **Clone commands**: External repositories must be cloned using the specific URLs provided in the root README's one-click script section.
- **Comparison tools**: The `elo-leaderboard` script enables head-to-head agent comparison using ELO ratings across any supported benchmark.
- **Performance metrics**: Use `model-benchmark` to measure API latency and `agent-cost-analysis` to calculate token-level economic costs for multi-turn workflows.
- **Theoretical grounding**: Implementation details and interpretation guidelines are documented in [`book-en/chapter6.md`](https://github.com/bojieli/ai-agent-book/blob/main/book-en/chapter6.md).

## Frequently Asked Questions

### How do I choose between SWE-bench, GAIA, and OSWorld for my agent?

**Select SWE-bench** if your agent generates or modifies code to resolve software issues, **GAIA** if your agent performs research and reasoning with search tools and APIs, and **OSWorld** if your agent interacts with graphical interfaces and operating system primitives. According to the ai-agent-book source code, these benchmarks are complementary—production agents often require evaluation across all three domains to assess general capability.

### What is the minimum setup required to run these benchmarks?

You must clone the external repositories into the `chapter6/` directory using the commands provided in the root README, then install dependencies via the [`requirements.txt`](https://github.com/bojieli/ai-agent-book/blob/main/requirements.txt) files found in each tool subdirectory (e.g., [`chapter6/elo-leaderboard/requirements.txt`](https://github.com/bojieli/ai-agent-book/blob/main/chapter6/elo-leaderboard/requirements.txt)). The benchmarks themselves require significant computational resources: SWE-bench needs Docker containers for safe code execution, while OSWorld requires a virtualized desktop environment.

### How does the ELO leaderboard handle different benchmark types?

The [`elo-leaderboard/main.py`](https://github.com/bojieli/ai-agent-book/blob/main/elo-leaderboard/main.py) script abstracts benchmark-specific scoring behind a unified interface. It loads tasks from the specified benchmark (SWE-bench, GAIA, or OSWorld), executes each agent against those tasks, and converts binary or graded success metrics into ELO rating adjustments. This allows direct comparison between agents even when their absolute scores come from different evaluation paradigms.

### Where can I find the theoretical background on benchmark design?

The design principles, contamination risks, and methodological considerations are detailed in [`book-en/chapter6.md`](https://github.com/bojieli/ai-agent-book/blob/main/book-en/chapter6.md). This file explains why the repository structures evaluation as a *stage* (the execution environment) and the dataset as a *script* (the task specification), emphasizing that poor evaluation infrastructure introduces noise that masks true agent capabilities.