How to Evaluate AI Agent Performance Using GAIA, OSWorld, and SWE-bench

You can evaluate AI Agent performance by running specialized benchmarks—SWE-bench for code patching, GAIA for tool-augmented reasoning, and OSWorld for operating-system tasks—using the companion tools and wrapper scripts provided in Chapter 6 of the ai-agent-book repository.

The ai-agent-book repository (bojieli/ai-agent-book) provides a complete teaching platform that combines theoretical foundations with hands-on evaluation code. Chapter 6 is specifically dedicated to Agent Evaluation and integrates three industry-standard benchmarks that measure distinct capabilities: generating software patches, executing multi-step reasoning with tools, and manipulating full desktop environments. According to the repository's design philosophy, the evaluation environment serves as the stage and the dataset acts as the script—the quality of your evaluation setup directly determines the signal quality you obtain from agent runs【/cache/repos/github.com/bojieli/ai-agent-book/main/book-en/chapter6.md†L155-L167】.

Understanding the Three Core AI Agent Benchmarks

The repository structures evaluation around three external benchmarks, each targeting a specific dimension of agent capability.

SWE-bench: Real-World Code Generation

SWE-bench measures an agent's ability to generate correct patches for real GitHub issues. It includes multiple variants—Lite, Verified, and Multimodal—to accommodate different resource constraints and testing scenarios. In the repository, this benchmark lives in chapter6/SWE-bench/ and serves as the standard for assessing software engineering agents【/cache/repos/github.com/bojieli/ai-agent-book/main/chapter6/README.en.md】.

GAIA: Tool-Augmented Reasoning

GAIA evaluates tool-augmented, search-enabled autonomous reasoning across over 450 non-trivial questions distributed across three difficulty levels. This benchmark tests whether agents can effectively use external tools and search capabilities to solve complex problems. The wrapper code resides in chapter6/GAIA/【/cache/repos/github.com/bojieli/ai-agent-book/main/chapter6/README.en.md】.

OSWorld: Full Operating System Tasks

OSWorld assesses agent performance on full-operating-system tasks including file management, application interaction, and system configuration within a realistic desktop environment. Located in chapter6/OSWorld/, this benchmark is essential for evaluating agents designed for general computer use【/cache/repos/github.com/bojieli/ai-agent-book/main/chapter6/README.en.md】.

Setting Up the Evaluation Environment

The benchmarks are external repositories that must be cloned into the Chapter 6 workspace. The root README provides a "一键克隆脚本" (one-click clone script) section with the specific commands【/cache/repos/github.com/bojieli/ai-agent-book/main/README.md†L76-L88】.

Clone the three evaluation benchmarks from the repository root:

git clone https://github.com/SWE-bench/SWE-bench.git   chapter6/SWE-bench
git clone https://github.com/xlang-ai/GAIA.git       chapter6/GAIA
git clone https://github.com/xlang-ai/OSWorld.git    chapter6/OSWorld

Each cloned directory contains the original benchmark suite plus repository-specific wrapper scripts that standardize the evaluation interface.

Running Evaluations with Companion Tools

Beyond the raw benchmarks, the repository provides three specialized tools in chapter6/ that facilitate systematic evaluation: an ELO leaderboard for comparative analysis, a latency benchmark for API performance, and a cost analyzer for economic assessment.

ELO Leaderboard Comparisons

The elo-leaderboard tool allows you to compare multiple agents across benchmarks using the ELO rating system. This is implemented in chapter6/elo-leaderboard/main.py and supports head-to-head agent comparison【/cache/repos/github.com/bojieli/ai-agent-book/main/chapter6/elo-leaderboard/README.md†L1-L12】.

Install requirements and run a comparison between two agents on SWE-bench:

pip install -r chapter6/elo-leaderboard/requirements.txt

python chapter6/elo-leaderboard/main.py \
    --benchmark SWE-bench \
    --agents agent_a.py agent_b.py \
    --output results.json

The script loads benchmark tasks, executes each agent to produce patches or answers, and calculates relative performance scores using the ELO algorithm.

API Latency Benchmarking

The model-benchmark utility measures LLM API performance characteristics including Time To First Token (TTFT), percentile latencies (p50/p95), and success rates. This is critical when evaluating agent performance, as response latency directly impacts multi-step task completion times【/cache/repos/github.com/bojieli/ai-agent-book/main/chapter6/model-benchmark/README.md†L1-L9】.

Benchmark a specific model's latency profile:

pip install -r chapter6/model-benchmark/requirements.txt

python chapter6/model-benchmark/benchmark.py \
    --model openai/gpt-4o \
    --prompt "Explain the difference between SWE-bench and GAIA." \
    --concurrency 8 \
    --output latency.csv

Cost Analysis for Multi-turn Agents

The agent-cost-analysis framework traces token-level usage across multi-turn agent workflows to calculate precise economic costs. This tool runs in chapter6/agent-cost-analysis/demo.py and breaks down expenses per task step【/cache/repos/github.com/bojieli/ai-agent-book/main/chapter6/agent-cost-analysis/README.md†L1-L7】.

Analyze costs for a customer-service agent task:

pip install -r chapter6/agent-cost-analysis/requirements.txt

python chapter6/agent-cost-analysis/demo.py \
    --task refund_customer \
    --model openai/gpt-4o-mini

The demo executes a realistic multi-turn dialogue, tracks token consumption at each step, and outputs a detailed cost table enabling data-driven decisions about model selection and prompt optimization.

Design Philosophy and Interpretation

The theoretical foundation for these evaluations is documented in book-en/chapter6.md, which explains benchmark design principles, data contamination challenges, and evaluation-driven agent selection workflows. The repository treats evaluation not merely as a testing phase but as an integral component of agent architecture design—what the documentation describes as the interplay between the evaluation stage and the dataset script【/cache/repos/github.com/bojieli/ai-agent-book/main/book-en/chapter6.md†L155-L167】.

When interpreting results across SWE-bench, GAIA, and OSWorld, consider that each benchmark captures a distinct capability vector: SWE-bench validates code correctness against ground-truth patches, GAIA measures information synthesis and tool use accuracy, and OSWorld evaluates UI interaction and system-level task completion rates.

Summary

• Three benchmark types: Use SWE-bench for code generation, GAIA for reasoning with tools, and OSWorld for desktop automation tasks.
• Repository location: All evaluation code resides in chapter6/, with benchmark wrappers in chapter6/SWE-bench/, chapter6/GAIA/, and chapter6/OSWorld/.
• Clone commands: External repositories must be cloned using the specific URLs provided in the root README's one-click script section.
• Comparison tools: The elo-leaderboard script enables head-to-head agent comparison using ELO ratings across any supported benchmark.
• Performance metrics: Use model-benchmark to measure API latency and agent-cost-analysis to calculate token-level economic costs for multi-turn workflows.
• Theoretical grounding: Implementation details and interpretation guidelines are documented in book-en/chapter6.md.

Frequently Asked Questions

How do I choose between SWE-bench, GAIA, and OSWorld for my agent?

Select SWE-bench if your agent generates or modifies code to resolve software issues, GAIA if your agent performs research and reasoning with search tools and APIs, and OSWorld if your agent interacts with graphical interfaces and operating system primitives. According to the ai-agent-book source code, these benchmarks are complementary—production agents often require evaluation across all three domains to assess general capability.

What is the minimum setup required to run these benchmarks?

You must clone the external repositories into the chapter6/ directory using the commands provided in the root README, then install dependencies via the requirements.txt files found in each tool subdirectory (e.g., chapter6/elo-leaderboard/requirements.txt). The benchmarks themselves require significant computational resources: SWE-bench needs Docker containers for safe code execution, while OSWorld requires a virtualized desktop environment.

How does the ELO leaderboard handle different benchmark types?

The elo-leaderboard/main.py script abstracts benchmark-specific scoring behind a unified interface. It loads tasks from the specified benchmark (SWE-bench, GAIA, or OSWorld), executes each agent against those tasks, and converts binary or graded success metrics into ELO rating adjustments. This allows direct comparison between agents even when their absolute scores come from different evaluation paradigms.

Where can I find the theoretical background on benchmark design?

The design principles, contamination risks, and methodological considerations are detailed in book-en/chapter6.md. This file explains why the repository structures evaluation as a stage (the execution environment) and the dataset as a script (the task specification), emphasizing that poor evaluation infrastructure introduces noise that masks true agent capabilities.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}