# Evaluation Benchmarks for AI Agents: A Complete Guide to SWE-bench, GAIA, and OSWorld > Master AI agent evaluation with SWE-bench, GAIA, and OSWorld benchmarks. This guide covers real-world code repair, multimodal reasoning, and OS control. Explore production-ready tools and wrappers to effectively assess AI agent... - Repository: [Bojie Li/ai-agent-book](https://github.com/bojieli/ai-agent-book) - Tags: tutorial - Published: 2026-07-21 --- **SWE-bench, GAIA, and OSWorld are the three dominant evaluation benchmarks for AI agents**, testing real-world code repair, multimodal reasoning with tool use, and full operating system control respectively, and the ai-agent-book repository provides production-ready wrappers and meta-evaluation tools for all three. The `bojieli/ai-agent-book` repository structures **Chapter 6** as a comprehensive Agent Evaluation module that bridges theoretical frameworks with hands-on implementation. According to the source code, the chapter treats the evaluation environment as the "stage" and the dataset as the "script"—with the quality of the script directly influencing the signal you obtain from the stage【/cache/repos/github.com/bojieli/ai-agent-book/main/book-en/chapter6.md†L155-L167】. This design philosophy is implemented through dedicated subdirectories that wrap external benchmark repositories while providing additional analysis utilities for comparing agents and measuring economic impact. ## The Three Core Evaluation Benchmarks for AI Agents The repository organizes the three major benchmarks under `chapter6/`, each serving a distinct evaluation purpose in the AI agent landscape. ### SWE-bench: Real-World Software Engineering **SWE-bench** measures an agent’s ability to generate correct patches for real GitHub issues drawn from production codebases. The benchmark includes multiple variants—**Lite**, **Verified**, and **Multimodal**—to accommodate different resource constraints and input modalities. In the repository structure, this benchmark lives at `chapter6/SWE-bench/` and is referenced in the Chapter 6 overview table as the standard for code generation capabilities. ### GAIA: Tool-Augmented Reasoning **GAIA** evaluates tool-augmented, search-enabled autonomous reasoning through more than 450 non-trivial questions across three difficulty levels. Unlike code-specific benchmarks, GAIA tests an agent’s ability to orchestrate multiple tools and information sources to answer complex questions. The companion code resides in `chapter6/GAIA/` and includes utilities for running the evaluation protocol against multimodal agents. ### OSWorld: Full Operating System Control **OSWorld** provides a realistic desktop environment for testing full-operating-system tasks including file management, application interaction, and system configuration. This benchmark assesses an agent’s capacity to navigate graphical user interfaces and execute long-horizon workflows in a sandboxed Ubuntu environment. The implementation is located at `chapter6/OSWorld/` and supports both headless and visual evaluation modes. ## Setting Up the Evaluation Environment These benchmarks are external repositories that the ai-agent-book project wraps and extends. The root [`README.md`](https://github.com/bojieli/ai-agent-book/blob/main/README.md) provides a one-click clone script to pull all three dependencies simultaneously【/cache/repos/github.com/bojieli/ai-agent-book/main/README.md†L76-L88】. Run the following commands from the repository root to set up the complete evaluation suite: ```bash git clone https://github.com/SWE-bench/SWE-bench.git chapter6/SWE-bench git clone https://github.com/xlang-ai/GAIA.git chapter6/GAIA git clone https://github.com/xlang-ai/OSWorld.git chapter6/OSWorld ``` After cloning, each subdirectory contains benchmark-specific setup instructions and the wrapper scripts necessary to integrate with the repository’s meta-evaluation tools. ## Running Evaluations with Companion Tools Beyond the benchmarks themselves, the repository provides three specialized tools for analyzing agent performance across these evaluation frameworks. ### Cross-Benchmark Comparison with ELO Leaderboard The `elo-leaderboard` utility enables statistical comparison of multiple agents across any of the three benchmarks. Located at `chapter6/elo-leaderboard/`, this tool implements an ELO rating system to rank agents based on head-to-head performance【/cache/repos/github.com/bojieli/ai-agent-book/main/chapter6/elo-leaderboard/README.md†L1-L12】. ```bash pip install -r chapter6/elo-leaderboard/requirements.txt python chapter6/elo-leaderboard/main.py \ --benchmark SWE-bench \ --agents agent_a.py agent_b.py \ --output results.json ``` The script loads benchmark tasks, executes each agent to produce patches or answers, and calculates relative skill ratings. ### Latency and Throughput Benchmarking The `model-benchmark` tool measures API performance characteristics critical for production agent deployments. This utility, found in `chapter6/model-benchmark/`, records time-to-first-token (TTFT), p50/p95 latency percentiles, and success rates for different providers【/cache/repos/github.com/bojieli/ai-agent-book/main/chapter6/model-benchmark/README.md†L1-L9】. ```bash pip install -r chapter6/model-benchmark/requirements.txt python chapter6/model-benchmark/benchmark.py \ --model openai/gpt-4o \ --prompt "Explain the difference between SWE-bench and GAIA." \ --concurrency 8 \ --output latency.csv ``` ### Economic Cost Analysis for Multi-Turn Workflows The `agent-cost-analysis` framework breaks down token-level expenses for complex agent workflows. Located at `chapter6/agent-cost-analysis/`, this tool traces usage per step in realistic scenarios like customer service dialogues【/cache/repos/github.com/bojieli/ai-agent-book/main/chapter6/agent-cost-analysis/README.md†L1-L7】. ```bash pip install -r chapter6/agent-cost-analysis/requirements.txt python chapter6/agent-cost-analysis/demo.py \ --task refund_customer \ --model openai/gpt-4o-mini ``` ## Benchmark Design Philosophy and Data Contamination The theoretical foundation for these evaluations is documented in [`book-en/chapter6.md`](https://github.com/bojieli/ai-agent-book/blob/main/book-en/chapter6.md), which explains the design principles behind each benchmark and addresses critical issues like data contamination. The chapter emphasizes that **evaluation-driven selection** requires understanding how benchmark construction influences measured capabilities. According to the source, the repository implements a "stage and script" metaphor where the evaluation environment (stage) must be carefully matched to the dataset (script) to produce valid performance signals【/cache/repos/github.com/bojieli/ai-agent-book/main/book-en/chapter6.md†L155-L167】. ## Summary - **SWE-bench**, **GAIA**, and **OSWorld** represent the standard evaluation suite for AI agents, covering code generation, multimodal reasoning, and OS control respectively. - The `bojieli/ai-agent-book` repository wraps these external benchmarks in `chapter6/SWE-bench/`, `chapter6/GAIA/`, and `chapter6/OSWorld/` with one-click clone scripts available in the root README. - **Meta-evaluation tools** including the ELO leaderboard, model benchmark, and cost analysis frameworks provide infrastructure for comparing agents and measuring production economics. - The theoretical framework in [`book-en/chapter6.md`](https://github.com/bojieli/ai-agent-book/blob/main/book-en/chapter6.md) emphasizes the relationship between evaluation environment design and signal quality, particularly regarding data contamination risks. ## Frequently Asked Questions ### What is SWE-bench used for? SWE-bench evaluates an AI agent’s ability to automatically resolve real GitHub issues by generating correct code patches. It draws from actual bug reports and feature requests across popular Python repositories, testing both code understanding and generation capabilities in realistic software engineering contexts. ### How does GAIA differ from OSWorld? While both benchmarks test tool use, **GAIA** focuses on question-answering tasks requiring information retrieval and reasoning across modalities, whereas **OSWorld** tests direct manipulation of a computer desktop environment including file operations, application usage, and system configuration. GAIA measures research and synthesis capabilities; OSWorld measures GUI automation and long-horizon task execution. ### Where are the evaluation scripts located in the repository? The primary evaluation wrappers are located in `chapter6/SWE-bench/`, `chapter6/GAIA/`, and `chapter6/OSWorld/` respectively. Additional meta-evaluation utilities reside in `chapter6/elo-leaderboard/`, `chapter6/model-benchmark/`, and `chapter6/agent-cost-analysis/`, with comprehensive documentation in [`chapter6/README.en.md`](https://github.com/bojieli/ai-agent-book/blob/main/chapter6/README.en.md). ### How do I set up all three benchmarks at once? The repository provides a one-click clone script in the root [`README.md`](https://github.com/bojieli/ai-agent-book/blob/main/README.md) that clones all three external benchmark repositories into their respective `chapter6/` subdirectories. This ensures directory structure compatibility with the repository’s wrapper scripts and analysis tools.