Evaluation Benchmarks for AI Agents: A Complete Guide to SWE-bench, GAIA, and OSWorld

SWE-bench, GAIA, and OSWorld are the three dominant evaluation benchmarks for AI agents, testing real-world code repair, multimodal reasoning with tool use, and full operating system control respectively, and the ai-agent-book repository provides production-ready wrappers and meta-evaluation tools for all three.

The bojieli/ai-agent-book repository structures Chapter 6 as a comprehensive Agent Evaluation module that bridges theoretical frameworks with hands-on implementation. According to the source code, the chapter treats the evaluation environment as the "stage" and the dataset as the "script"—with the quality of the script directly influencing the signal you obtain from the stage【/cache/repos/github.com/bojieli/ai-agent-book/main/book-en/chapter6.md†L155-L167】. This design philosophy is implemented through dedicated subdirectories that wrap external benchmark repositories while providing additional analysis utilities for comparing agents and measuring economic impact.

The Three Core Evaluation Benchmarks for AI Agents

The repository organizes the three major benchmarks under chapter6/, each serving a distinct evaluation purpose in the AI agent landscape.

SWE-bench: Real-World Software Engineering

SWE-bench measures an agent’s ability to generate correct patches for real GitHub issues drawn from production codebases. The benchmark includes multiple variants—Lite, Verified, and Multimodal—to accommodate different resource constraints and input modalities. In the repository structure, this benchmark lives at chapter6/SWE-bench/ and is referenced in the Chapter 6 overview table as the standard for code generation capabilities.

GAIA: Tool-Augmented Reasoning

GAIA evaluates tool-augmented, search-enabled autonomous reasoning through more than 450 non-trivial questions across three difficulty levels. Unlike code-specific benchmarks, GAIA tests an agent’s ability to orchestrate multiple tools and information sources to answer complex questions. The companion code resides in chapter6/GAIA/ and includes utilities for running the evaluation protocol against multimodal agents.

OSWorld: Full Operating System Control

OSWorld provides a realistic desktop environment for testing full-operating-system tasks including file management, application interaction, and system configuration. This benchmark assesses an agent’s capacity to navigate graphical user interfaces and execute long-horizon workflows in a sandboxed Ubuntu environment. The implementation is located at chapter6/OSWorld/ and supports both headless and visual evaluation modes.

Setting Up the Evaluation Environment

These benchmarks are external repositories that the ai-agent-book project wraps and extends. The root README.md provides a one-click clone script to pull all three dependencies simultaneously【/cache/repos/github.com/bojieli/ai-agent-book/main/README.md†L76-L88】.

Run the following commands from the repository root to set up the complete evaluation suite:

git clone https://github.com/SWE-bench/SWE-bench.git   chapter6/SWE-bench
git clone https://github.com/xlang-ai/GAIA.git       chapter6/GAIA
git clone https://github.com/xlang-ai/OSWorld.git    chapter6/OSWorld

After cloning, each subdirectory contains benchmark-specific setup instructions and the wrapper scripts necessary to integrate with the repository’s meta-evaluation tools.

Running Evaluations with Companion Tools

Beyond the benchmarks themselves, the repository provides three specialized tools for analyzing agent performance across these evaluation frameworks.

Cross-Benchmark Comparison with ELO Leaderboard

The elo-leaderboard utility enables statistical comparison of multiple agents across any of the three benchmarks. Located at chapter6/elo-leaderboard/, this tool implements an ELO rating system to rank agents based on head-to-head performance【/cache/repos/github.com/bojieli/ai-agent-book/main/chapter6/elo-leaderboard/README.md†L1-L12】.

pip install -r chapter6/elo-leaderboard/requirements.txt
python chapter6/elo-leaderboard/main.py \
    --benchmark SWE-bench \
    --agents agent_a.py agent_b.py \
    --output results.json

The script loads benchmark tasks, executes each agent to produce patches or answers, and calculates relative skill ratings.

Latency and Throughput Benchmarking

The model-benchmark tool measures API performance characteristics critical for production agent deployments. This utility, found in chapter6/model-benchmark/, records time-to-first-token (TTFT), p50/p95 latency percentiles, and success rates for different providers【/cache/repos/github.com/bojieli/ai-agent-book/main/chapter6/model-benchmark/README.md†L1-L9】.

pip install -r chapter6/model-benchmark/requirements.txt
python chapter6/model-benchmark/benchmark.py \
    --model openai/gpt-4o \
    --prompt "Explain the difference between SWE-bench and GAIA." \
    --concurrency 8 \
    --output latency.csv

Economic Cost Analysis for Multi-Turn Workflows

The agent-cost-analysis framework breaks down token-level expenses for complex agent workflows. Located at chapter6/agent-cost-analysis/, this tool traces usage per step in realistic scenarios like customer service dialogues【/cache/repos/github.com/bojieli/ai-agent-book/main/chapter6/agent-cost-analysis/README.md†L1-L7】.

pip install -r chapter6/agent-cost-analysis/requirements.txt
python chapter6/agent-cost-analysis/demo.py \
    --task refund_customer \
    --model openai/gpt-4o-mini

Benchmark Design Philosophy and Data Contamination

The theoretical foundation for these evaluations is documented in book-en/chapter6.md, which explains the design principles behind each benchmark and addresses critical issues like data contamination. The chapter emphasizes that evaluation-driven selection requires understanding how benchmark construction influences measured capabilities. According to the source, the repository implements a "stage and script" metaphor where the evaluation environment (stage) must be carefully matched to the dataset (script) to produce valid performance signals【/cache/repos/github.com/bojieli/ai-agent-book/main/book-en/chapter6.md†L155-L167】.

Summary

• SWE-bench, GAIA, and OSWorld represent the standard evaluation suite for AI agents, covering code generation, multimodal reasoning, and OS control respectively.
• The bojieli/ai-agent-book repository wraps these external benchmarks in chapter6/SWE-bench/, chapter6/GAIA/, and chapter6/OSWorld/ with one-click clone scripts available in the root README.
• Meta-evaluation tools including the ELO leaderboard, model benchmark, and cost analysis frameworks provide infrastructure for comparing agents and measuring production economics.
• The theoretical framework in book-en/chapter6.md emphasizes the relationship between evaluation environment design and signal quality, particularly regarding data contamination risks.

Frequently Asked Questions

What is SWE-bench used for?

SWE-bench evaluates an AI agent’s ability to automatically resolve real GitHub issues by generating correct code patches. It draws from actual bug reports and feature requests across popular Python repositories, testing both code understanding and generation capabilities in realistic software engineering contexts.

How does GAIA differ from OSWorld?

While both benchmarks test tool use, GAIA focuses on question-answering tasks requiring information retrieval and reasoning across modalities, whereas OSWorld tests direct manipulation of a computer desktop environment including file operations, application usage, and system configuration. GAIA measures research and synthesis capabilities; OSWorld measures GUI automation and long-horizon task execution.

Where are the evaluation scripts located in the repository?

The primary evaluation wrappers are located in chapter6/SWE-bench/, chapter6/GAIA/, and chapter6/OSWorld/ respectively. Additional meta-evaluation utilities reside in chapter6/elo-leaderboard/, chapter6/model-benchmark/, and chapter6/agent-cost-analysis/, with comprehensive documentation in chapter6/README.en.md.

How do I set up all three benchmarks at once?

The repository provides a one-click clone script in the root README.md that clones all three external benchmark repositories into their respective chapter6/ subdirectories. This ensures directory structure compatibility with the repository’s wrapper scripts and analysis tools.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}