# How to Evaluate AI Agent Performance Using Metrics and Statistical Significance

> Master your AI agent's performance with our guide on metrics and statistical significance. Learn to measure correctness, tool discovery, and reuse for impactful results.

- Repository: [Bojie Li/ai-agent-book](https://github.com/bojieli/ai-agent-book)
- Tags: best-practices
- Published: 2026-07-22

---

**AI Agent performance is evaluated through a four-layer scoring framework (L1–L4) that measures correctness, tool discovery, tool quality, and reuse efficiency, followed by statistical hypothesis testing to validate significant differences between agents.**

The `bojieli/ai-agent-book` repository provides a systematic, multi-layered evaluation framework designed for agents that generate, discover, and reuse tools. The core implementation resides in [`chapter8/self-evolution-eval/harness.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter8/self-evolution-eval/harness.py), which orchestrates distinct scoring layers and aggregation logic to produce reproducible performance profiles. This guide explains how to leverage these metrics and apply statistical rigor to compare AI agent implementations.

## The Four-Layer Evaluation Framework

The evaluation harness implements four complementary scoring layers that capture distinct aspects of agent behavior. Each layer produces a normalized score between 0 and 1, enabling granular analysis of specific capabilities.

### L1 – Correctness: Binary Task Completion

The **correctness layer** verifies whether the agent’s final answer matches the ground truth. Implemented in `layer1_correctness()` at lines 31–42 of [`harness.py`](https://github.com/bojieli/ai-agent-book/blob/main/harness.py), this layer returns a binary score:

- **Score**: 1.0 if correct, 0.0 if incorrect
- **Implementation**: Compares `trajectory["answer"]` against `task["Final answer"]`
- **Output**: Dictionary containing `score` and detailed reasoning

This layer serves as the foundation for evaluation, ensuring the agent solves the task regardless of the tools used.

### L2 – Tool Discovery: Measuring Exploration Efficiency

The **discovery layer** quantifies how effectively an agent identifies necessary tools from the available library. The `layer2_discovery()` function (lines 59–71) calculates:

- **Metric**: Fraction of discovered tools appearing in the "gold" library list (score ∈ [0, 1])
- **Logic**: Extracts the set of libraries used in the trajectory and computes the hit ratio against the task’s known toolbox
- **Use case**: Evaluates whether the agent avoids unnecessary tool generation by utilizing existing resources

### L3 – Tool Quality: LLM-as-Judge Assessment

The **quality layer** assesses the relevance and usefulness of generated tools using an external judge model. Implemented in `layer3_tool_quality()` (lines 136–169):

- **Metric**: Weighted average of judge-provided scores (0–1 scale)
- **Mechanism**: Calls a configurable `judge_model` that returns a JSON verdict for each tool
- **Aggregation**: Scores are aggregated and rounded to three decimal places for precision

This layer requires configuration via [`chapter8/self-evolution-eval/config.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter8/self-evolution-eval/config.py) to specify the judge LLM endpoint and prompts.

### L4 – Reuse Efficiency: Optimizing Tool Utilization

The **reuse layer** determines whether the agent efficiently leverages previously discovered tools rather than regenerating them. The `layer4_reuse()` function (lines 183–200) assigns categorical scores:

- **1.0**: Perfect reuse (no redundant generation)
- **0.5**: Partial reuse (some inefficiency detected)
- **0.0**: No reuse (complete regeneration of existing tools)

This layer inspects the variant trajectory for duplicated searches or unnecessary tool generation attempts.

## Aggregating Multi-Layer Scores

Individual layer scores feed into the **`aggregate()`** function (lines 211–224), which normalizes available scores and computes a unified performance report. The aggregation applies **equal weighting** (¼ contribution per layer) to balance correctness, discovery, quality, and reuse:

```python

# Simplified aggregation logic from harness.py

def aggregate(self, scores):
    # scores contains L1-L4 results (None if not applicable)

    valid_scores = [s for s in scores if s is not None]
    if not valid_scores:
        return {"score": None, "detail": "No valid scores"}
    
    final_score = sum(valid_scores) / len(valid_scores)
    return {
        "score": round(final_score, 3),
        "layers": scores,
        "detail": "Equal-weighted average of available layers"
    }

```

This equal-weighting approach provides an interpretable single metric while preserving the granular breakdown for diagnostic analysis.

## Measuring Statistical Significance

While the framework produces deterministic scores per run, meaningful comparison requires multiple trajectories across random seeds or task permutations. The repository documentation recommends standard statistical procedures using **NumPy** and **SciPy** to validate performance differences.

### Computing Confidence Intervals

After running multiple evaluations via [`demo.py`](https://github.com/bojieli/ai-agent-book/blob/main/demo.py), calculate confidence intervals for the aggregated scores:

```python
import numpy as np
from scipy import stats

# `reports` is a list of dicts returned by Evaluator.evaluate()

overall_scores = np.array([r["score"] for r in reports if r["score"] is not None])

mean = overall_scores.mean()
std_err = overall_scores.std(ddof=1) / np.sqrt(len(overall_scores))
ci_low, ci_high = stats.t.interval(
    alpha=0.95, 
    df=len(overall_scores)-1,
    loc=mean, 
    scale=std_err
)

print(f"Mean score = {mean:.3f} ± {std_err:.3f} (95% CI [{ci_low:.3f}, {ci_high:.3f}])")

```

### Comparative Hypothesis Testing

To determine if Agent A significantly outperforms Agent B:

1. **Paired t-test**: Use when comparing two agents on identical task sets to account for task difficulty variance
2. **Wilcoxon signed-rank test**: Apply when score distributions violate normality assumptions
3. **Effect size**: Calculate Cohen's d to measure practical significance beyond statistical significance

Execute these tests on per-task score vectors extracted from the JSON reports generated by the evaluation harness.

## Implementing the Evaluation Pipeline

Follow this workflow to obtain statistically robust performance measurements:

1. **Configure the environment**: Set the judge model in [`config.py`](https://github.com/bojieli/ai-agent-book/blob/main/config.py) for L3 quality assessment
2. **Run batch evaluation**: Execute `python -m chapter8.self-evolution-eval.demo` to generate trajectory reports across multiple seeds
3. **Extract metrics**: Parse the `reports` list containing L1–L4 scores and the aggregated result
4. **Analyze significance**: Apply the statistical tests above to compare configurations or agent versions
5. **Interpret layer breakdowns**: Identify specific weaknesses (e.g., high correctness but low reuse efficiency indicates redundant tool generation)

Key implementation files include:
- [`chapter8/self-evolution-eval/harness.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter8/self-evolution-eval/harness.py): Core scoring logic
- [`chapter8/self-evolution-eval/demo.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter8/self-evolution-eval/demo.py): Command-line driver for batch evaluation
- [`chapter8/self-evolution-eval/agent.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter8/self-evolution-eval/agent.py): Agent implementation for trajectory generation
- [`chapter8/self-evolution-eval/config.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter8/self-evolution-eval/config.py): Judge model configuration

## Summary

- **Four-layer framework**: L1 (Correctness), L2 (Tool Discovery), L3 (Tool Quality), and L4 (Reuse Efficiency) provide comprehensive coverage of agent capabilities
- **Equal-weighted aggregation**: The `aggregate()` function combines layers with ¼ weighting each, producing an interpretable 0–1 performance score
- **Statistical validation**: Use repeated trials with confidence intervals and paired t-tests to ensure observed differences are significant, not random
- **Implementation path**: Leverage [`harness.py`](https://github.com/bojieli/ai-agent-book/blob/main/harness.py) for scoring logic and [`demo.py`](https://github.com/bojieli/ai-agent-book/blob/main/demo.py) for batch execution, then analyze results with standard scientific Python libraries

## Frequently Asked Questions

### How does the four-layer scoring handle tasks that don't require tool generation?

The `aggregate()` function automatically excludes `None` values for inapplicable layers. If a task requires no tool discovery or reuse, only L1 (Correctness) and potentially L3 (Quality) contribute to the final score, with weights rebalanced equally among available layers.

### What judge model configuration is recommended for the L3 quality assessment?

According to [`config.py`](https://github.com/bojieli/ai-agent-book/blob/main/config.py), you should configure a capable LLM (typically GPT-4 or equivalent) as the `judge_model`. The judge evaluates tool utility based on the task requirements, returning JSON verdicts that `layer3_tool_quality()` parses into numerical scores rounded to three decimals.

### Can I modify the weighting of different scoring layers?

While the default `aggregate()` implementation uses equal ¼ weighting, the source code in [`harness.py`](https://github.com/bojieli/ai-agent-book/blob/main/harness.py) (lines 211–224) provides a clear template for customization. You can subclass the `Evaluator` and override `aggregate()` to apply domain-specific weightings (e.g., emphasizing correctness over reuse for critical applications).

### How many trajectories are needed for statistically significant results?

The repository documentation suggests running **at least 30 trajectories** per agent configuration to satisfy the Central Limit Theorem for parametric tests. For high-stakes comparisons, 100+ runs provide narrower confidence intervals and greater power to detect small effect sizes in the 0.05–0.10 score range.