# Harness Engineering for AI Agents: The Four-Layer Evaluation Framework

> Discover harness engineering a four-layer AI agent evaluation framework for task correctness, tool discovery, quality, and reuse. Ensure thorough testing for autonomous systems.

- Repository: [Bojie Li/ai-agent-book](https://github.com/bojieli/ai-agent-book)
- Tags: deep-dive
- Published: 2026-07-22

---

**Harness engineering is a systematic four-layer evaluation methodology that assesses AI agent behavior across task correctness, tool discovery, tool creation quality, and tool reuse capability to ensure autonomous systems are tested holistically rather than by a single aggregate metric.**

In the `bojieli/ai-agent-book` repository, harness engineering is implemented as a structured testing framework within [`chapter8/self-evolution-eval/harness.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter8/self-evolution-eval/harness.py). This approach addresses the unique challenge of evaluating AI agents capable of **self-evolution**—systems that autonomously discover, create, and reuse tools—by providing modular, reproducible metrics that extend far beyond simple pass/fail scoring.

## What is Harness Engineering?

Harness engineering refers to a systematic evaluation architecture designed to provide **repeatable, modular, and granular assessment** of an AI agent's behavior. The harness takes two primary inputs—a **trajectory** (the complete sequence of actions an agent performed on a task) and the **task definition**—and produces orthogonal scores across four distinct dimensions.

### L1: Task Correctness

The `layer1_correctness` function in [`harness.py`](https://github.com/bojieli/ai-agent-book/blob/main/harness.py) checks the final answer against the task's `correctness_criteria` using regex or keyword matching. This foundational layer ensures the agent actually solves the problem before considering how it arrived at the solution.

### L2: Tool-Discovery Effectiveness

Implemented in `layer2_discovery`, this layer heuristically evaluates whether the agent's search queries were on-topic, whether it visited relevant web pages, and whether it selected recommended libraries while avoiding known pitfalls. It measures the efficiency of the agent's research phase during tool discovery.

### L3: Tool-Creation Quality

The `layer3_tool_quality` function sends generated tool code to an **LLM-as-a-Judge** (configured via `Config.JUDGE_MODEL`) and scores it on error handling, input validation, documentation, and robustness using a predefined rubric. This layer requires an external language model to evaluate code quality.

### L4: Tool-Reuse Capability

The `layer4_reuse` function analyzes a **variant trajectory** (a second execution of a similar task) to determine if the agent retrieves a previously registered tool from its registry instead of redundantly re-researching or re-creating it. This measures the agent's ability to leverage its own learned capabilities.

## Why Harness Engineering Matters for AI Agents

**Comprehensive Coverage** – By separating correctness, discovery, creation, and reuse into distinct layers, the harness forces developers to consider *all* aspects of an autonomous agent, not just end-to-end task completion.

**Modular and Extensible Architecture** – Each layer is implemented as a pure function (`layer1_*`, `layer2_*`, etc.) that can be swapped out or extended without modifying other components, encouraging clean architecture and facilitating future research iterations.

**Reproducible Benchmarks** – The same trajectory and task JSON can be fed to the harness repeatedly, guaranteeing deterministic scores for L1, L2, and L4, while L3 can be reproduced consistently with a fixed judge model.

**Actionable Diagnostics** – Detailed per-layer `detail` strings (e.g., "搜索切题=True 访问网页=False") provide concrete clues about where an agent succeeds or fails, enabling targeted improvements rather than blind optimization.

**Alignment with Production Constraints** – L4's focus on re-using registered tools mirrors real-world deployment scenarios where redundant API calls or duplicated code generation are costly and inefficient.

## Implementation Architecture

The `FourLayerEvaluator` class serves as the primary interface, wrapping the evaluation process and allowing callers to select which layers to run via the `layers` parameter. After individual layers produce scores (possibly `None` for skipped layers), the `aggregate` function recombines them using predefined weights:

```python
LAYER_WEIGHTS = {"L1": 0.35, "L2": 0.25, "L3": 0.25, "L4": 0.15}

```

The system normalizes these weights dynamically if certain layers are excluded, ensuring the overall rating remains consistent even when running in offline mode (skipping L3) or focusing on specific capabilities.

## Practical Usage Examples

### Running a Full Evaluation (Online Mode)

```python
from chapter8.self_evolution_eval.harness import FourLayerEvaluator, ALL_LAYERS
from chapter8.self_evolution_eval.agent import SelfEvolutionAgent, STRONG
from chapter8.self_evolution_eval.config import Config

# Assume `task` and `trajectory` are dicts loaded from the dataset

evaluator = FourLayerEvaluator(judge_model=Config.JUDGE_MODEL, layers=ALL_LAYERS)

# First run (discovery + creation)

agent = SelfEvolutionAgent(model=Config.AGENT_MODEL)
first_trajectory = agent.run(task, profile=STRONG, use_variant=False)

# Second run (reuse probe)

variant_trajectory = agent.run(task, profile=STRONG, use_variant=True)

report = evaluator.evaluate(task, first_trajectory, variant_trajectory)
print(report)   # Contains per-layer scores and the overall rating

```

### Running Only Deterministic Layers (Offline)

```python
from chapter8.self_evolution_eval.harness import FourLayerEvaluator

offline_layers = ("L1", "L2", "L4")          # Skip L3 which requires an LLM judge

evaluator = FourLayerEvaluator(layers=offline_layers)

# `task` and `trajectory` are the same as above but no LLM call is made

report = evaluator.evaluate(task, trajectory, variant_trajectory=None)
print(report["summary"]["overall"])          # Overall score based on L1/L2/L4 only

```

## Summary

- Harness engineering provides a **four-layer evaluation framework** (L1-L4) that assesses task correctness, discovery effectiveness, creation quality, and reuse capability.
- The implementation in [`chapter8/self-evolution-eval/harness.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter8/self-evolution-eval/harness.py) uses pure functions for each layer, enabling modular testing and extension.
- **Weighted aggregation** (`L1: 0.35, L2: 0.25, L3: 0.25, L4: 0.15`) produces a holistic score while allowing selective layer execution via `FourLayerEvaluator`.
- The framework supports both **online evaluation** (with LLM-as-a-Judge for L3) and **offline evaluation** (deterministic layers only), making it suitable for CI pipelines and research benchmarks.
- Detailed per-layer diagnostics enable developers to identify specific weaknesses in agent behavior, from search strategy inefficiencies to redundant tool creation.

## Frequently Asked Questions

### How does harness engineering differ from traditional AI evaluation?

Traditional evaluation often relies on single-metric accuracy or end-to-end success rates. Harness engineering decomposes agent behavior into four orthogonal dimensions—correctness, discovery, creation, and reuse—allowing developers to identify whether an agent fails at problem-solving, research, implementation, or memory optimization specifically.

### Can I run the harness without using an LLM judge?

Yes. By passing `layers=("L1", "L2", "L4")` to the `FourLayerEvaluator` constructor, you can execute only the deterministic layers that rely on regex matching, heuristic analysis, and trajectory comparison. This "offline mode" skips `layer3_tool_quality`, which requires the configured `Config.JUDGE_MODEL` to evaluate code quality.

### What does the L4 reuse layer measure specifically?

The `layer4_reuse` function compares a primary trajectory against a **variant trajectory** (generated with `use_variant=True`) to detect whether the agent retrieves a previously registered tool from its internal registry. It measures the agent's ability to recognize similar tasks and avoid redundant research or code generation, which is critical for efficient long-term operation.

### How are the layer weights determined in the aggregate score?

The default weights are defined as `LAYER_WEIGHTS = {"L1": 0.35, "L2": 0.25, "L3": 0.25, "L4": 0.15}` in the harness configuration. These values prioritize task correctness (L1) while balancing the importance of discovery (L2) and creation quality (L3), with reuse capability (L4) receiving slightly less weight. The `aggregate` function normalizes these weights dynamically if you exclude specific layers from evaluation.