Harness Engineering for AI Agents: The Four-Layer Evaluation Framework

Harness engineering is a systematic four-layer evaluation methodology that assesses AI agent behavior across task correctness, tool discovery, tool creation quality, and tool reuse capability to ensure autonomous systems are tested holistically rather than by a single aggregate metric.

In the bojieli/ai-agent-book repository, harness engineering is implemented as a structured testing framework within chapter8/self-evolution-eval/harness.py. This approach addresses the unique challenge of evaluating AI agents capable of self-evolution—systems that autonomously discover, create, and reuse tools—by providing modular, reproducible metrics that extend far beyond simple pass/fail scoring.

What is Harness Engineering?

Harness engineering refers to a systematic evaluation architecture designed to provide repeatable, modular, and granular assessment of an AI agent's behavior. The harness takes two primary inputs—a trajectory (the complete sequence of actions an agent performed on a task) and the task definition—and produces orthogonal scores across four distinct dimensions.

L1: Task Correctness

The layer1_correctness function in harness.py checks the final answer against the task's correctness_criteria using regex or keyword matching. This foundational layer ensures the agent actually solves the problem before considering how it arrived at the solution.

L2: Tool-Discovery Effectiveness

Implemented in layer2_discovery, this layer heuristically evaluates whether the agent's search queries were on-topic, whether it visited relevant web pages, and whether it selected recommended libraries while avoiding known pitfalls. It measures the efficiency of the agent's research phase during tool discovery.

L3: Tool-Creation Quality

The layer3_tool_quality function sends generated tool code to an LLM-as-a-Judge (configured via Config.JUDGE_MODEL) and scores it on error handling, input validation, documentation, and robustness using a predefined rubric. This layer requires an external language model to evaluate code quality.

L4: Tool-Reuse Capability

The layer4_reuse function analyzes a variant trajectory (a second execution of a similar task) to determine if the agent retrieves a previously registered tool from its registry instead of redundantly re-researching or re-creating it. This measures the agent's ability to leverage its own learned capabilities.

Why Harness Engineering Matters for AI Agents

Comprehensive Coverage – By separating correctness, discovery, creation, and reuse into distinct layers, the harness forces developers to consider all aspects of an autonomous agent, not just end-to-end task completion.

Modular and Extensible Architecture – Each layer is implemented as a pure function (layer1_*, layer2_*, etc.) that can be swapped out or extended without modifying other components, encouraging clean architecture and facilitating future research iterations.

Reproducible Benchmarks – The same trajectory and task JSON can be fed to the harness repeatedly, guaranteeing deterministic scores for L1, L2, and L4, while L3 can be reproduced consistently with a fixed judge model.

Actionable Diagnostics – Detailed per-layer detail strings (e.g., "搜索切题=True 访问网页=False") provide concrete clues about where an agent succeeds or fails, enabling targeted improvements rather than blind optimization.

Alignment with Production Constraints – L4's focus on re-using registered tools mirrors real-world deployment scenarios where redundant API calls or duplicated code generation are costly and inefficient.

Implementation Architecture

The FourLayerEvaluator class serves as the primary interface, wrapping the evaluation process and allowing callers to select which layers to run via the layers parameter. After individual layers produce scores (possibly None for skipped layers), the aggregate function recombines them using predefined weights:

LAYER_WEIGHTS = {"L1": 0.35, "L2": 0.25, "L3": 0.25, "L4": 0.15}

The system normalizes these weights dynamically if certain layers are excluded, ensuring the overall rating remains consistent even when running in offline mode (skipping L3) or focusing on specific capabilities.

Practical Usage Examples

Running a Full Evaluation (Online Mode)

from chapter8.self_evolution_eval.harness import FourLayerEvaluator, ALL_LAYERS
from chapter8.self_evolution_eval.agent import SelfEvolutionAgent, STRONG
from chapter8.self_evolution_eval.config import Config

# Assume `task` and `trajectory` are dicts loaded from the dataset

evaluator = FourLayerEvaluator(judge_model=Config.JUDGE_MODEL, layers=ALL_LAYERS)

# First run (discovery + creation)

agent = SelfEvolutionAgent(model=Config.AGENT_MODEL)
first_trajectory = agent.run(task, profile=STRONG, use_variant=False)

# Second run (reuse probe)

variant_trajectory = agent.run(task, profile=STRONG, use_variant=True)

report = evaluator.evaluate(task, first_trajectory, variant_trajectory)
print(report)   # Contains per-layer scores and the overall rating

Running Only Deterministic Layers (Offline)

from chapter8.self_evolution_eval.harness import FourLayerEvaluator

offline_layers = ("L1", "L2", "L4")          # Skip L3 which requires an LLM judge

evaluator = FourLayerEvaluator(layers=offline_layers)

# `task` and `trajectory` are the same as above but no LLM call is made

report = evaluator.evaluate(task, trajectory, variant_trajectory=None)
print(report["summary"]["overall"])          # Overall score based on L1/L2/L4 only

Summary

• Harness engineering provides a four-layer evaluation framework (L1-L4) that assesses task correctness, discovery effectiveness, creation quality, and reuse capability.
• The implementation in chapter8/self-evolution-eval/harness.py uses pure functions for each layer, enabling modular testing and extension.
• Weighted aggregation (L1: 0.35, L2: 0.25, L3: 0.25, L4: 0.15) produces a holistic score while allowing selective layer execution via FourLayerEvaluator.
• The framework supports both online evaluation (with LLM-as-a-Judge for L3) and offline evaluation (deterministic layers only), making it suitable for CI pipelines and research benchmarks.
• Detailed per-layer diagnostics enable developers to identify specific weaknesses in agent behavior, from search strategy inefficiencies to redundant tool creation.

Frequently Asked Questions

How does harness engineering differ from traditional AI evaluation?

Traditional evaluation often relies on single-metric accuracy or end-to-end success rates. Harness engineering decomposes agent behavior into four orthogonal dimensions—correctness, discovery, creation, and reuse—allowing developers to identify whether an agent fails at problem-solving, research, implementation, or memory optimization specifically.

Can I run the harness without using an LLM judge?

Yes. By passing layers=("L1", "L2", "L4") to the FourLayerEvaluator constructor, you can execute only the deterministic layers that rely on regex matching, heuristic analysis, and trajectory comparison. This "offline mode" skips layer3_tool_quality, which requires the configured Config.JUDGE_MODEL to evaluate code quality.

What does the L4 reuse layer measure specifically?

The layer4_reuse function compares a primary trajectory against a variant trajectory (generated with use_variant=True) to detect whether the agent retrieves a previously registered tool from its internal registry. It measures the agent's ability to recognize similar tasks and avoid redundant research or code generation, which is critical for efficient long-term operation.

How are the layer weights determined in the aggregate score?

The default weights are defined as LAYER_WEIGHTS = {"L1": 0.35, "L2": 0.25, "L3": 0.25, "L4": 0.15} in the harness configuration. These values prioritize task correctness (L1) while balancing the importance of discovery (L2) and creation quality (L3), with reuse capability (L4) receiving slightly less weight. The aggregate function normalizes these weights dynamically if you exclude specific layers from evaluation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}