# How Forge Measures Tool-Calling Reliability: Message History Analysis in the Evaluation Harness

> Forge measures tool-calling reliability by analyzing message history. Discover how retry nudges, step violations, tool errors, and wasted calls are quantified within the evaluation harness.

- Repository: [Antoine/forge](https://github.com/antoinezambelli/forge)
- Tags: deep-dive
- Published: 2026-05-22

---

**The Forge evaluation harness quantifies tool-calling reliability by analyzing the complete message history of scenario runs to count retry nudges, step violations, tool errors, and wasted calls.**

The `antoinezambelli/forge` repository provides a robust evaluation framework for assessing how reliably large language models interact with external tools. Understanding tool-calling reliability is critical for production AI systems where consistent, error-free tool usage directly impacts task completion rates. The harness implements a message-history-based analytics pipeline that transforms raw execution traces into quantitative reliability scores.

## Capturing Message History During Scenario Execution

The evaluation process begins in [`tests/eval/eval_runner.py`](https://github.com/antoinezambelli/forge/blob/main/tests/eval/eval_runner.py), where the `run_scenario` function orchestrates workflow execution. When the `keep_message_history` configuration flag is enabled, the harness appends a callback to collect every `Message` object emitted during the run.

```python

# tests/eval/eval_runner.py

if config.keep_message_history:
    callbacks.append(collected_messages.append)

```

These messages contain metadata tags defined in [`src/forge/core/messages.py`](https://github.com/antoinezambelli/forge/blob/main/src/forge/core/messages.py), including `MessageType.TOOL_CALL`, `MessageType.RETRY_NUDGE`, `MessageType.STEP_NUDGE`, and `MessageType.TOOL_RESULT`. This type system enables the harness to classify and count specific interaction patterns automatically without parsing raw text.

## Computing Reliability Metrics from Message History

After scenario execution, the `analyze_history` function in [`tests/eval/metrics.py`](https://github.com/antoinezambelli/forge/blob/main/tests/eval/metrics.py) processes the collected messages to produce a `HistoryStats` object. This function tallies critical reliability indicators by matching against the `MessageType` enumeration:

```python

# tests/eval/metrics.py

for msg in messages:
    match msg.metadata.type:
        case MessageType.TOOL_CALL:   stats.total_tool_calls += 1
        case MessageType.RETRY_NUDGE: stats.retry_nudges += 1
        case MessageType.STEP_NUDGE:  stats.step_nudges += 1
        case MessageType.TOOL_RESULT:
            if "[ToolError]" in msg.content: stats.tool_errors += 1
        case MessageType.REASONING:   stats.reasoning_messages += 1

```

The resulting statistics include `total_tool_calls` for volume analysis, `unique_tools_called` for diversity measurement, and error-specific counters like `retry_nudges`, `step_nudges`, and `tool_errors` that directly indicate reliability issues.

## Aggregating Reliability Scores Across Runs

For statistical significance, the harness aggregates per-run statistics using `compute_metrics` in [`tests/eval/metrics.py`](https://github.com/antoinezambelli/forge/blob/main/tests/eval/metrics.py). This produces scenario-level averages that quantify tool-calling reliability:

- **avg_retry_nudges**: Mean retry nudges per run, indicating transient failure recovery frequency
- **avg_step_nudges**: Mean step-enforcement nudges, measuring workflow adherence violations
- **avg_tool_errors**: Mean `[ToolError]` occurrences per run
- **avg_wasted_calls**: Extra tool calls beyond the ideal iteration count

These aggregated fields provide a numeric picture of how reliably the model uses tools without requiring manual inspection of individual execution traces.

## Implementing Custom Reliability Analysis

Developers can leverage the evaluation API to programmatically assess tool-calling reliability for specific scenarios:

```python
from tests.eval.metrics import compute_metrics, analyze_history

# `results` is the list of RunResult objects for a single scenario

scenario_metrics = compute_metrics(scenario, results)

print("Avg retry nudges:", scenario_metrics.avg_retry_nudges)
print("Avg step nudges:", scenario_metrics.avg_step_nudges)
print("Avg tool errors:", scenario_metrics.avg_tool_errors)
print("Avg wasted calls:", scenario_metrics.avg_wasted_calls)

```

For single-run debugging, the `analyze_history` function provides immediate feedback on specific execution patterns:

```python
from tests.eval.metrics import analyze_history

# `msgs` is a list of Message objects from a run

stats = analyze_history(msgs)
print(f"Total calls: {stats.total_tool_calls}, Retries: {stats.retry_nudges}")

```

## Reporting and Interpretation

The `print_report` function in [`tests/eval/report.py`](https://github.com/antoinezambelli/forge/blob/main/tests/eval/report.py) generates human-readable summaries incorporating these reliability metrics alongside completion and timing data. High tool-calling reliability manifests as low averages across all nudge and error counters, while elevated `avg_wasted_calls` specifically indicates inefficient or redundant tool selection patterns that require workflow optimization.

## Summary

- The Forge harness measures tool-calling reliability by analyzing `Message` history stored during scenario execution when `config.keep_message_history` is enabled.
- The `analyze_history` function in [`tests/eval/metrics.py`](https://github.com/antoinezambelli/forge/blob/main/tests/eval/metrics.py) categorizes messages by `MessageType` to count retry nudges, step nudges, and tool errors containing `[ToolError]`.
- Aggregated metrics like `avg_retry_nudges` and `avg_wasted_calls` provide quantitative reliability scores across multiple scenario runs via `compute_metrics`.
- Retry nudges signal transient failures requiring automatic recovery, while step nudges indicate violations of required workflow step ordering.
- The reporting pipeline in [`tests/eval/report.py`](https://github.com/antoinezambelli/forge/blob/main/tests/eval/report.py) presents these statistics alongside completion and correctness metrics for comprehensive evaluation.

## Frequently Asked Questions

### What is a "retry nudge" in Forge's evaluation system?

A retry nudge is a `MessageType.RETRY_NUDGE` message emitted when the workflow runner automatically retries a tool call after a transient failure. The harness counts these in `analyze_history` to calculate `avg_retry_nudges`, which measures how often the model's initial tool calls fail and require recovery attempts, directly indicating tool-calling reliability issues.

### How does Forge distinguish between successful and failed tool calls?

The system inspects `MessageType.TOOL_RESULT` messages for the presence of `[ToolError]` in the content string. Results containing this substring increment the `tool_errors` counter in `HistoryStats`, while successful calls lack this error marker. This detection occurs within the message processing loop in [`tests/eval/metrics.py`](https://github.com/antoinezambelli/forge/blob/main/tests/eval/metrics.py).

### What constitutes a "wasted call" in the reliability metrics?

Wasted calls represent tool invocations that exceed the optimal number required to complete a scenario. The `compute_metrics` function calculates `avg_wasted_calls` by comparing actual tool call counts against the ideal iteration count for the scenario, highlighting inefficient, redundant, or exploratory tool usage patterns that degrade performance.

### Where can I configure message history collection?

Message history collection is controlled through the `EvalConfig` dataclass in [`tests/eval/eval_runner.py`](https://github.com/antoinezambelli/forge/blob/main/tests/eval/eval_runner.py). Setting `keep_message_history=True` enables the callback mechanism that appends messages to the collection list, which is subsequently passed to `analyze_history` for reliability analysis.