How Forge Measures Tool-Calling Reliability: Message History Analysis in the Evaluation Harness

The Forge evaluation harness quantifies tool-calling reliability by analyzing the complete message history of scenario runs to count retry nudges, step violations, tool errors, and wasted calls.

The antoinezambelli/forge repository provides a robust evaluation framework for assessing how reliably large language models interact with external tools. Understanding tool-calling reliability is critical for production AI systems where consistent, error-free tool usage directly impacts task completion rates. The harness implements a message-history-based analytics pipeline that transforms raw execution traces into quantitative reliability scores.

Capturing Message History During Scenario Execution

The evaluation process begins in tests/eval/eval_runner.py, where the run_scenario function orchestrates workflow execution. When the keep_message_history configuration flag is enabled, the harness appends a callback to collect every Message object emitted during the run.

# tests/eval/eval_runner.py

if config.keep_message_history:
    callbacks.append(collected_messages.append)

These messages contain metadata tags defined in src/forge/core/messages.py, including MessageType.TOOL_CALL, MessageType.RETRY_NUDGE, MessageType.STEP_NUDGE, and MessageType.TOOL_RESULT. This type system enables the harness to classify and count specific interaction patterns automatically without parsing raw text.

Computing Reliability Metrics from Message History

After scenario execution, the analyze_history function in tests/eval/metrics.py processes the collected messages to produce a HistoryStats object. This function tallies critical reliability indicators by matching against the MessageType enumeration:

# tests/eval/metrics.py

for msg in messages:
    match msg.metadata.type:
        case MessageType.TOOL_CALL:   stats.total_tool_calls += 1
        case MessageType.RETRY_NUDGE: stats.retry_nudges += 1
        case MessageType.STEP_NUDGE:  stats.step_nudges += 1
        case MessageType.TOOL_RESULT:
            if "[ToolError]" in msg.content: stats.tool_errors += 1
        case MessageType.REASONING:   stats.reasoning_messages += 1

The resulting statistics include total_tool_calls for volume analysis, unique_tools_called for diversity measurement, and error-specific counters like retry_nudges, step_nudges, and tool_errors that directly indicate reliability issues.

Aggregating Reliability Scores Across Runs

For statistical significance, the harness aggregates per-run statistics using compute_metrics in tests/eval/metrics.py. This produces scenario-level averages that quantify tool-calling reliability:

• avg_retry_nudges: Mean retry nudges per run, indicating transient failure recovery frequency
• avg_step_nudges: Mean step-enforcement nudges, measuring workflow adherence violations
• avg_tool_errors: Mean [ToolError] occurrences per run
• avg_wasted_calls: Extra tool calls beyond the ideal iteration count

These aggregated fields provide a numeric picture of how reliably the model uses tools without requiring manual inspection of individual execution traces.

Implementing Custom Reliability Analysis

Developers can leverage the evaluation API to programmatically assess tool-calling reliability for specific scenarios:

from tests.eval.metrics import compute_metrics, analyze_history

# `results` is the list of RunResult objects for a single scenario

scenario_metrics = compute_metrics(scenario, results)

print("Avg retry nudges:", scenario_metrics.avg_retry_nudges)
print("Avg step nudges:", scenario_metrics.avg_step_nudges)
print("Avg tool errors:", scenario_metrics.avg_tool_errors)
print("Avg wasted calls:", scenario_metrics.avg_wasted_calls)

For single-run debugging, the analyze_history function provides immediate feedback on specific execution patterns:

from tests.eval.metrics import analyze_history

# `msgs` is a list of Message objects from a run

stats = analyze_history(msgs)
print(f"Total calls: {stats.total_tool_calls}, Retries: {stats.retry_nudges}")

Reporting and Interpretation

The print_report function in tests/eval/report.py generates human-readable summaries incorporating these reliability metrics alongside completion and timing data. High tool-calling reliability manifests as low averages across all nudge and error counters, while elevated avg_wasted_calls specifically indicates inefficient or redundant tool selection patterns that require workflow optimization.

Summary

• The Forge harness measures tool-calling reliability by analyzing Message history stored during scenario execution when config.keep_message_history is enabled.
• The analyze_history function in tests/eval/metrics.py categorizes messages by MessageType to count retry nudges, step nudges, and tool errors containing [ToolError].
• Aggregated metrics like avg_retry_nudges and avg_wasted_calls provide quantitative reliability scores across multiple scenario runs via compute_metrics.
• Retry nudges signal transient failures requiring automatic recovery, while step nudges indicate violations of required workflow step ordering.
• The reporting pipeline in tests/eval/report.py presents these statistics alongside completion and correctness metrics for comprehensive evaluation.

Frequently Asked Questions

What is a "retry nudge" in Forge's evaluation system?

A retry nudge is a MessageType.RETRY_NUDGE message emitted when the workflow runner automatically retries a tool call after a transient failure. The harness counts these in analyze_history to calculate avg_retry_nudges, which measures how often the model's initial tool calls fail and require recovery attempts, directly indicating tool-calling reliability issues.

How does Forge distinguish between successful and failed tool calls?

The system inspects MessageType.TOOL_RESULT messages for the presence of [ToolError] in the content string. Results containing this substring increment the tool_errors counter in HistoryStats, while successful calls lack this error marker. This detection occurs within the message processing loop in tests/eval/metrics.py.

What constitutes a "wasted call" in the reliability metrics?

Wasted calls represent tool invocations that exceed the optimal number required to complete a scenario. The compute_metrics function calculates avg_wasted_calls by comparing actual tool call counts against the ideal iteration count for the scenario, highlighting inefficient, redundant, or exploratory tool usage patterns that degrade performance.

Where can I configure message history collection?

Message history collection is controlled through the EvalConfig dataclass in tests/eval/eval_runner.py. Setting keep_message_history=True enables the callback mechanism that appends messages to the collection list, which is subsequently passed to analyze_history for reliability analysis.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}