How to Monitor AI Agent Performance Over Time: A Complete Observability Pipeline

You monitor AI agent performance over time by capturing execution trajectories in SQLite, extracting behavioral metrics with behavior_metrics(), and aggregating results into version-controlled audit reports that track latency, throughput, and error rates against historical benchmarks.

The bojieli/ai-agent-book repository provides a production-ready framework for tracking how AI agents behave across thousands of runs. Instead of relying on scattered logs, this system persists every tool call, retry decision, and circuit-breaker state as structured evidence, enabling you to detect regressions, validate fixes, and maintain performance SLAs as your agent evolves.

The Three-Layer Observability Architecture

The monitoring system implemented in bojieli/ai-agent-book separates concerns into distinct layers: raw data collection, metric extraction, and audit reporting. This separation allows you to diagnose failures at the trajectory level while tracking high-level trends across campaign runs.

Data Collection with Trajectories

Every agent execution generates a trajectory—a JSON record capturing the complete sequence of LLM calls, tool invocations, and state transitions. The diagnose routine in chapter8/self-modifying-agent/evolution.py (lines 60-84) walks these raw trajectories to extract failure patterns and retry behaviors.

These trajectories persist in a SQLite evidence database (evidence.sqlite) with a schema designed for time-series analysis. The structure captures not just successful completions, but transient errors, timeout recoveries, and circuit-breaker triggers that indicate systemic resilience issues.

Metric Extraction with behavior_metrics

Raw trajectories contain too much noise for trend analysis. The behavior_metrics(candidate_source, trajectories) function (defined in chapter8/self-modifying-agent/evolution.py, lines 75-94) distills trajectory collections into comparable numerical signatures.

This helper returns a dictionary containing:

• mean_nonretryable_calls: Average fatal errors per run that bypass retry logic
• temporary_error_recovery_rate: Success rate of automatic recovery from transient failures
• old_task_regressions: Count of previously solved tasks that now fail (critical for detecting degradation)

You invoke this function against specific candidate versions to compare behavioral deltas before promoting code changes.

Audit and Reporting

The completion_audit function in chapter6/model-benchmark/analysis.py (lines 40-58) merges per-run metrics into campaign-wide health checklists. This audit validates design compliance, availability percentages, and rate-limit health.

Crucially, the audit performs external benchmark comparison via external_benchmark_comparison (lines 86-113). It compares current latency and throughput measurements against dated reference benchmarks, surfacing drift through structured delta calculations. The system flags missing external references (external_monitoring_reference_compared) when your comparison data becomes stale.

Implementing the Monitoring Pipeline

To monitor AI agent performance over time, wire these components into a repeatable workflow:

1. Execute experiment runs using run_experiment_8_5.py, which populates evidence.sqlite with trajectory data
2. Generate candidate metrics by feeding trajectories through behavior_metrics() to compute failure patterns
3. Run campaign analysis using analysis.py to produce JSON and Markdown reports containing workload summaries
4. Version control reports by committing campaign_report.md files to create a queryable performance timeline

This pipeline creates an immutable record of how each agent version performed against standard workloads, enabling git diff style comparisons between deployment candidates.

Practical Code Examples

Capturing Execution Evidence

Run the experiment driver to populate the evidence database:

python chapter8/self-modifying-agent/run_experiment_8_5.py \
    --output evidence.sqlite

This command writes trajectory data to SQLite, capturing every tool call and retry decision for subsequent analysis.

Computing Behavioral Metrics

Extract quantitative signatures from candidate code versions:

from chapter8.self_modifying_agent.evolution import behavior_metrics
import sqlite3
import json

# Load the candidate source (e.g., modified retry policy)

with open("candidate/retry_policy.py") as f:
    candidate_src = f.read()

# Load trajectories from SQLite evidence

conn = sqlite3.connect("evidence.sqlite")
conn.row_factory = sqlite3.Row
rows = conn.execute("SELECT payload FROM trajectories").fetchall()
trajectories = [json.loads(r["payload"]) for r in rows]

metrics = behavior_metrics(candidate_src, trajectories)
print(metrics)

# → {'mean_nonretryable_calls': 1.0,

#    'temporary_error_recovery_rate': 1.0,

#    'old_task_regressions': 0}

This pattern isolates the behavioral impact of specific code changes before they reach production.

Running the Full Audit

Generate comprehensive performance reports with external benchmark comparisons:

python chapter6/model-benchmark/analysis.py \
    --db evidence.sqlite \
    --config config.yaml \
    --campaign-id experiment-8-5 \
    --json report.json \
    --markdown report.md

The resulting report.md contains structured tables showing latency percentiles (TTFT p50/p95/p99), throughput metrics, and error rates across standard workloads.

Tracking Temporal Trends

Store campaign reports chronologically and visualize performance drift:

import pandas as pd
import json
import glob
import matplotlib.pyplot as plt

files = sorted(glob.glob("reports/**/*.json"))
records = []

for f in files:
    with open(f) as fh:
        data = json.load(fh)
    for w in data["workload"]:
        records.append({
            "date": f.split("/")[-2],
            "provider": w["provider"],
            "model": w["model"],
            "ttft_p50": w["ttft_s"]["p50"],
        })

df = pd.DataFrame(records)
df.groupby(["date", "model"])["ttft_p50"].mean().unstack().plot(marker="o")
plt.title("TTFT p50 Over Time")
plt.ylabel("seconds")
plt.show()

This approach transforms discrete campaign audits into continuous performance timelines.

Key Files and Their Roles

Path	Responsibility
chapter8/self-modifying-agent/evolution.py	Metric extraction – Implements behavior_metrics(), diagnose(), and release-gate validation logic
chapter6/model-benchmark/analysis.py	Campaign auditing – Contains completion_audit() and external_benchmark_comparison() for cross-referencing against historic baselines
chapter8/self-modifying-agent/run_experiment_8_5.py	Data collection – Orchestrates trajectory generation and SQLite persistence
chapter6/model-benchmark/campaign.yaml	Configuration – Defines workload matrices, rate-limit thresholds, and external benchmark references
chapter8/trajectory-verifier/validator.py	Validation – Verifies that candidate behavior adheres to expected retry and circuit-breaker policies
chapter8/browser-use-rpa/browser_use/watchdogs/	Runtime monitoring – Live watchdogs that detect crashes, network timeouts, and storage state anomalies during execution

Summary

• Persist trajectories in SQLite using the experiment runner to create queryable evidence of every agent decision
• Extract metrics with behavior_metrics() to quantify failure rates, recovery success, and regression counts per code version
• Audit campaigns via completion_audit() to validate availability, latency SLAs, and rate-limit health against external benchmarks
• Version control reports to maintain historical records that enable diff-based regression detection and trend visualization over time

Frequently Asked Questions

What specific metrics indicate AI agent health?

According to the bojieli/ai-agent-book implementation, you should track mean_nonretryable_calls (fatal errors), temporary_error_recovery_rate (resilience), and old_task_regressions (backward compatibility). Additionally, monitor latency percentiles (TTFT p50/p95/p99) and throughput tokens-per-second to detect infrastructure degradation.

How does the trajectory-based approach help with debugging?

Trajectories captured in evidence.sqlite contain complete execution paths including retry decisions and circuit-breaker states. The diagnose function in evolution.py analyzes these paths to identify failure patterns that aggregate metrics might miss, such as specific tool call sequences that trigger cascading timeouts.

Can I integrate this with external monitoring tools?

Yes. The external_benchmark_comparison function in chapter6/model-benchmark/analysis.py specifically supports ingesting third-party benchmark data via JSON URLs. The audit computes deltas between measured metrics and external references, allowing you to correlate internal agent performance against provider SLAs or industry standards.

How do I detect performance regressions over time?

Store each campaign's report.json in a date-based directory structure, then load the time-series data with pandas to plot trends. By comparing old_task_regressions counts and latency percentiles across commits, you can identify exactly when performance degradation began and correlate it with specific candidate code changes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}