# How to Monitor AI Agent Performance Over Time: A Complete Observability Pipeline

> Monitor AI agent performance over time using an observability pipeline. Capture trajectories, extract metrics, and generate version-controlled reports to track latency, throughput, and errors.

- Repository: [Bojie Li/ai-agent-book](https://github.com/bojieli/ai-agent-book)
- Tags: how-to-guide
- Published: 2026-07-30

---

**You monitor AI agent performance over time by capturing execution trajectories in SQLite, extracting behavioral metrics with `behavior_metrics()`, and aggregating results into version-controlled audit reports that track latency, throughput, and error rates against historical benchmarks.**

The `bojieli/ai-agent-book` repository provides a production-ready framework for tracking how AI agents behave across thousands of runs. Instead of relying on scattered logs, this system persists every tool call, retry decision, and circuit-breaker state as structured evidence, enabling you to detect regressions, validate fixes, and maintain performance SLAs as your agent evolves.

## The Three-Layer Observability Architecture

The monitoring system implemented in `bojieli/ai-agent-book` separates concerns into distinct layers: raw data collection, metric extraction, and audit reporting. This separation allows you to diagnose failures at the trajectory level while tracking high-level trends across campaign runs.

### Data Collection with Trajectories

Every agent execution generates a **trajectory**—a JSON record capturing the complete sequence of LLM calls, tool invocations, and state transitions. The `diagnose` routine in [`chapter8/self-modifying-agent/evolution.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter8/self-modifying-agent/evolution.py) (lines 60-84) walks these raw trajectories to extract failure patterns and retry behaviors.

These trajectories persist in a SQLite evidence database (`evidence.sqlite`) with a schema designed for time-series analysis. The structure captures not just successful completions, but transient errors, timeout recoveries, and circuit-breaker triggers that indicate systemic resilience issues.

### Metric Extraction with behavior_metrics

Raw trajectories contain too much noise for trend analysis. The `behavior_metrics(candidate_source, trajectories)` function (defined in [`chapter8/self-modifying-agent/evolution.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter8/self-modifying-agent/evolution.py), lines 75-94) distills trajectory collections into comparable numerical signatures.

This helper returns a dictionary containing:

- **mean_nonretryable_calls**: Average fatal errors per run that bypass retry logic
- **temporary_error_recovery_rate**: Success rate of automatic recovery from transient failures
- **old_task_regressions**: Count of previously solved tasks that now fail (critical for detecting degradation)

You invoke this function against specific candidate versions to compare behavioral deltas before promoting code changes.

### Audit and Reporting

The `completion_audit` function in [`chapter6/model-benchmark/analysis.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter6/model-benchmark/analysis.py) (lines 40-58) merges per-run metrics into campaign-wide health checklists. This audit validates design compliance, availability percentages, and rate-limit health.

Crucially, the audit performs **external benchmark comparison** via `external_benchmark_comparison` (lines 86-113). It compares current latency and throughput measurements against dated reference benchmarks, surfacing drift through structured delta calculations. The system flags missing external references (`external_monitoring_reference_compared`) when your comparison data becomes stale.

## Implementing the Monitoring Pipeline

To monitor AI agent performance over time, wire these components into a repeatable workflow:

1. **Execute experiment runs** using [`run_experiment_8_5.py`](https://github.com/bojieli/ai-agent-book/blob/main/run_experiment_8_5.py), which populates `evidence.sqlite` with trajectory data
2. **Generate candidate metrics** by feeding trajectories through `behavior_metrics()` to compute failure patterns
3. **Run campaign analysis** using [`analysis.py`](https://github.com/bojieli/ai-agent-book/blob/main/analysis.py) to produce JSON and Markdown reports containing workload summaries
4. **Version control reports** by committing [`campaign_report.md`](https://github.com/bojieli/ai-agent-book/blob/main/campaign_report.md) files to create a queryable performance timeline

This pipeline creates an immutable record of how each agent version performed against standard workloads, enabling `git diff` style comparisons between deployment candidates.

## Practical Code Examples

### Capturing Execution Evidence

Run the experiment driver to populate the evidence database:

```bash
python chapter8/self-modifying-agent/run_experiment_8_5.py \
    --output evidence.sqlite

```

This command writes trajectory data to SQLite, capturing every tool call and retry decision for subsequent analysis.

### Computing Behavioral Metrics

Extract quantitative signatures from candidate code versions:

```python
from chapter8.self_modifying_agent.evolution import behavior_metrics
import sqlite3
import json

# Load the candidate source (e.g., modified retry policy)

with open("candidate/retry_policy.py") as f:
    candidate_src = f.read()

# Load trajectories from SQLite evidence

conn = sqlite3.connect("evidence.sqlite")
conn.row_factory = sqlite3.Row
rows = conn.execute("SELECT payload FROM trajectories").fetchall()
trajectories = [json.loads(r["payload"]) for r in rows]

metrics = behavior_metrics(candidate_src, trajectories)
print(metrics)

# → {'mean_nonretryable_calls': 1.0,

#    'temporary_error_recovery_rate': 1.0,

#    'old_task_regressions': 0}

```

This pattern isolates the behavioral impact of specific code changes before they reach production.

### Running the Full Audit

Generate comprehensive performance reports with external benchmark comparisons:

```bash
python chapter6/model-benchmark/analysis.py \
    --db evidence.sqlite \
    --config config.yaml \
    --campaign-id experiment-8-5 \
    --json report.json \
    --markdown report.md

```

The resulting [`report.md`](https://github.com/bojieli/ai-agent-book/blob/main/report.md) contains structured tables showing latency percentiles (TTFT p50/p95/p99), throughput metrics, and error rates across standard workloads.

### Tracking Temporal Trends

Store campaign reports chronologically and visualize performance drift:

```python
import pandas as pd
import json
import glob
import matplotlib.pyplot as plt

files = sorted(glob.glob("reports/**/*.json"))
records = []

for f in files:
    with open(f) as fh:
        data = json.load(fh)
    for w in data["workload"]:
        records.append({
            "date": f.split("/")[-2],
            "provider": w["provider"],
            "model": w["model"],
            "ttft_p50": w["ttft_s"]["p50"],
        })

df = pd.DataFrame(records)
df.groupby(["date", "model"])["ttft_p50"].mean().unstack().plot(marker="o")
plt.title("TTFT p50 Over Time")
plt.ylabel("seconds")
plt.show()

```

This approach transforms discrete campaign audits into continuous performance timelines.

## Key Files and Their Roles

| Path | Responsibility |
|------|----------------|
| [`chapter8/self-modifying-agent/evolution.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter8/self-modifying-agent/evolution.py) | **Metric extraction** – Implements `behavior_metrics()`, `diagnose()`, and release-gate validation logic |
| [`chapter6/model-benchmark/analysis.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter6/model-benchmark/analysis.py) | **Campaign auditing** – Contains `completion_audit()` and `external_benchmark_comparison()` for cross-referencing against historic baselines |
| [`chapter8/self-modifying-agent/run_experiment_8_5.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter8/self-modifying-agent/run_experiment_8_5.py) | **Data collection** – Orchestrates trajectory generation and SQLite persistence |
| [`chapter6/model-benchmark/campaign.yaml`](https://github.com/bojieli/ai-agent-book/blob/main/chapter6/model-benchmark/campaign.yaml) | **Configuration** – Defines workload matrices, rate-limit thresholds, and external benchmark references |
| [`chapter8/trajectory-verifier/validator.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter8/trajectory-verifier/validator.py) | **Validation** – Verifies that candidate behavior adheres to expected retry and circuit-breaker policies |
| `chapter8/browser-use-rpa/browser_use/watchdogs/` | **Runtime monitoring** – Live watchdogs that detect crashes, network timeouts, and storage state anomalies during execution |

## Summary

- **Persist trajectories** in SQLite using the experiment runner to create queryable evidence of every agent decision
- **Extract metrics** with `behavior_metrics()` to quantify failure rates, recovery success, and regression counts per code version
- **Audit campaigns** via `completion_audit()` to validate availability, latency SLAs, and rate-limit health against external benchmarks
- **Version control reports** to maintain historical records that enable diff-based regression detection and trend visualization over time

## Frequently Asked Questions

### What specific metrics indicate AI agent health?

According to the `bojieli/ai-agent-book` implementation, you should track **mean_nonretryable_calls** (fatal errors), **temporary_error_recovery_rate** (resilience), and **old_task_regressions** (backward compatibility). Additionally, monitor latency percentiles (TTFT p50/p95/p99) and throughput tokens-per-second to detect infrastructure degradation.

### How does the trajectory-based approach help with debugging?

Trajectories captured in `evidence.sqlite` contain complete execution paths including retry decisions and circuit-breaker states. The `diagnose` function in [`evolution.py`](https://github.com/bojieli/ai-agent-book/blob/main/evolution.py) analyzes these paths to identify failure patterns that aggregate metrics might miss, such as specific tool call sequences that trigger cascading timeouts.

### Can I integrate this with external monitoring tools?

Yes. The `external_benchmark_comparison` function in [`chapter6/model-benchmark/analysis.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter6/model-benchmark/analysis.py) specifically supports ingesting third-party benchmark data via JSON URLs. The audit computes deltas between measured metrics and external references, allowing you to correlate internal agent performance against provider SLAs or industry standards.

### How do I detect performance regressions over time?

Store each campaign's [`report.json`](https://github.com/bojieli/ai-agent-book/blob/main/report.json) in a date-based directory structure, then load the time-series data with pandas to plot trends. By comparing **old_task_regressions** counts and latency percentiles across commits, you can identify exactly when performance degradation began and correlate it with specific candidate code changes.