How no-mistakes Collects and Tracks Agent Invocation Metrics for Fidelity

no-mistakes captures agent invocation metrics through a three-stage pipeline where the Codex adapter extracts telemetry from streaming JSON, the generic agent wrapper attaches it to step results, and the persistence layer stores it in SQLite for fidelity testing and CLI reporting.

The no-mistakes repository implements a comprehensive telemetry system to track every interaction with LLM-based agents. By recording granular agent invocation metrics—including token counts, tool calls, and model latency—the system enables reproducible fidelity testing and real-time performance monitoring through commands like axi status.

The InvocationMetrics Data Model

At the core of the telemetry system lies the InvocationMetrics struct defined in internal/agent/invocationmetrics.go. This compact data structure captures the essential operational characteristics of each agent turn without exposing sensitive prompt contents.

Core Fields and Definitions

The struct aggregates five key measurements:

type InvocationMetrics struct {
    // Number of tool calls made during the turn.
    ToolCalls int64 `json:"toolCalls,omitempty"`

    // Tokens fed to the model (prompt + system messages).
    InputTokens int64 `json:"inputTokens,omitempty"`

    // Tokens generated by the model (completion).
    OutputTokens int64 `json:"outputTokens,omitempty"`

    // Total tokens (input + output). Helpful for cost‑estimation.
    TotalTokens int64 `json:"totalTokens,omitempty"`

    // Wall‑clock time the model spent (ms). Used for latency checks.
    ModelTimeMS int64 `json:"modelTimeMs,omitempty"`
}

These fields provide the quantitative foundation for cost estimation, latency monitoring, and behavioral analysis across agent runs.

Metric Extraction from Agent Output

The system extracts raw telemetry by parsing the streaming JSON events emitted by underlying LLM executors. The Codex adapter serves as the primary implementation for this extraction phase.

Parsing Codex JSON Streams

In internal/agent/codex_metrics.go, the codexMetricsAccumulator consumes the --json output stream from codex exec commands. As events arrive, the accumulator updates internal counters for tool invocations and token consumption.

Once the stream completes, the accumulator produces a fully populated metrics instance:

func (m *codexMetricsAccumulator) metrics() InvocationMetrics {
    return InvocationMetrics{
        ToolCalls:    m.toolCalls,
        InputTokens:  m.inputTokens,
        OutputTokens: m.outputTokens,
        TotalTokens:  m.inputTokens + m.outputTokens,
        ModelTimeMS:  m.modelTimeMs,
    }
}

This design ensures that metric calculation happens at the source, before data propagates through the rest of the pipeline.

Propagation Through the Agent Pipeline

After extraction, metrics flow through the generic agent wrapper to maintain separation between adapter-specific logic and core orchestration.

Attaching Metrics to StepResult

The agent runner in internal/agent/agent.go receives the *InvocationMetrics pointer from the underlying adapter. Upon completing a turn, it embeds these metrics into the StepResult structure:

result := StepResult{
    // … other fields …
    Metrics: metricsPtr, // *InvocationMetrics or nil
}

If an adapter does not support metric collection, the pointer remains nil, signaling "no data" to downstream consumers. This optional attachment pattern allows the system to gracefully handle adapters that lack telemetry capabilities without breaking the execution flow.

Persistence and Storage

Long-term retention of agent invocation metrics occurs in a local SQLite database, enabling offline analysis and historical comparison across runs.

SQLite Schema and Insertion

The database schema, defined in internal/db/schema.go, creates the agent_invocation table to store per-step telemetry. The persistence logic in internal/db/agent_invocation.go handles the actual insertion:

_, err := db.Exec(`
    INSERT INTO agent_invocation (
        run_id, step, tool_calls, input_tokens,
        output_tokens, total_tokens, model_time_ms
    ) VALUES (?, ?, ?, ?, ?, ?, ?)
`, runID, stepIdx,
    result.Metrics.ToolCalls,
    result.Metrics.InputTokens,
    result.Metrics.OutputTokens,
    result.Metrics.TotalTokens,
    result.Metrics.ModelTimeMS)

Each row correlates to a specific step within a run, maintaining referential integrity through foreign keys while preserving the quantitative agent invocation metrics needed for analysis.

Consumption by Fidelity Tests and UI

The persisted metrics serve dual purposes: validating reproducibility through automated testing and surfacing operational insights to end users.

Fidelity Instrumentation Tests

The internal/pipeline/instrument_fidelity_test.go file contains assertions that verify metric consistency across runs. These tests query the database to ensure that token counts, tool call frequencies, and latency measurements remain within expected bounds when replaying identical prompts. This guarantees that agent behavior is measurable and deterministic, catching regressions in underlying models or infrastructure changes.

CLI Telemetry and Status Reporting

End users interact with these metrics through the axi command-line interface. The internal/cli/axi_telemetry_test.go validates that commands like axi status and axi stats correctly aggregate and display agent invocation metrics from the database. This provides immediate visibility into resource consumption and performance characteristics without requiring direct database access.

Practical Implementation Examples

When building custom steps or analysis tools, developers can access these metrics programmatically.

Accessing metrics during custom step execution:

func myCustomStep(ctx context.Context, run *db.Run) error {
    // Run the agent (Codex) and get the StepResult.
    stepResult, err := agent.Run(ctx, prompt)
    if err != nil {
        return err
    }

    // Metrics are optional – check for nil.
    if stepResult.Metrics != nil {
        log.Printf("Tool calls: %d, tokens: %d (in) + %d (out), model time: %d ms",
            stepResult.Metrics.ToolCalls,
            stepResult.Metrics.InputTokens,
            stepResult.Metrics.OutputTokens,
            stepResult.Metrics.ModelTimeMS)
    }
    return nil
}

Querying historical metrics for analysis:

func printRunMetrics(db *sql.DB, runID int64) error {
    rows, err := db.Query(`
        SELECT step, tool_calls, input_tokens, output_tokens,
               total_tokens, model_time_ms
        FROM agent_invocation
        WHERE run_id = ?
        ORDER BY step
    `, runID)
    if err != nil {
        return err
    }
    defer rows.Close()

    for rows.Next() {
        var step, toolCalls, inputTokens, outputTokens, totalTokens, modelTime int64
        if err := rows.Scan(&step, &toolCalls, &inputTokens,
            &outputTokens, &totalTokens, &modelTime); err != nil {
            return err
        }
        fmt.Printf("step %d – tools:%d tokens:%d/%d total:%d time:%d ms\n",
            step, toolCalls, inputTokens, outputTokens, totalTokens, modelTime)
    }
    return rows.Err()
}

Summary

• no-mistakes defines agent invocation metrics in internal/agent/invocationmetrics.go as a compact struct tracking tool calls, input/output tokens, total tokens, and model latency.
• The Codex adapter extracts these metrics from streaming JSON output in internal/agent/codex_metrics.go, calculating totals before propagating upstream.
• The generic agent wrapper in internal/agent/agent.go attaches metrics to StepResult, treating them as optional pointers to maintain compatibility with non-instrumented adapters.
• SQLite persistence in internal/db/agent_invocation.go stores metrics per step, enabling historical analysis and cross-run comparisons.
• Fidelity tests (internal/pipeline/instrument_fidelity_test.go) verify metric consistency to ensure reproducible agent behavior, while the CLI surfaces these values through axi status and related commands.

Frequently Asked Questions

What specific fields are captured in agent invocation metrics?

The InvocationMetrics struct captures five quantitative fields: ToolCalls (count of function invocations), InputTokens (prompt size), OutputTokens (completion size), TotalTokens (sum of input and output), and ModelTimeMS (wall-clock latency in milliseconds). These values provide sufficient granularity for cost estimation and performance monitoring without exposing sensitive prompt data.

How does the Codex adapter extract metrics from streaming output?

The adapter in internal/agent/codex_metrics.go implements a codexMetricsAccumulator that parses the --json event stream emitted by codex exec commands. It increments counters for tool calls and tokens as events arrive, then calculates the final TotalTokens and ModelTimeMS values before returning a populated InvocationMetrics struct.

Where are agent invocation metrics stored long-term?

Metrics persist to the agent_invocation table in a local SQLite database, as defined in internal/db/schema.go and implemented in internal/db/agent_invocation.go. Each row records the metrics for a specific step within a run, linked by run_id and step indices to maintain referential integrity with the broader execution context.

How does no-mistakes use these metrics for fidelity testing?

The fidelity instrumentation tests in internal/pipeline/instrument_fidelity_test.go query the agent_invocation table to verify that recorded metrics match expected ranges across identical runs. This validation ensures that agent behavior remains deterministic and that changes to underlying models or infrastructure do not introduce unexpected variations in token usage, latency, or tool invocation patterns.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}