How Tracing Works in aisuite Agent Execution: Events, Stores, and Viewer

aisuite captures every step of an LLM-agent run as structured trace events, persists them via configurable sinks to JSONL files or memory, and provides a built-in web viewer to visualize model calls, tool executions, and errors in a searchable timeline.

Tracing in aisuite agent execution provides complete observability into how large language model agents interact with tools and models. The andrewyng/aisuite repository implements a three-stage pipeline—event generation, persistent storage, and interactive visualization—that records every significant operation from initial prompt to final output.

The Three-Component Tracing Architecture

The tracing system in aisuite consists of three integrated layers that handle event capture, storage, and display.

Trace Sinks (aisuite/tracing/sinks.py)

Trace sinks act as the collectors. They receive event dictionaries from the running agent and forward them to a persistence layer. The module provides three primary implementations:

• InMemoryTraceSink – Buffers events in Python memory for unit testing or temporary inspection.
• LocalTraceSink – Writes events to a local JSONL file via JsonlTraceStore.
• HttpTraceSink – POSTs events to a remote HTTP endpoint for centralized logging.

Each sink implements the emit(record) method, which receives a dictionary containing the event type and payload.

Trace Stores (aisuite/tracing/store.py)

Trace stores handle durability and retrieval. The two concrete implementations are:

• InMemoryTraceStore – Maintains records in a list[dict] with methods like list_runs() and get_run(trace_id).
• JsonlTraceStore – Appends each event as a single JSON line to a .jsonl file, providing read-only queries without loading the entire file into memory.

The store abstraction allows the viewer and analysis tools to query run metadata and event histories uniformly, regardless of the underlying storage medium.

The Viewer Server (aisuite/tracing/viewer.py)

The ViewerServer serves a minimal web interface that renders the stored runs. It exposes REST endpoints (/api/runs, /api/runs/<trace_id>) and static assets from viewer-ui/static/viewer. When you call start_viewer(), the server periodically refreshes the trace file (every 1.5 seconds) to show live updates during long-running agent executions.

Event Generation During Agent Execution

During execution, the agent emits discrete events that mark state transitions. Each event is a dictionary with an event_type string and a data payload containing context-specific details.

Critical event types include:

• Lifecycle: run.started, run.completed, run.failed
• Model interaction: model.send, model.response, model.error
• Tool governance: tool.allowed, tool.denied
• Tool execution: tool.started, tool.completed, tool.failed

The helpers in aisuite/tracing/normalize.py (e.g., _event_summary, _event_tone) transform these raw events into human-readable text for the UI, extracting tool names, argument previews, and error summaries.

Persisting Events with Sinks and Stores

The connection between sinks and stores is implemented in TraceStoreSink, found in aisuite/tracing/sinks.py:

class TraceStoreSink(TraceSink):
    def __init__(self, store: TraceStore):
        self.store = store

    def emit(self, record: dict[str, Any]) -> None:
        self.store.append_records([record])

When using JsonlTraceStore, each emit call appends a line to the file. The public API in aisuite/tracing/__init__.py provides read_trace_file() to load these events:

def read_trace_file(trace_file: str | Path) -> list[dict[str, Any]]:
    return prepare_viewer_runs(JsonlTraceStore(trace_file).list_runs())

This function returns UI-ready structures without requiring manual JSON parsing.

Transforming Raw Events into Display Summaries

Before rendering, raw events undergo transformation via prepare_viewer_runs (called by the viewer and read_trace_file). This pipeline:

1. Collects hierarchy metadata – Maps parent_run_id relationships and calculates child_count.
2. Extracts aggregates – Gathers tool names, approval statuses, error counts, and token usage statistics.
3. Generates display dictionaries – Creates a display object containing:
   • title and subtitle for the run header
   • status tone (success, error, warning)
   • timeline preview items

The _sanitize_for_viewer function ensures long strings are truncated to compact previews while preserving artifact references for downstream debugging.

Aggregating Activities for the Timeline

The viewer groups granular events into higher-level activities using _run_activities in aisuite/tracing/viewer.py. This aggregation logic identifies:

• Model calls – Pairs of model.send and model.response events merged into a single activity with duration_ms.
• Tool calls – Sequences of tool.allowed/denied, tool.started, and tool.completed/failed collapsed into one item showing approval status and execution result.
• Standalone events – Unpaired events like run.failed displayed individually.

Each activity object contains:

• duration_ms, tone, title, summary
• Optional tool_name, model, approval status
• Artifact references pointing to full input/output data

These activities populate the timeline array that drives the visual debugger.

Starting the Trace Viewer

To inspect traces interactively, use start_viewer from aisuite/tracing/__init__.py:

import ai.tracing as tracing
from pathlib import Path

viewer = tracing.start_viewer(trace_file=Path("run.jsonl"), port=0)
print(f"Open {viewer.url} to inspect the trace")

Setting port=0 allows the OS to assign a free port. The server serves:

• Static UI – HTML/CSS from viewer-ui/static/viewer or the packaged PACKAGE_UI_DIST.
• REST API – /api/runs lists all runs; /api/runs/<trace_id> returns full run details including the display object and raw events.

Practical Code Examples

Enable Tracing to a JSONL File

import ai
import ai.tracing as tracing
from pathlib import Path

trace_file = Path("agent_run.jsonl")
trace_sink = tracing.LocalTraceSink(trace_file)

client = ai.Client(trace_sink=trace_sink)
result = client.run(prompt="Generate a Python function to sort a list")

Launch the Local Web Viewer

import ai.tracing as tracing
from pathlib import Path

viewer = tracing.start_viewer(trace_file=Path("agent_run.jsonl"), port=0)

# Opens browser to http://127.0.0.1:8000/?embed=1

Query Trace Data Programmatically

import ai.tracing as tracing
from pathlib import Path

runs = tracing.read_trace_file(Path("agent_run.jsonl"))
for run in runs:
    print(f"{run['trace_id']}: {run['status']} - {run['display']['title']}")

Access Detailed Run Information

state = tracing.ViewerTraceState(trace_file=Path("agent_run.jsonl"))
detail = state.get_run("trace_abc123")
print(detail["final_output"])
print(detail["display"]["timeline"][0])  # First activity

Summary

• aisuite/agent execution tracing relies on three layers: sinks (aisuite/tracing/sinks.py) for collection, stores (aisuite/tracing/store.py) for persistence, and the viewer (aisuite/tracing/viewer.py) for visualization.
• Events capture every model call, tool approval, and execution step with structured payloads.
• LocalTraceSink combined with JsonlTraceStore provides durable, queryable logs without external dependencies.
• The start_viewer function launches a self-contained web server that aggregates raw events into a navigable timeline via prepare_viewer_runs and _run_activities.

Frequently Asked Questions

What event types does aisuite trace during agent execution?

aisuite emits lifecycle events (run.started, run.completed, run.failed), model interaction events (model.send, model.response, model.error), and comprehensive tool events (tool.allowed, tool.denied, tool.started, tool.completed, tool.failed). Each event contains an event_type and a data payload with arguments, results, or error details, as defined in the tracing normalization logic.

How do I configure tracing to write to a local JSONL file?

Instantiate tracing.LocalTraceSink with a Path object pointing to your desired .jsonl file, then pass that sink to the ai.Client constructor via the trace_sink parameter. The LocalTraceSink automatically uses JsonlTraceStore to append each event as a new JSON line, making the file safe for concurrent writes and easy to parse later.

Can I view traces without using the built-in web server?

Yes. Use tracing.read_trace_file() to load a JSONL trace into a list of run dictionaries, or instantiate tracing.ViewerTraceState directly to query specific runs. These APIs provide the same data structures used by the web viewer—including the display summary and timeline activities—allowing you to analyze agent behavior in Jupyter notebooks or custom dashboards.

What is the performance overhead of enabling tracing in aisuite?

Overhead is minimal for the JsonlTraceStore because events are appended incrementally to disk without locking, and the default sink buffers in memory before writing. InMemoryTraceSink adds no I/O overhead but consumes RAM proportional to the number of events. For high-throughput production environments, consider using HttpTraceSink to offload storage to a remote service, keeping the agent process unblocked.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}