Tracing and Observability Features in aisuite: A Complete Technical Guide

aisuite implements a full-stack tracing and observability system that records every step of an LLM run—from the high-level run lifecycle down to individual model and tool interactions—and renders the data in a searchable, filterable web UI.

This guide explores the tracing and observability features available in the andrewyng/aisuite repository, covering how the framework captures execution events, stores them persistently, and provides real-time visibility through a built-in viewer. The system is designed to work out-of-the-box while remaining modular enough to integrate with external observability platforms.

Core Architecture of the Tracing System

The observability stack in aisuite consists of four loosely-coupled layers: trace events, trace sinks, trace stores, and a viewer server. Each layer is implemented as a distinct Python module, allowing you to substitute components as needed.

Trace Events: Immutable Execution Records

At the foundation of the system lies the TraceEvent dataclass defined in aisuite/tracing/sinks.py. These are immutable records describing a single point in time, such as run.started, model.send, or tool.completed.

Every event carries:

• A trace_id (UUID) for correlation
• Timestamps for precise timing
• Optional span hierarchy for nested operations
• A free-form data payload containing request parameters, responses, and metadata

Trace Sinks: Pluggable Event Back-ends

Sinks are responsible for emitting events to their destinations. The framework ships with three built-in implementations in aisuite/tracing/sinks.py:

• LocalTraceSink: Writes events as JSON-Lines to a local file (default: .aisuite/events.jsonl)
• HttpTraceSink: POSTs events to a remote HTTP endpoint for external collection
• InMemoryTraceSink: Retains events in RAM, ideal for unit testing

The TraceSink protocol allows you to define custom sinks by implementing the emit(event: TraceEvent) method.

Trace Stores: Persistent Storage and Reconstruction

Stores handle the append, list, and reconstruction operations. Located in aisuite/tracing/store.py, two concrete implementations are provided:

• JsonlTraceStore: File-backed storage using JSON-Lines format
• InMemoryTraceStore: Volatile storage for testing and ephemeral runs

The reconstruct_runs() function merges raw events by trace_id, infers run status (running, completed, or failed), and calculates derived metrics including duration, message counts, and tool event tallies.

Viewer Server: Real-time Web Interface

The ViewerServer class in aisuite/tracing/viewer.py hosts a lightweight HTTP server that serves a single-page React-style UI. It exposes endpoints like /api/runs that return pre-processed summaries generated by prepare_viewer_run_summaries() and detailed views from prepare_viewer_runs().

The browser client polls the API every 1.5 seconds to display:

• Timeline visualizations of event sequences
• Tool argument and result previews
• Approval/denial decision logs
• Latency statistics and artifact references

How Tracing Works in Practice

Understanding the data flow helps you optimize observability for production workloads.

1. Context Creation: When an agent starts, aisuite/agents/context.py generates a UUID trace_id and attaches the list of active trace_sinks to the runtime context. This context propagates to all tools and sub-agents automatically.

2. Event Emission: Framework components call emit_event(sinks, event) whenever significant actions occur. For example, sending an LLM request creates a model.send event, while the response generates model.response. Tool interactions emit tool.allowed, tool.started, and tool.completed events.

3. Storage: Each sink writes to its downstream store. The default CLI configuration uses a LocalTraceSink writing to .aisuite/events.jsonl, while you can optionally add an HttpTraceSink to forward copies to external collectors like OpenTelemetry.

4. Run Reconstruction: When viewing history, JsonlTraceStore.list_runs() reads the raw JSONL, and reconstruct_runs() aggregates events by trace_id to produce normalized run dictionaries with computed fields.

5. Live Visualization: Calling start_viewer() spawns a server that reads the trace store, builds UI-ready structures, and serves the bundled static assets from viewer-ui/dist (or the repository directory during development).

Configuring Tracing and Observability in aisuite

The public ai.tracing namespace exposes all observability primitives. Here is how to use the key features in your applications.

Emitting Custom Trace Events

Create and emit custom events to track application-specific logic:

from aisuite.tracing import TraceEvent, emit_event, get_configured_sinks

# Create a custom event

event = TraceEvent(
    event_type="custom.validation",
    trace_id="trace_123",
    agent_name="my_agent",
    run_name="demo_run",
)

# Emit to all configured sinks

emit_event(get_configured_sinks(), event)

Starting the Local Viewer

Launch the web UI to inspect runs in real-time:

from aisuite.tracing import start_viewer

viewer = start_viewer(trace_file="my_trace.jsonl", port=8765)
print("Open your browser at:", viewer.url)  # http://127.0.0.1:8765

The viewer automatically refreshes every 1.5 seconds and displays timeline events, tool panels, and latency charts without external dependencies.

Reading Trace Files Programmatically

Process historical traces for testing or reporting:

from aisuite.tracing import read_trace_file

runs = read_trace_file("my_trace.jsonl")
print("Found runs:", len(runs))
print("First run status:", runs[0]["status"])
print("Duration:", runs[0].get("duration_ms"), "ms")

Forwarding Events to External Collectors

Configure an HTTP sink to integrate with enterprise observability platforms:

from aisuite.tracing import HttpTraceSink, configure

http_sink = HttpTraceSink(
    "http://otel-collector.example.com/v1/traces",
    fail_silently=False
)

configure(http_sink)  # All subsequent emit_event() calls forward to the endpoint

Summary

• aisuite/tracing/sinks.py defines TraceEvent and three built-in sinks (LocalTraceSink, HttpTraceSink, InMemoryTraceSink) with the emit_event() helper.
• aisuite/tracing/store.py implements JsonlTraceStore and InMemoryTraceStore, plus the reconstruct_runs() logic that aggregates raw events into queryable run summaries.
• aisuite/tracing/viewer.py hosts the ViewerServer and preparation functions that transform stored events into the web UI format.
• The aisuite/agents/context.py module automatically provisions trace_id and trace_sinks to all agent executions, ensuring zero-instrumentation observability.
• Use start_viewer() to launch the local UI, read_trace_file() for programmatic access, and configure() with HttpTraceSink for external forwarding.

Frequently Asked Questions

How do I enable tracing in an existing aisuite application?

Tracing is enabled by default when using the standard CLI or agent constructors. The framework automatically creates a LocalTraceSink writing to .aisuite/events.jsonl. You can customize sinks by calling configure() with specific sink instances before starting your agents, or by setting the trace_sinks parameter in the agent context.

What is the difference between a trace sink and a trace store?

A trace sink is an active emitter that receives events as they happen and forwards them to destinations (files, HTTP endpoints, or memory). A trace store is a persistent storage abstraction that supports appending, listing, and reconstructing historical runs. The LocalTraceSink uses a JsonlTraceStore internally, while HttpTraceSink sends data to remote collectors without local persistence.

Can I use aisuite tracing with OpenTelemetry or Datadog?

Yes. Implement the TraceSink protocol to create a custom sink that formats TraceEvent objects into OpenTelemetry spans or Datadog logs, then pass it to configure(). Alternatively, use HttpTraceSink to POST events to a proxy service that translates aisuite's JSON format into your observability platform's native protocol.

Where are the web UI static assets located?

The viewer serves static assets from the viewer-ui/dist directory within the repository, or from bundled package data when installed via pip. The ViewerServer in aisuite/tracing/viewer.py handles routing, API endpoints, and static file serving automatically when you call start_viewer().

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}