How to Set Up Observability and Tracing for Agent-Native Applications: A Complete Guide

Agent-Native provides built-in observability and tracing that automatically captures every agent run, evaluates quality with five automated scorers, and surfaces metrics through a React dashboard without requiring custom instrumentation code.

Setting up observability and tracing for Agent-Native applications requires zero boilerplate because the framework ships with a full-stack telemetry layer. Every request passing through the production agent in packages/core/src/agent/production-agent.ts is automatically instrumented to record token usage, latency, tool calls, and cost-derived metrics directly to your SQL database.

Core Architecture

The observability pipeline consists of five integrated components that handle capture, storage, evaluation, and visualization according to the BuilderIO/agent-native source code. Each component is implemented as a first-class module in the Agent-Native core package.

Production Agent (packages/core/src/agent/production-agent.ts) serves as the entry point for every chat request, injecting instrumentation hooks that log token consumption, model latency, and tool execution results. The implementation writes trace data directly to the database tables without requiring external services or manual span creation.

Observer (packages/core/src/agent/observational-memory/observer.ts) manages context window compression by triggering an internal agent call when unobserved token counts exceed the configurable observationTokenThreshold. It stores the resulting summary in the agent_observational_memory table with tier: "observation" to preserve audit trails while freeing context space for future turns.

Dashboard UI (@agent-native/core/client → ObservabilityDashboard) provides a React component that queries auto-mounted API routes at /_agent-native/observability/* and renders five scoped tabs: Overview, Conversations, Evals, Experiments, and Feedback. All data displayed is automatically scoped to the signed-in user.

API Endpoints (/_agent-native/observability/*) are thin Nitro routes that read and write trace tables including agent_trace_summaries, agent_trace_spans, and agent_feedback, while accepting experiment definitions and feedback submissions. These routes provide the backend interface for the dashboard UI.

Configuration Store (observability-config setting) persists JSON settings via putSetting that control the master switch, prompt capture toggles, LLM-as-judge sampling rates, and OTLP exporter endpoints. Changes take effect immediately without requiring application restarts.

Automatic Capture: What Gets Recorded

When a user sends a message, the production agent records comprehensive telemetry automatically. The instrumentation lives entirely within packages/core/src/agent/production-agent.ts and requires no developer intervention.

The framework captures five specific categories:

• Token Usage: Input, output, and cache read/write counts
• Cost: Derived from per-token pricing for the specific model
• Latency: Total runtime and per-tool execution timing
• Tool Calls: Name, success/failure status, and duration
• Automated Evals: Five deterministic scorers run after every execution measuring tool success rate, step efficiency, latency score, cost efficiency, and error recovery

This automatic evaluation runs after every agent execution to provide immediate quality signals for monitoring and alerting.

Setting Up the Observability Dashboard

Add the built-in dashboard to any Agent-Native application by creating a single route file. Import the ObservabilityDashboard component from @agent-native/core/client and render it within your page layout:

// app/routes/observability.tsx
import { ObservabilityDashboard } from "@agent-native/core/client";

export default function ObservabilityPage() {
  return (
    <div className="min-h-screen p-6">
      <ObservabilityDashboard />
    </div>
  );
}

Source reference: [templates/design/app/routes/observability.tsx](https://github.com/BuilderIO/agent-native/blob/main/templates/design/app/routes/observability.tsx)

The component automatically fetches data from the API endpoints and displays:

• Overview: Aggregate metrics including total runs, average cost, latency percentiles, and satisfaction scores
• Conversations: Drill-down into individual trace entries with full span hierarchies (agent.run → tool.call → llm.call)
• Evals: Time-series charts of the five automated quality scores
• Experiments: A/B test management interface using consistent hashing to ensure users see the same variant across sessions
• Feedback: Thumbs-up/down ratings and a derived frustration index based on retry patterns, abandonment rates, sentiment analysis, and message length trends

Exporting Traces to External Backends

To forward traces to external observability platforms like Langfuse, Datadog, or Grafana, enable OpenTelemetry OTLP exporters through the configuration store. First install @opentelemetry/api; the framework degrades gracefully to a no-op if the package is absent.

Configure the exporter by updating the observability-config setting:

import { putSetting } from "@agent-native/core/settings";

await putSetting("observability-config", {
  enabled: true,
  exporters: [
    {
      type: "otlp",
      endpoint: "https://cloud.langfuse.com/api/public/otel",
      headers: { Authorization: "Bearer sk-REPLACE-WITH-TOKEN" },
    },
  ],
});

Agent-Native emits semantic convention spans using the gen_ai.* attribute namespace that complies with the OpenTelemetry GenAI specification. When a registered TracerProvider exists, spans export automatically; otherwise they remain in-process with minimal overhead.

Observational Memory and Cost Control

Long-running conversations risk exceeding LLM context limits. The observer implementation in packages/core/src/agent/observational-memory/observer.ts automatically compacts thread history once the unobserved token count passes observationTokenThreshold.

The compaction process runs an internal, tool-less agent call that summarizes the message window into a dense observation entry. Original messages are marked as observed, preserving audit trails while freeing tokens for future turns and significantly reducing costs for long sessions.

For advanced use cases, trigger compaction manually:

import { runObserver } from "@agent-native/core/agent/observational-memory";

await runObserver({
  threadId: "my-thread",
  messages: threadMessages,
  ownerEmail: userEmail,
  orgId: orgId,
});

Implementation reference: [observer.ts](https://github.com/BuilderIO/agent-native/blob/main/packages/core/src/agent/observational-memory/observer.ts)

Configuring Experiments and LLM-as-Judge Evaluations

Beyond automatic deterministic scoring, you can enable probabilistic LLM-as-judge evaluation to augment the five automated scorers with nuanced quality assessments. Configure the sampling rate through the same observability-config setting:

await putSetting("observability-config", {
  enabled: true,
  evalSampleRate: 0.05, // Evaluate 5% of runs with a sampled LLM judge
});

Create A/B experiments by posting to the internal API. Experiment assignment uses consistent hashing on user identifiers to ensure deterministic variant selection across sessions:

// POST /_agent-native/observability/experiments
{
  "name": "model-a-vs-b",
  "variants": [
    { "id": "control", "weight": 50, "config": { "model": "gpt-4o-mini" } },
    { "id": "treatment", "weight": 50, "config": { "model": "gpt-4o" } }
  ],
  "metrics": ["cost", "latency", "satisfaction"]
}

These experiments track specified metrics across variants to quantify the impact of configuration changes.

Summary

• Agent-Native observability requires zero instrumentation code; the production agent automatically captures traces, costs, and latency metrics to SQL tables.
• Add the dashboard by importing ObservabilityDashboard from @agent-native/core/client into a Nitro route file.
• Configure via observability-config to toggle capture, adjust LLM-as-judge sampling rates, and define OTLP exporters for external platforms.
• Context windows are automatically managed by the observer in observer.ts, which compacts long threads into cost-efficient observation entries stored in agent_observational_memory.
• A/B experiments and feedback are handled through auto-mounted API routes with consistent hashing for deterministic variant assignment.

Frequently Asked Questions

How do I disable observability in Agent-Native?

Set the master switch to false in your configuration. Call await putSetting("observability-config", { enabled: false }) and the production agent in packages/core/src/agent/production-agent.ts will skip writing to trace tables. Existing data remains available for the dashboard, but new runs will not be recorded until re-enabled.

Where are the trace records stored?

All telemetry persists in the application's own SQL database across four tables: agent_trace_summaries for high-level run data, agent_trace_spans for detailed timing, agent_observational_memory for compressed context windows, and agent_feedback for user ratings. This eliminates external dependencies and ensures data remains within your infrastructure by default.

Can I use my existing OpenTelemetry setup?

Yes. Install @opentelemetry/api and configure an OTLP exporter in the observability-config setting. Agent-Native emits GenAI-compliant semantic spans (gen_ai.* attributes) that integrate with any OpenTelemetry-compatible backend including Datadog, Grafana, and Langfuse without requiring custom span processors or instrumentation code.

What is the "frustration index" in the feedback tab?

The frustration index is a derived heuristic calculated from user behavior patterns including re-phrasing frequency, retry counts, session abandonment, sentiment analysis of messages, and message length trends. It appears alongside explicit thumbs-up/down feedback to surface potentially problematic agent interactions without requiring manual user input, helping identify conversations where users struggle despite not clicking negative feedback buttons.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}