# How to Set Up Observability and Tracing for Agent-Native Applications: A Complete Guide

> Learn how to set up observability and tracing for Agent-Native applications. Discover automatic run capture, quality evaluation, and metrics surfacing without custom code. Complete guide for BuilderIO Agent-Native.

- Repository: [Builder.io/agent-native](https://github.com/BuilderIO/agent-native)
- Tags: how-to-guide
- Published: 2026-07-02

---

**Agent-Native provides built-in observability and tracing that automatically captures every agent run, evaluates quality with five automated scorers, and surfaces metrics through a React dashboard without requiring custom instrumentation code.**

Setting up observability and tracing for Agent-Native applications requires zero boilerplate because the framework ships with a full-stack telemetry layer. Every request passing through the production agent in [`packages/core/src/agent/production-agent.ts`](https://github.com/BuilderIO/agent-native/blob/main/packages/core/src/agent/production-agent.ts) is automatically instrumented to record token usage, latency, tool calls, and cost-derived metrics directly to your SQL database.

## Core Architecture

The observability pipeline consists of five integrated components that handle capture, storage, evaluation, and visualization according to the BuilderIO/agent-native source code. Each component is implemented as a first-class module in the Agent-Native core package.

**Production Agent** ([`packages/core/src/agent/production-agent.ts`](https://github.com/BuilderIO/agent-native/blob/main/packages/core/src/agent/production-agent.ts)) serves as the entry point for every chat request, injecting instrumentation hooks that log token consumption, model latency, and tool execution results. The implementation writes trace data directly to the database tables without requiring external services or manual span creation.

**Observer** ([`packages/core/src/agent/observational-memory/observer.ts`](https://github.com/BuilderIO/agent-native/blob/main/packages/core/src/agent/observational-memory/observer.ts)) manages context window compression by triggering an internal agent call when unobserved token counts exceed the configurable `observationTokenThreshold`. It stores the resulting summary in the `agent_observational_memory` table with `tier: "observation"` to preserve audit trails while freeing context space for future turns.

**Dashboard UI** (`@agent-native/core/client` → `ObservabilityDashboard`) provides a React component that queries auto-mounted API routes at `/_agent-native/observability/*` and renders five scoped tabs: Overview, Conversations, Evals, Experiments, and Feedback. All data displayed is automatically scoped to the signed-in user.

**API Endpoints** (`/_agent-native/observability/*`) are thin Nitro routes that read and write trace tables including `agent_trace_summaries`, `agent_trace_spans`, and `agent_feedback`, while accepting experiment definitions and feedback submissions. These routes provide the backend interface for the dashboard UI.

**Configuration Store** (`observability-config` setting) persists JSON settings via `putSetting` that control the master switch, prompt capture toggles, LLM-as-judge sampling rates, and OTLP exporter endpoints. Changes take effect immediately without requiring application restarts.

## Automatic Capture: What Gets Recorded

When a user sends a message, the production agent records comprehensive telemetry automatically. The instrumentation lives entirely within [`packages/core/src/agent/production-agent.ts`](https://github.com/BuilderIO/agent-native/blob/main/packages/core/src/agent/production-agent.ts) and requires no developer intervention.

The framework captures five specific categories:

- **Token Usage**: Input, output, and cache read/write counts
- **Cost**: Derived from per-token pricing for the specific model
- **Latency**: Total runtime and per-tool execution timing
- **Tool Calls**: Name, success/failure status, and duration
- **Automated Evals**: Five deterministic scorers run after every execution measuring tool success rate, step efficiency, latency score, cost efficiency, and error recovery

This automatic evaluation runs after every agent execution to provide immediate quality signals for monitoring and alerting.

## Setting Up the Observability Dashboard

Add the built-in dashboard to any Agent-Native application by creating a single route file. Import the `ObservabilityDashboard` component from `@agent-native/core/client` and render it within your page layout:

```tsx
// app/routes/observability.tsx
import { ObservabilityDashboard } from "@agent-native/core/client";

export default function ObservabilityPage() {
  return (
    <div className="min-h-screen p-6">
      <ObservabilityDashboard />
    </div>
  );
}

```

*Source reference:* [[`templates/design/app/routes/observability.tsx`](https://github.com/BuilderIO/agent-native/blob/main/templates/design/app/routes/observability.tsx)](https://github.com/BuilderIO/agent-native/blob/main/templates/design/app/routes/observability.tsx)

The component automatically fetches data from the API endpoints and displays:

- **Overview**: Aggregate metrics including total runs, average cost, latency percentiles, and satisfaction scores
- **Conversations**: Drill-down into individual trace entries with full span hierarchies (`agent.run` → `tool.call` → `llm.call`)
- **Evals**: Time-series charts of the five automated quality scores
- **Experiments**: A/B test management interface using consistent hashing to ensure users see the same variant across sessions
- **Feedback**: Thumbs-up/down ratings and a derived **frustration index** based on retry patterns, abandonment rates, sentiment analysis, and message length trends

## Exporting Traces to External Backends

To forward traces to external observability platforms like Langfuse, Datadog, or Grafana, enable OpenTelemetry OTLP exporters through the configuration store. First install `@opentelemetry/api`; the framework degrades gracefully to a no-op if the package is absent.

Configure the exporter by updating the `observability-config` setting:

```ts
import { putSetting } from "@agent-native/core/settings";

await putSetting("observability-config", {
  enabled: true,
  exporters: [
    {
      type: "otlp",
      endpoint: "https://cloud.langfuse.com/api/public/otel",
      headers: { Authorization: "Bearer sk-REPLACE-WITH-TOKEN" },
    },
  ],
});

```

Agent-Native emits **semantic convention** spans using the `gen_ai.*` attribute namespace that complies with the OpenTelemetry GenAI specification. When a registered `TracerProvider` exists, spans export automatically; otherwise they remain in-process with minimal overhead.

## Observational Memory and Cost Control

Long-running conversations risk exceeding LLM context limits. The observer implementation in [`packages/core/src/agent/observational-memory/observer.ts`](https://github.com/BuilderIO/agent-native/blob/main/packages/core/src/agent/observational-memory/observer.ts) automatically compacts thread history once the unobserved token count passes `observationTokenThreshold`.

The compaction process runs an internal, tool-less agent call that summarizes the message window into a dense observation entry. Original messages are marked as observed, preserving audit trails while freeing tokens for future turns and significantly reducing costs for long sessions.

For advanced use cases, trigger compaction manually:

```ts
import { runObserver } from "@agent-native/core/agent/observational-memory";

await runObserver({
  threadId: "my-thread",
  messages: threadMessages,
  ownerEmail: userEmail,
  orgId: orgId,
});

```

*Implementation reference:* [[`observer.ts`](https://github.com/BuilderIO/agent-native/blob/main/observer.ts)](https://github.com/BuilderIO/agent-native/blob/main/packages/core/src/agent/observational-memory/observer.ts)

## Configuring Experiments and LLM-as-Judge Evaluations

Beyond automatic deterministic scoring, you can enable probabilistic LLM-as-judge evaluation to augment the five automated scorers with nuanced quality assessments. Configure the sampling rate through the same `observability-config` setting:

```ts
await putSetting("observability-config", {
  enabled: true,
  evalSampleRate: 0.05, // Evaluate 5% of runs with a sampled LLM judge
});

```

Create A/B experiments by posting to the internal API. Experiment assignment uses consistent hashing on user identifiers to ensure deterministic variant selection across sessions:

```ts
// POST /_agent-native/observability/experiments
{
  "name": "model-a-vs-b",
  "variants": [
    { "id": "control", "weight": 50, "config": { "model": "gpt-4o-mini" } },
    { "id": "treatment", "weight": 50, "config": { "model": "gpt-4o" } }
  ],
  "metrics": ["cost", "latency", "satisfaction"]
}

```

These experiments track specified metrics across variants to quantify the impact of configuration changes.

## Summary

- **Agent-Native observability requires zero instrumentation code**; the production agent automatically captures traces, costs, and latency metrics to SQL tables.
- **Add the dashboard** by importing `ObservabilityDashboard` from `@agent-native/core/client` into a Nitro route file.
- **Configure via `observability-config`** to toggle capture, adjust LLM-as-judge sampling rates, and define OTLP exporters for external platforms.
- **Context windows are automatically managed** by the observer in [`observer.ts`](https://github.com/BuilderIO/agent-native/blob/main/observer.ts), which compacts long threads into cost-efficient observation entries stored in `agent_observational_memory`.
- **A/B experiments and feedback** are handled through auto-mounted API routes with consistent hashing for deterministic variant assignment.

## Frequently Asked Questions

### How do I disable observability in Agent-Native?

Set the master switch to false in your configuration. Call `await putSetting("observability-config", { enabled: false })` and the production agent in [`packages/core/src/agent/production-agent.ts`](https://github.com/BuilderIO/agent-native/blob/main/packages/core/src/agent/production-agent.ts) will skip writing to trace tables. Existing data remains available for the dashboard, but new runs will not be recorded until re-enabled.

### Where are the trace records stored?

All telemetry persists in the application's own SQL database across four tables: `agent_trace_summaries` for high-level run data, `agent_trace_spans` for detailed timing, `agent_observational_memory` for compressed context windows, and `agent_feedback` for user ratings. This eliminates external dependencies and ensures data remains within your infrastructure by default.

### Can I use my existing OpenTelemetry setup?

Yes. Install `@opentelemetry/api` and configure an OTLP exporter in the `observability-config` setting. Agent-Native emits GenAI-compliant semantic spans (`gen_ai.*` attributes) that integrate with any OpenTelemetry-compatible backend including Datadog, Grafana, and Langfuse without requiring custom span processors or instrumentation code.

### What is the "frustration index" in the feedback tab?

The **frustration index** is a derived heuristic calculated from user behavior patterns including re-phrasing frequency, retry counts, session abandonment, sentiment analysis of messages, and message length trends. It appears alongside explicit thumbs-up/down feedback to surface potentially problematic agent interactions without requiring manual user input, helping identify conversations where users struggle despite not clicking negative feedback buttons.