# How to Debug Ax Applications and Trace Operations with OpenTelemetry

> Debug Ax applications and trace AI operations with OpenTelemetry. Configure the Node SDK to monitor prompts to responses and control payload visibility for effective troubleshooting.

- Repository: [Ax/ax](https://github.com/ax-llm/ax)
- Tags: how-to-guide
- Published: 2026-02-25

---

**Ax embeds OpenTelemetry throughout its core modules, allowing you to trace every AI operation from prompt generation to final response by configuring the Node SDK and setting `excludeContentFromTelemetry` to control payload visibility.**

Debugging AI applications requires visibility into complex, multi-step workflows. The ax-llm/ax repository provides native OpenTelemetry integration, making it straightforward to debug Ax applications and trace operations with OpenTelemetry across the flow engine, DSP layer, and AI provider calls.

## How Ax Implements OpenTelemetry Tracing

Ax instruments every layer of its architecture with OpenTelemetry, creating spans and metrics that give you end-to-end visibility. The implementation relies on the `@opentelemetry/api` types (`Tracer`, `Meter`, `Span`) and propagates context automatically through all operations.

### Flow Engine Instrumentation

In [`src/ax/flow/flow.ts`](https://github.com/ax-llm/ax/blob/main/src/ax/flow/flow.ts), the flow engine creates a span for every step of a flow, propagates context, and records metrics including counters, gauges, and histograms. Each node in your flow automatically generates telemetry data, allowing you to pinpoint bottlenecks in multi-step AI workflows.

### DSP Layer Telemetry

The dynamic-step-programming (DSP) layer in [`src/ax/dsp/generate.ts`](https://github.com/ax-llm/ax/blob/main/src/ax/dsp/generate.ts), [`src/ax/dsp/optimizer.ts`](https://github.com/ax-llm/ax/blob/main/src/ax/dsp/optimizer.ts), and [`src/ax/dsp/metrics.ts`](https://github.com/ax-llm/ax/blob/main/src/ax/dsp/metrics.ts) instruments generation, optimization, and learning phases. This emits telemetry events and updates metrics for latency, token usage, and error rates, giving you granular insight into AI model performance.

### AI Provider Wrappers

Ax wraps calls to LLM providers (OpenAI, Gemini, Anthropic, etc.) in [`src/ax/ai/base.ts`](https://github.com/ax-llm/ax/blob/main/src/ax/ai/base.ts) and [`src/ax/util/apicall.ts`](https://github.com/ax-llm/ax/blob/main/src/ax/util/apicall.ts). These wrappers start a span for each provider request and respect the `excludeContentFromTelemetry` flag. When enabled, this flag omits request and response payloads from span attributes, preventing sensitive data from reaching your telemetry collector.

### Database and Storage Tracing

Database interactions in [`src/ax/db/base.ts`](https://github.com/ax-llm/ax/blob/main/src/ax/db/base.ts) produce spans for vector store reads and writes, letting you see latency per query. This is crucial when debugging retrieval-augmented generation (RAG) pipelines where database performance directly impacts overall latency.

## Setting Up OpenTelemetry for Ax Applications

To start debugging Ax applications with OpenTelemetry, you need to configure the Node SDK and initialize Ax with telemetry enabled.

### Installing Dependencies

The Ax monorepo already lists OpenTelemetry packages as dependencies. Ensure your project includes:

```bash
npm install @opentelemetry/api @opentelemetry/sdk-node @opentelemetry/exporter-trace-otlp-http

```

### Configuring the Node SDK

Create a Node SDK instance that configures a trace exporter and metric exporter. This example sends traces to an OTLP collector:

```typescript
import { NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { Resource } from '@opentelemetry/resources';
import { SemanticResourceAttributes } from '@opentelemetry/semantic-conventions';

const sdk = new NodeSDK({
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: 'ax-demo',
  }),
  traceExporter: new OTLPTraceExporter({
    url: 'http://localhost:4318/v1/traces',
  }),
});

sdk.start();

```

### Enabling Ax Telemetry

When creating an Ax AI instance, set `excludeContentFromTelemetry` to control whether raw prompts and responses appear in your traces:

```typescript
import { ai } from '@ax-llm/ax';

const llm = ai({
  name: 'openai',
  apiKey: process.env.OPENAI_APIKEY!,
  excludeContentFromTelemetry: true, // Omits sensitive content from spans
});

```

## Complete Debugging Example

The official example in [`src/examples/telemetry.ts`](https://github.com/ax-llm/ax/blob/main/src/examples/telemetry.ts) demonstrates a minimal setup. Below is a complete workflow that wires OpenTelemetry with Ax, creates a flow, and executes it while sending telemetry to a collector:

```typescript
// 1️⃣ Import OpenTelemetry SDK pieces
import { trace } from '@opentelemetry/api';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { NodeSDK } from '@opentelemetry/sdk-node';
import { SimpleSpanProcessor } from '@opentelemetry/sdk-trace-base';
import { Resource } from '@opentelemetry/resources';
import { SemanticResourceAttributes } from '@opentelemetry/semantic-conventions';

// 2️⃣ Configure the OpenTelemetry SDK
const sdk = new NodeSDK({
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: 'ax-demo',
  }),
  traceExporter: new OTLPTraceExporter({
    url: 'http://localhost:4318/v1/traces',
  }),
});
sdk.start();

// 3️⃣ Create an Ax AI instance
import { ai } from '@ax-llm/ax';
const llm = ai({
  name: 'openai',
  apiKey: process.env.OPENAI_APIKEY!,
  excludeContentFromTelemetry: true,
});

// 4️⃣ Define a simple flow
import { s, ax } from '@ax-llm/ax';
const signature = s(`
  question:string "User question" ->
  answer:string "LLM answer"
`);
const generator = ax(`
  prompt:string "Prompt for LLM" ->
  response:string "LLM raw output"
`);

// 5️⃣ Execute with manual span for correlation
async function runDemo() {
  const rootSpan = trace.getTracer('ax-demo').startSpan('runDemo');
  try {
    const result = await generator.run({
      prompt: 'Explain OpenTelemetry in one sentence.',
    });
    console.log('Result →', result.response);
  } finally {
    rootSpan.end();
  }
}
runDemo().catch(console.error);

```

This example shows how `NodeSDK` bootstraps tracing, how `excludeContentFromTelemetry` protects sensitive data, and how Ax's `generator.run()` automatically creates child spans under your manual parent span.

## Adding Custom Spans and Attributes

To enrich traces with business context, add custom spans inside your flow steps using the OpenTelemetry API:

```typescript
import { type Span, trace } from '@opentelemetry/api';

async function myStep(input: string) {
  const span: Span = trace.getTracer('my-app').startSpan('myStep');
  try {
    span.setAttribute('my.input.length', input.length);
    const result = await someAsyncWork(input);
    span.setAttribute('my.result.success', result.ok);
    return result;
  } finally {
    span.end();
  }
}

```

Because Ax respects the OpenTelemetry Context propagation API implemented in [`src/ax/flow/flow.ts`](https://github.com/ax-llm/ax/blob/main/src/ax/flow/flow.ts), these custom spans automatically correlate with the built-in spans from the flow engine and AI wrappers.

## Summary

- Ax embeds OpenTelemetry natively across [`src/ax/flow/flow.ts`](https://github.com/ax-llm/ax/blob/main/src/ax/flow/flow.ts), [`src/ax/dsp/generate.ts`](https://github.com/ax-llm/ax/blob/main/src/ax/dsp/generate.ts), and [`src/ax/ai/base.ts`](https://github.com/ax-llm/ax/blob/main/src/ax/ai/base.ts), creating spans for every flow step, DSP operation, and LLM call.
- Configure the `NodeSDK` with an `OTLPTraceExporter` to send telemetry to Jaeger, Zipkin, or any OTLP collector.
- Use `excludeContentFromTelemetry` when creating AI instances to prevent sensitive prompt data from appearing in span attributes.
- Metrics are defined in [`src/ax/dsp/metrics.ts`](https://github.com/ax-llm/ax/blob/main/src/ax/dsp/metrics.ts) and [`src/ax/ai/metrics.ts`](https://github.com/ax-llm/ax/blob/main/src/ax/ai/metrics.ts), exposing counters for requests and errors plus histograms for latency.
- Add custom spans using the OpenTelemetry `trace` API to enrich traces with business-specific context that correlates automatically with Ax's built-in instrumentation.

## Frequently Asked Questions

### How do I view Ax traces in Jaeger?

Start Jaeger locally using Docker with `docker run -p 16686:16686 -p 4318:4318 jaegertracing/all-in-one:latest`, configure your `OTLPTraceExporter` to point to `http://localhost:4318/v1/traces`, and run your Ax application. Open `http://localhost:16686` to see the trace tree with spans from [`src/ax/flow/flow.ts`](https://github.com/ax-llm/ax/blob/main/src/ax/flow/flow.ts) and [`src/ax/ai/base.ts`](https://github.com/ax-llm/ax/blob/main/src/ax/ai/base.ts).

### What is the difference between `excludeContentFromTelemetry` and custom span attributes?

`excludeContentFromTelemetry` is a boolean flag set when creating an AI instance in [`src/ax/ai/base.ts`](https://github.com/ax-llm/ax/blob/main/src/ax/ai/base.ts) that prevents raw prompt and response payloads from being attached to spans automatically. Custom span attributes are key-value pairs you manually add to spans using `span.setAttribute()` to include business context such as user IDs or input lengths, which are always recorded regardless of the `excludeContentFromTelemetry` setting.

### Can I correlate Ax traces across multiple services?

Yes. Ax respects the OpenTelemetry Context propagation API implemented in [`src/ax/flow/flow.ts`](https://github.com/ax-llm/ax/blob/main/src/ax/flow/flow.ts). When you propagate the trace context via HTTP headers using `propagate.inject()` from `@opentelemetry/api` from an upstream service to your Ax microservice, Ax will continue the trace rather than starting a new one, allowing you to see the full request path across service boundaries.

### Where are metrics defined in the Ax source code?

Metrics are defined in [`src/ax/dsp/metrics.ts`](https://github.com/ax-llm/ax/blob/main/src/ax/dsp/metrics.ts) and [`src/ax/ai/metrics.ts`](https://github.com/ax-llm/ax/blob/main/src/ax/ai/metrics.ts). These files export counters for requests and errors, plus histograms for latency and token usage, which are updated during DSP operations in [`src/ax/dsp/generate.ts`](https://github.com/ax-llm/ax/blob/main/src/ax/dsp/generate.ts) and AI provider calls in [`src/ax/ai/base.ts`](https://github.com/ax-llm/ax/blob/main/src/ax/ai/base.ts).