How to Debug Ax Applications and Trace Operations with OpenTelemetry

Ax embeds OpenTelemetry throughout its core modules, allowing you to trace every AI operation from prompt generation to final response by configuring the Node SDK and setting excludeContentFromTelemetry to control payload visibility.

Debugging AI applications requires visibility into complex, multi-step workflows. The ax-llm/ax repository provides native OpenTelemetry integration, making it straightforward to debug Ax applications and trace operations with OpenTelemetry across the flow engine, DSP layer, and AI provider calls.

How Ax Implements OpenTelemetry Tracing

Ax instruments every layer of its architecture with OpenTelemetry, creating spans and metrics that give you end-to-end visibility. The implementation relies on the @opentelemetry/api types (Tracer, Meter, Span) and propagates context automatically through all operations.

Flow Engine Instrumentation

In src/ax/flow/flow.ts, the flow engine creates a span for every step of a flow, propagates context, and records metrics including counters, gauges, and histograms. Each node in your flow automatically generates telemetry data, allowing you to pinpoint bottlenecks in multi-step AI workflows.

DSP Layer Telemetry

The dynamic-step-programming (DSP) layer in src/ax/dsp/generate.ts, src/ax/dsp/optimizer.ts, and src/ax/dsp/metrics.ts instruments generation, optimization, and learning phases. This emits telemetry events and updates metrics for latency, token usage, and error rates, giving you granular insight into AI model performance.

AI Provider Wrappers

Ax wraps calls to LLM providers (OpenAI, Gemini, Anthropic, etc.) in src/ax/ai/base.ts and src/ax/util/apicall.ts. These wrappers start a span for each provider request and respect the excludeContentFromTelemetry flag. When enabled, this flag omits request and response payloads from span attributes, preventing sensitive data from reaching your telemetry collector.

Database and Storage Tracing

Database interactions in src/ax/db/base.ts produce spans for vector store reads and writes, letting you see latency per query. This is crucial when debugging retrieval-augmented generation (RAG) pipelines where database performance directly impacts overall latency.

Setting Up OpenTelemetry for Ax Applications

To start debugging Ax applications with OpenTelemetry, you need to configure the Node SDK and initialize Ax with telemetry enabled.

Installing Dependencies

The Ax monorepo already lists OpenTelemetry packages as dependencies. Ensure your project includes:

npm install @opentelemetry/api @opentelemetry/sdk-node @opentelemetry/exporter-trace-otlp-http

Configuring the Node SDK

Create a Node SDK instance that configures a trace exporter and metric exporter. This example sends traces to an OTLP collector:

import { NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { Resource } from '@opentelemetry/resources';
import { SemanticResourceAttributes } from '@opentelemetry/semantic-conventions';

const sdk = new NodeSDK({
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: 'ax-demo',
  }),
  traceExporter: new OTLPTraceExporter({
    url: 'http://localhost:4318/v1/traces',
  }),
});

sdk.start();

Enabling Ax Telemetry

When creating an Ax AI instance, set excludeContentFromTelemetry to control whether raw prompts and responses appear in your traces:

import { ai } from '@ax-llm/ax';

const llm = ai({
  name: 'openai',
  apiKey: process.env.OPENAI_APIKEY!,
  excludeContentFromTelemetry: true, // Omits sensitive content from spans
});

Complete Debugging Example

The official example in src/examples/telemetry.ts demonstrates a minimal setup. Below is a complete workflow that wires OpenTelemetry with Ax, creates a flow, and executes it while sending telemetry to a collector:

// 1️⃣ Import OpenTelemetry SDK pieces
import { trace } from '@opentelemetry/api';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { NodeSDK } from '@opentelemetry/sdk-node';
import { SimpleSpanProcessor } from '@opentelemetry/sdk-trace-base';
import { Resource } from '@opentelemetry/resources';
import { SemanticResourceAttributes } from '@opentelemetry/semantic-conventions';

// 2️⃣ Configure the OpenTelemetry SDK
const sdk = new NodeSDK({
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: 'ax-demo',
  }),
  traceExporter: new OTLPTraceExporter({
    url: 'http://localhost:4318/v1/traces',
  }),
});
sdk.start();

// 3️⃣ Create an Ax AI instance
import { ai } from '@ax-llm/ax';
const llm = ai({
  name: 'openai',
  apiKey: process.env.OPENAI_APIKEY!,
  excludeContentFromTelemetry: true,
});

// 4️⃣ Define a simple flow
import { s, ax } from '@ax-llm/ax';
const signature = s(`
  question:string "User question" ->
  answer:string "LLM answer"
`);
const generator = ax(`
  prompt:string "Prompt for LLM" ->
  response:string "LLM raw output"
`);

// 5️⃣ Execute with manual span for correlation
async function runDemo() {
  const rootSpan = trace.getTracer('ax-demo').startSpan('runDemo');
  try {
    const result = await generator.run({
      prompt: 'Explain OpenTelemetry in one sentence.',
    });
    console.log('Result →', result.response);
  } finally {
    rootSpan.end();
  }
}
runDemo().catch(console.error);

This example shows how NodeSDK bootstraps tracing, how excludeContentFromTelemetry protects sensitive data, and how Ax's generator.run() automatically creates child spans under your manual parent span.

Adding Custom Spans and Attributes

To enrich traces with business context, add custom spans inside your flow steps using the OpenTelemetry API:

import { type Span, trace } from '@opentelemetry/api';

async function myStep(input: string) {
  const span: Span = trace.getTracer('my-app').startSpan('myStep');
  try {
    span.setAttribute('my.input.length', input.length);
    const result = await someAsyncWork(input);
    span.setAttribute('my.result.success', result.ok);
    return result;
  } finally {
    span.end();
  }
}

Because Ax respects the OpenTelemetry Context propagation API implemented in src/ax/flow/flow.ts, these custom spans automatically correlate with the built-in spans from the flow engine and AI wrappers.

Summary

• Ax embeds OpenTelemetry natively across src/ax/flow/flow.ts, src/ax/dsp/generate.ts, and src/ax/ai/base.ts, creating spans for every flow step, DSP operation, and LLM call.
• Configure the NodeSDK with an OTLPTraceExporter to send telemetry to Jaeger, Zipkin, or any OTLP collector.
• Use excludeContentFromTelemetry when creating AI instances to prevent sensitive prompt data from appearing in span attributes.
• Metrics are defined in src/ax/dsp/metrics.ts and src/ax/ai/metrics.ts, exposing counters for requests and errors plus histograms for latency.
• Add custom spans using the OpenTelemetry trace API to enrich traces with business-specific context that correlates automatically with Ax's built-in instrumentation.

Frequently Asked Questions

How do I view Ax traces in Jaeger?

Start Jaeger locally using Docker with docker run -p 16686:16686 -p 4318:4318 jaegertracing/all-in-one:latest, configure your OTLPTraceExporter to point to http://localhost:4318/v1/traces, and run your Ax application. Open http://localhost:16686 to see the trace tree with spans from src/ax/flow/flow.ts and src/ax/ai/base.ts.

What is the difference between excludeContentFromTelemetry and custom span attributes?

excludeContentFromTelemetry is a boolean flag set when creating an AI instance in src/ax/ai/base.ts that prevents raw prompt and response payloads from being attached to spans automatically. Custom span attributes are key-value pairs you manually add to spans using span.setAttribute() to include business context such as user IDs or input lengths, which are always recorded regardless of the excludeContentFromTelemetry setting.

Can I correlate Ax traces across multiple services?

Yes. Ax respects the OpenTelemetry Context propagation API implemented in src/ax/flow/flow.ts. When you propagate the trace context via HTTP headers using propagate.inject() from @opentelemetry/api from an upstream service to your Ax microservice, Ax will continue the trace rather than starting a new one, allowing you to see the full request path across service boundaries.

Where are metrics defined in the Ax source code?

Metrics are defined in src/ax/dsp/metrics.ts and src/ax/ai/metrics.ts. These files export counters for requests and errors, plus histograms for latency and token usage, which are updated during DSP operations in src/ax/dsp/generate.ts and AI provider calls in src/ax/ai/base.ts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}