Set Up X-Ray Tracing with the ADOT Collector: A Complete Guide for AWS Observability

To set up X-Ray tracing with the ADOT collector, configure the OpenTelemetry collector with the awsxray exporter and awsproxy extension, deploy it as a sidecar to receive OTLP traffic on ports 4317/4318, and set OTEL_TRACES_SAMPLER=xray in your applications to enable centralized sampling.

The AWS Distro for OpenTelemetry (ADOT) collector provides a modern, vendor-neutral path to AWS X-Ray tracing while simultaneously routing metrics to CloudWatch. According to the aws/agent-toolkit-for-aws repository, this approach replaces the legacy X-Ray daemon with a unified observability pipeline that processes standard OTLP telemetry from your services.

Architectural Overview: ADOT Collector vs. X-Ray Daemon

The ADOT collector runs as a sidecar—deployed as an EKS DaemonSet, ECS task, EC2 agent, or Lambda layer—and receives trace data in OTLP format from instrumented services. It forwards spans to the X-Ray backend while optionally emitting metrics to CloudWatch via the EMF exporter.

[Your Application] → OTLP → ADOT Collector → awsxray exporter → X-Ray
                                 ↘ awsemf exporter → CloudWatch Metrics

Critical deployment constraint: Both the ADOT collector and the legacy X-Ray daemon listen on port 2000. You must stop the X-Ray daemon before starting the collector to avoid port conflicts. For centralized sampling rules, the collector requires the awsproxy extension to query the X-Ray sampling service, as detailed in skills/core-skills/aws-observability/references/tracing.md.

Collector Configuration for X-Ray Tracing

The repository provides a production-ready configuration in skills/core-skills/aws-observability/assets/otel-config.yaml that ships traces to X-Ray and metrics to CloudWatch EMF.

Receivers and Processors

The OTLP receiver exposes gRPC on port 4317 and HTTP on port 4318 to accept telemetry from your services:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

The batch processor improves throughput by grouping spans before export:

processors:
  batch:
    timeout: 30s
    send_batch_size: 8192

Exporters (awsxray and awsemf)

Configure the awsxray exporter with your target region, and the awsemf exporter to namespace your CloudWatch metrics:

exporters:
  awsxray:
    region: us-east-1
  awsemf:
    namespace: MyApplication
    region: us-east-1

The awsproxy Extension for Centralized Sampling

To use centralized sampling rules from the X-Ray console, enable the awsproxy extension:

extensions:
  awsproxy:
    endpoint: 127.0.0.1:2000

This extension exposes the X-Ray sampling API locally so the collector can retrieve_sampling_rules from the backend service.

Pipeline Assembly

Wire the components together in the service section:

service:
  extensions: [awsproxy]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [awsxray]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [awsemf]

Version requirement: The collector must be version 0.34.0 or newer; older releases silently drop W3C-format trace IDs, breaking trace continuity.

Instrumenting Applications for ADOT

CDK Configuration for Lambda and API Gateway

Enable tracing in your CDK constructs to allow the collector to handle export:

import { aws_lambda as lambda, aws_apigateway as apigateway } from 'aws-cdk-lib';

// Lambda with active tracing
const fn = new lambda.Function(this, 'MyFunction', {
  runtime: lambda.Runtime.NODEJS_20_X,
  handler: 'index.handler',
  code: lambda.Code.fromAsset('lambda'),
  tracing: lambda.Tracing.ACTIVE,
});

// API Gateway tracing
const api = new apigateway.RestApi(this, 'MyApi', {
  deployOptions: { tracingEnabled: true },
});

To include the ADOT layer for Lambda:

layers: [
  lambda.LayerVersion.fromLayerVersionArn(
    stack,
    'AdotLayer',
    'arn:aws:lambda:us-east-1:556603146660:layer:AWSOTelNodeJS:4'
  )
]

OpenTelemetry SDK Environment Variables

For services using the OpenTelemetry SDK directly (Java, Python, Node.js), set these variables so the SDK communicates with the local collector:

export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318"
export OTEL_TRACES_SAMPLER="xray"
export OTEL_TRACES_SAMPLER_ARG="endpoint=http://localhost:2000"
export OTEL_PROPAGATORS="xray,tracecontext"

The OTEL_TRACES_SAMPLER=xray setting enables the SDK to respect centralized sampling rules retrieved via the awsproxy extension.

Deploying the ADOT Collector

EKS DaemonSet Deployment

Deploy the collector as a DaemonSet to ensure each node runs a collector instance:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: adot-collector
spec:
  template:
    spec:
      containers:
        - name: adot-collector
          image: public.ecr.aws/aws-observability/aws-otel-collector:latest
          resources:
            limits:
              memory: "200Mi"
              cpu: "250m"
          volumeMounts:
            - name: config
              mountPath: /etc/otel
          args: ["--config", "/etc/otel/otel-config.yaml"]
      volumes:
        - name: config
          configMap:
            name: adot-collector-config

ECS and Fargate Sidecar Configuration

For ECS or Fargate, add the ADOT container as a sidecar and mount the configuration via ConfigMap or Secrets Manager. The task execution role requires these permissions:

• xray:PutTraceSegments
• xray:PutTelemetryRecords
• cloudwatch:PutMetricData

Troubleshooting Common Configuration Issues

Port conflicts with X-Ray daemon Stop the legacy daemon (/var/aws/xray/xray daemon stop) before starting the ADOT collector, as both services bind to port 2000.

Missing centralized sampling If sampling rules defined in the X-Ray console are not applied, verify the awsproxy extension is included in your collector configuration and listed in service.extensions.

W3C trace ID compatibility Upgrade to ADOT 0.34.0+ with X-Ray exporter 0.86.0+ to ensure W3C-formatted trace IDs are accepted rather than dropped silently.

Promoting attributes to annotations To make custom attributes searchable in the X-Ray console, add them to the aws.xray.annotations list:

span.set_attribute("aws.xray.annotations", ["order_id"])

Lambda cold-start overhead The ADOT layer adds memory and initialization time. For latency-critical functions, consider keeping the native X-Ray SDK; otherwise, accept the overhead for multi-backend observability benefits.

Summary

• ADOT replaces the X-Ray daemon with a unified collector that handles both traces and metrics.
• Configuration requires three components: OTLP receivers, the awsxray exporter, and the awsproxy extension for centralized sampling.
• Applications send OTLP to ports 4317/4318 and use OTEL_TRACES_SAMPLER=xray to respect console-defined sampling rules.
• Deployment patterns include EKS DaemonSets, ECS sidecars, and Lambda layers, each requiring specific IAM permissions for X-Ray and CloudWatch.
• Version 0.34.0+ is mandatory to prevent W3C trace ID truncation.

Frequently Asked Questions

What is the difference between the ADOT collector and the X-Ray daemon?

The X-Ray daemon is a lightweight UDP proxy that forwards segments to AWS, while the ADOT collector is a full OpenTelemetry implementation that receives OTLP-formatted telemetry, processes it through configurable pipelines, and exports to multiple backends simultaneously (X-Ray for traces, CloudWatch for metrics). The collector supports centralized sampling and W3C trace context propagation, which the daemon does not.

Why do I need the awsproxy extension?

The awsproxy extension exposes the X-Ray GetSamplingRules and GetSamplingTargets API calls locally to your collector instance. Without it, your applications cannot retrieve centralized sampling rules from the AWS X-Ray console, and the xray sampler in your OpenTelemetry SDK falls back to local default sampling rates. As implemented in skills/core-skills/aws-observability/references/tracing.md, this extension is required for any deployment using console-managed sampling rules.

How do I migrate from the X-Ray SDK to ADOT without losing trace continuity?

Set OTEL_PROPAGATORS=xray,tracecontext in your application environment to support both X-Ray and W3C trace headers during the migration period. Stop the X-Ray daemon to free port 2000, then deploy the ADOT collector with the configuration from skills/core-skills/aws-observability/assets/otel-config.yaml. Ensure your collector version is 0.34.0+ to handle W3C-formatted trace IDs that newer OpenTelemetry SDKs emit.

What IAM permissions does the ADOT collector require?

The collector needs xray:PutTraceSegments and xray:PutTelemetryRecords to send trace data, plus cloudwatch:PutMetricData if using the EMF exporter for metrics. If using centralized sampling, the collector also requires xray:GetSamplingRules and xray:GetSamplingTargets. These permissions should be attached to the task role (ECS), instance profile (EC2), or service account role (EKS IRSA).

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}