Configuring AWS X-Ray Tracing and Migrating to ADOT: A Complete Guide

AWS X-Ray SDK is now in maintenance mode, so you should migrate to ADOT (AWS Distro for OpenTelemetry) by deploying the collector with a custom configuration, updating environment variables for centralized sampling, and stopping the X-Ray daemon to avoid port conflicts.

The aws/agent-toolkit-for-aws repository provides definitive guidance for implementing distributed tracing across serverless and containerized workloads. Whether you are enabling X-Ray tracing on new services or modernizing legacy instrumented applications, the recommended path forward involves adopting the OpenTelemetry standard via ADOT.

Why Migrate from X-Ray SDK to ADOT?

The X-Ray SDK is currently in maintenance mode, while ADOT is actively developed and supports vendor-neutral instrumentation. According to the source code analysis in skills/core-skills/aws-observability/references/tracing.md, ADOT offers superior flexibility for modern observability stacks.

Feature	X-Ray SDK	ADOT (OpenTelemetry)
Status	Maintenance mode	Actively developed
Back-ends	X-Ray only	X-Ray, CloudWatch, Prometheus, OpenSearch
Auto-instrumentation	Limited manual subsegments	Java, Python, Node.js Lambda layers
Vendor Lock-in	AWS-specific	Vendor-neutral (OTel standard)
Lambda Support	Built-in daemon	Lambda layer with auto-instrumentation

Enabling X-Ray Tracing in AWS Services

Before migrating, you must enable active tracing on your AWS services to generate trace data.

Lambda Functions

Set tracing: Tracing.ACTIVE in your CDK construct to enable the Lambda runtime to emit segments automatically. This configuration adds the X-Amzn-Trace-Id header to inbound requests.

import { Tracing } from 'aws-cdk-lib/aws-lambda';
import * as lambda from 'aws-cdk-lib/aws-lambda';

const fn = new lambda.Function(this, 'MyFunction', {
  runtime: lambda.Runtime.NODEJS_20_X,
  handler: 'index.handler',
  code: lambda.Code.fromAsset('lambda'),
  tracing: Tracing.ACTIVE,  // Enables X-Ray tracing
});

Source: skills/core-skills/aws-observability/references/tracing.md (lines 174-185).

API Gateway

Enable tracing on the deployment stage to capture incoming request traces.

import * as apigateway from 'aws-cdk-lib/aws-apigateway';

const api = new apigateway.RestApi(this, 'MyApi', {
  deployOptions: {
    tracingEnabled: true,  // Enables X-Ray tracing
  },
});

Source: skills/core-skills/aws-observability/references/tracing.md (lines 188-194).

Trace Structure Concepts

X-Ray organizes telemetry into hierarchical units:

• Trace: A collection of segments identified by a unique trace ID.
• Segment: A JSON document (max 64 KB) describing work done by a service.
• Subsegment: Granular pieces inside a segment for downstream calls or custom code.
• Inferred Segment: Generated automatically when downstream services lack instrumentation.

X-Ray retains trace data for 30 days (non-configurable).

Understanding X-Ray Annotations and Metadata

Annotations are indexed key-value pairs that support filtering in the X-Ray console (limited to 50 per trace). Metadata is non-indexed and suitable for debug data only.

When migrating to ADOT, all span attributes become X-Ray metadata by default. To maintain searchable fields, you must explicitly designate them as annotations using the aws.xray.annotations attribute key.

Source: skills/core-skills/aws-observability/references/tracing.md (lines 55-62).

Configuring Sampling Rules

X-Ray applies a default sampling rule of 1 request per second plus 5% of additional requests. You can create custom centralized sampling rules to prioritize high-traffic services or critical paths.

During migration, the ADOT collector must include the awsproxy extension to honor X-Ray centralized sampling rules. Without this extension, the collector falls back to local default sampling, which may cause inconsistent trace coverage.

Source: skills/core-skills/aws-observability/references/tracing.md (lines 25-33, 71-77).

ADOT Collector Configuration

The repository ships a ready-to-use collector configuration in skills/core-skills/aws-observability/assets/otel-config.yaml. This configuration routes OTLP traces to X-Ray and metrics to CloudWatch via the EMF exporter.

Key components include:

• Receivers: otlp (gRPC on port 4317, HTTP on port 4318).
• Processors: memory_limiter, batch, and optional filter for cardinality reduction.
• Exporters: awsxray for traces and awsemf for metrics.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 5s
    limit_mib: 160
    spike_limit_mib: 40
  batch:
    timeout: 30s
    send_batch_size: 8192

exporters:
  awsxray:
    region: us-east-1
  awsemf:
    namespace: MyApplication
    region: us-east-1

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [awsxray]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [awsemf]

Step-by-Step Migration to ADOT

Follow this sequence to migrate from the X-Ray SDK to ADOT without losing observability coverage.

1. Deploy the ADOT collector as a daemon, sidecar, or Lambda layer using the provided otel-config.yaml.

2. Update environment variables to enable centralized sampling and dual-header propagation:

   export OTEL_TRACES_SAMPLER=xray
   export OTEL_TRACES_SAMPLER_ARG=endpoint=http://localhost:2000
   export OTEL_PROPAGATORS=xray,tracecontext

   Source: skills/core-skills/aws-observability/references/tracing.md (lines 36-38).

3. Convert existing X-Ray subsegments to OpenTelemetry spans. Replace SDK calls with OTel API calls (e.g., opentelemetry.trace).

4. Add explicit annotations for searchable data by setting the aws.xray.annotations attribute:

   from opentelemetry import trace
   
   tracer = trace.get_tracer(__name__)
   
   with tracer.start_as_current_span("process_order") as span:
       span.set_attribute("aws.xray.annotations", ["order_id", "customer_tier"])
       span.set_attribute("order_id", "12345")
       span.set_attribute("customer_tier", "gold")

   Source: skills/core-skills/aws-observability/references/tracing.md (lines 13-21).

5. Stop the X-Ray daemon before starting ADOT. Both services use port 2000, and running both simultaneously causes silent data loss.

   Source: skills/core-skills/aws-observability/references/tracing.md (lines 49-51).

6. Validate trace visibility in the X-Ray console and verify that annotations appear as searchable fields rather than metadata.

Common Pitfalls and Fixes

Avoid these frequent errors during your migration:

Pitfall	Fix
Using X-Ray SDK for new code	Switch to the ADOT Lambda layer or OpenTelemetry SDK for auto-instrumentation.
Searchable data stored as metadata	Add attribute keys to aws.xray.annotations to ensure indexing.
Exceeding 50 annotations per trace	Consolidate attributes; use metadata for non-searchable debug data.
Missing awsproxy extension	Include awsproxy in the collector config to honor centralized sampling rules.
Port 2000 conflict	Stop the X-Ray daemon before starting the ADOT collector.

Source: skills/core-skills/aws-observability/references/tracing.md (lines 60-68).

Summary

• The X-Ray SDK is in maintenance mode; ADOT is the recommended replacement for new and existing projects.
• Enable tracing via CDK properties (Tracing.ACTIVE for Lambda, tracingEnabled: true for API Gateway) to generate trace headers.
• Annotations (indexed) and Metadata (non-indexed) handle different observability needs; ADOT requires explicit annotation configuration.
• Configure the awsproxy extension in ADOT to respect X-Ray centralized sampling rules.
• Avoid port conflicts by stopping the X-Ray daemon (port 2000) before deploying the ADOT collector.
• Convert subsegments to OpenTelemetry spans and use aws.xray.annotations to maintain searchable fields.

Frequently Asked Questions

Is the X-Ray SDK still supported?

The X-Ray SDK is in maintenance mode according to the aws/agent-toolkit-for-aws repository. While existing implementations continue to function, AWS recommends ADOT for all new projects and active development. Maintenance mode means critical bug fixes may still occur, but new features and performance improvements are exclusive to ADOT.

What is the default X-Ray sampling rate?

X-Ray uses a default sampling rule of 1 request per second plus 5% of additional requests across your account. You can override this with custom centralized sampling rules that prioritize specific services or URL paths. When using ADOT, you must configure the awsproxy extension to honor these centralized rules; otherwise, the collector defaults to local sampling.

Can I run X-Ray SDK and ADOT simultaneously?

No. Running both the X-Ray daemon and the ADOT collector simultaneously causes silent data loss because both services attempt to bind to port 2000. As documented in skills/core-skills/aws-observability/references/tracing.md (lines 49-51), you must stop the X-Ray daemon before starting the ADOT collector to ensure trace data flows correctly to the backend.

How do I make span attributes searchable in X-Ray?

By default, ADOT exports all span attributes as metadata, which is not indexed. To make attributes searchable, explicitly list them in the aws.xray.annotations array on the span. Remember that X-Ray limits you to 50 annotations per trace, so prioritize high-cardinality fields like order_id or customer_tier for annotation status, and leave debug data as metadata.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}