Configuring X-Ray Tracing with ADOT Collector for Lambda Functions

The AWS Agent Toolkit for AWS provides a complete migration path from the legacy X-Ray SDK to the AWS Distro for OpenTelemetry (ADOT) collector, requiring a custom otel-config.yaml, specific IAM permissions, and ADOT Collector version 0.34.0 or later to support W3C trace IDs.

The aws/agent-toolkit-for-aws repository ships production-ready guidance for replacing the legacy X-Ray SDK with the ADOT collector in serverless environments. Configuring X-Ray tracing with ADOT collector for Lambda functions involves attaching a managed Lambda layer, deploying a custom OpenTelemetry configuration file, and ensuring your execution role has the appropriate IAM permissions. This approach aligns with AWS's current observability best practices while providing richer telemetry than the deprecated SDK.

ADOT vs. X-Ray SDK: Migration Recommendations

According to the source code in skills/core-skills/aws-observability/references/tracing.md, the X-Ray SDK is now in maintenance mode. AWS recommends ADOT for all new Lambda projects.

Maintenance Mode Status

The toolkit explicitly states that the X-Ray SDK remains available only for backward compatibility. New implementations should adopt the ADOT collector to ensure access to ongoing feature development and security updates.

Centralized Sampling Support

The ADOT configuration supports X-Ray centralized sampling rules when you include the awsproxy extension. This capability allows you to manage sampling rates across your entire fleet from the X-Ray console, rather than configuring sampling locally in each function.

Prerequisites and Version Requirements

You must use ADOT Collector version 0.34.0 or later with the X-Ray exporter version 0.86.0 or later. Earlier versions silently drop W3C-format trace IDs, breaking trace continuity for services that use the W3C propagation standard.

Implementing the ADOT Collector Configuration

Step 1: Attach the Lambda Layer

Add the AWS-managed ADOT layer to your function. The following AWS CLI example attaches the Python 3.10 layer:

aws lambda update-function-configuration \
  --function-name MyLambda \
  --layers arn:aws:lambda:<region>:901920570463:layer:AWSDistroOpenTelemetryCollector-Python310:3

When attached, the collector runs automatically inside the function execution environment.

Step 2: Configure the Collector

Create an otel-config.yaml file based on the reference implementation in skills/core-skills/aws-observability/assets/otel-config.yaml. This configuration sends traces to X-Ray and emits metrics to CloudWatch via EMF:

receivers:
  awsxray:
    endpoint: "0.0.0.0:2000"
exporters:
  awsxray:
  awsemf:
    namespace: "MyApp"
    region: "<AWS_REGION>"
processors:
  batch:
    timeout: 10s
service:
  pipelines:
    traces:
      receivers: [awsxray]
      exporters: [awsxray]
    metrics:
      receivers: [awsxray]
      exporters: [awsemf]
extensions:
  awsproxy: {}

Store this file alongside your function code and reference it via the OTEL_CONFIG_FILE environment variable set to /var/task/otel-config.yaml.

Step 3: Configure IAM Permissions

The Lambda execution role requires permissions to write trace data. Attach the AWSXRayDaemonWriteAccess managed policy, which includes xray:PutTelemetryRecords and xray:PutTraceSegments:

aws iam attach-role-policy \
  --role-name MyLambdaRole \
  --policy-arn arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess

If using the awsproxy extension for centralized sampling, ensure the role also has permissions for the CloudWatch Agent or relevant proxy operations as detailed in skills/core-skills/aws-observability/references/tracing.md.

Performance and Cold Start Considerations

The ADOT layer adds memory overhead and cold-start latency because the collector runs within the same execution environment as your function. For latency-critical workloads, the toolkit notes that you may still prefer the X-Ray SDK despite its maintenance status. Size your function's memory allocation accordingly to accommodate the collector process.

Summary

• ADOT replaces X-Ray SDK: The X-Ray SDK is in maintenance mode; ADOT is the recommended path for new Lambda functions.
• Version 0.34.0+ required: Earlier ADOT versions silently drop W3C-format trace IDs.
• Configuration: Deploy an otel-config.yaml with the awsproxy extension for centralized sampling.
• Permissions: Attach AWSXRayDaemonWriteAccess to your Lambda execution role.
• Performance: Expect increased cold-start latency and memory usage when using the ADOT layer.

Frequently Asked Questions

What is the difference between X-Ray SDK and ADOT for Lambda?

The X-Ray SDK is an AWS-specific instrumentation library now in maintenance mode. ADOT is an AWS-supported distribution of OpenTelemetry that provides standards-based instrumentation while maintaining compatibility with X-Ray backends. ADOT supports W3C trace propagation and centralized sampling rules, which the X-Ray SDK handles differently.

Why does my Lambda function drop W3C format trace IDs?

You are likely using an ADOT Collector version earlier than 0.34.0 or an X-Ray exporter version earlier than 0.86.0. These older versions do not recognize W3C trace ID formats and drop them silently. Upgrade to ADOT Collector 0.34.0 or later to ensure compatibility with W3C propagation standards.

How do I enable centralized sampling rules with ADOT?

Include the awsproxy extension in your otel-config.yaml file under the extensions section. This extension allows the collector to communicate with the X-Ray service to fetch centralized sampling rules configured in the AWS console. Without this extension, sampling decisions are made locally and ignore centralized configurations.

Does ADOT increase Lambda cold start latency?

Yes. Adding the ADOT layer to your Lambda function introduces additional memory overhead and cold-start latency because the collector runs as a separate process within the execution environment. For applications where cold-start performance is critical, consider whether the X-Ray SDK (despite being in maintenance mode) better suits your latency requirements.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}