# How Observability is Implemented in AWS Agent Toolkit: CloudWatch, X-Ray, and OpenTelemetry

> Explore how AWS Agent Toolkit implements observability using CloudWatch for metrics and logs, X-Ray for tracing, and OpenTelemetry for integrated monitoring. Learn more.

- Repository: [Amazon Web Services/agent-toolkit-for-aws](https://github.com/aws/agent-toolkit-for-aws)
- Tags: how-to-guide
- Published: 2026-06-28

---

**AWS Agent Toolkit implements observability through a three-layer stack: Amazon CloudWatch for metrics and logs, AWS X-Ray and ADOT (OpenTelemetry) for distributed tracing, and the ADOT Collector for centralized processing and export.**

The `aws/agent-toolkit-for-aws` repository provides a **full-stack observability framework** that covers every skill and plugin in the toolkit. This implementation is declarative, language-agnostic, and designed to work across Lambda, Fargate, and EC2 environments. The system automatically generates dashboards, manages alarm lifecycles, and correlates traces with logs to give operators a single pane of glass.

## Metrics and Logging with Amazon CloudWatch

The foundation of observability in AWS Agent Toolkit rests on **Amazon CloudWatch**, handling both numeric metrics and structured log data.

### Custom Metrics and EMF

Skills emit metrics via the **Embedded Metric Format (EMF)** or direct SDK calls. The toolkit defines standard metrics such as `AgentToolkitInvocationCount` and `SkillExecutionLatency` in [`skills/core-skills/aws-observability/SKILL.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/core-skills/aws-observability/SKILL.md).

From Python, you emit EMF by writing a structured JSON object to stdout:

```python
import json
import logging

logger = logging.getLogger()
logger.setLevel(logging.INFO)

def handler(event, context):
    metric = {
        "_aws": {
            "Timestamp": int(context.get_remaining_time_in_millis() / 1000),
            "CloudWatchMetrics": [
                {
                    "Namespace": "AgentToolkit",
                    "Dimensions": [["Skill"]],
                    "Metrics": [{"Name": "InvocationCount", "Unit": "Count"}]
                }
            ]
        },
        "Skill": "my-skill",
        "InvocationCount": 1
    }
    logger.info(json.dumps(metric))  # CloudWatch extracts metric automatically

```

This approach eliminates the need for synchronous `PutMetricData` calls, reducing latency and cost.

### Structured Logging and Insights

Every skill writes **structured JSON logs** to CloudWatch Logs. The repository includes ready-to-use **CloudWatch Logs Insights** query libraries in [`skills/core-skills/aws-observability/references/log-insights.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/core-skills/aws-observability/references/log-insights.md).

Queries filter by trace ID, skill name, or error type, enabling rapid root-cause analysis across distributed components.

### Automated Dashboards and Alarms

The [`scripts/generate_dashboards.py`](https://github.com/aws/agent-toolkit-for-aws/blob/main/scripts/generate_dashboards.py) utility automates infrastructure provisioning. It generates CloudFormation templates containing **standard alarm packs** and per-skill dashboards.

Deploy a complete observability stack for a serverless skill:

```bash
python3 scripts/generate_dashboards.py \
  --serverless my-skill \
  --output observability.json

aws cloudformation deploy \
  --template-file observability.json \
  --stack-name my-skill-observability

```

This script references [`skills/core-skills/aws-observability/references/dashboards.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/core-skills/aws-observability/references/dashboards.md) for widget layouts and [`skills/core-skills/aws-observability/references/alarms.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/core-skills/aws-observability/references/alarms.md) for threshold best practices.

## Distributed Tracing with X-Ray and ADOT

The tracing layer captures end-to-end request flows, supporting both legacy AWS X-Ray and modern OpenTelemetry implementations.

### Legacy X-Ray vs. Modern OpenTelemetry

AWS Agent Toolkit maintains the **X-Ray SDK** in maintenance mode for backward compatibility only. All new projects use the **AWS Distro for OpenTelemetry (ADOT)**.

The ADOT Collector runs as a Lambda layer, sidecar container, or EKS DaemonSet. It receives telemetry via OTLP (OpenTelemetry Protocol) and exports to X-Ray, CloudWatch, Prometheus, or OpenSearch.

Enable active tracing in AWS CDK:

```typescript
import { Tracing } from 'aws-cdk-lib/aws-lambda';
import * as lambda from 'aws-cdk-lib/aws-lambda';

const fn = new lambda.Function(this, 'MyFunction', {
  runtime: lambda.Runtime.NODEJS_20_X,
  handler: 'index.handler',
  code: lambda.Code.fromAsset('lambda'),
  tracing: Tracing.ACTIVE,  // Enables ADOT auto-instrumentation layer
});

```

### Annotations, Metadata, and Sampling

According to [`skills/core-skills/aws-observability/references/tracing.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/core-skills/aws-observability/references/tracing.md), the toolkit distinguishes between **annotations** and **metadata**:

- **Annotations** are indexed (maximum 50 per trace) and searchable in X-Ray
- **Metadata** is unindexed and stored for context only

Set annotations programmatically:

```python
from opentelemetry import trace

span = trace.get_current_span()
span.set_attribute("aws.xray.annotations", ["order_id", "customer_tier"])
span.set_attribute("order_id", "12345")          # Searchable

span.set_attribute("customer_tier", "gold")

```

**Sampling** defaults to 1 request per second plus 5% of additional traffic. Override this via centralized sampling rules using the `awsproxy` extension.

### Trace-Log Correlation

Span IDs are automatically injected into application logs. This correlation, documented in [`tracing.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/tracing.md), allows you to pivot from a metric anomaly in CloudWatch to the specific distributed trace in X-Ray without manual context switching.

## The ADOT Collector Pipeline

The **ADOT Collector** serves as the central nervous system for telemetry, configured via [`skills/core-skills/aws-observability/assets/otel-config.yaml`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/core-skills/aws-observability/assets/otel-config.yaml).

### Collector Configuration

The configuration defines receivers, processors, and exporters in a declarative YAML structure:

```yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 30s
    send_batch_size: 8192

exporters:
  awsxray:
    region: us-east-1
  awsemf:
    namespace: MyApplication
    region: us-east-1

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [awsxray]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [awsemf]

```

This pipeline batches telemetry to reduce API calls and supports multiple export destinations simultaneously.

### Cardinality Defense

The toolkit implements a **three-layer cardinality defense** to prevent cost explosions:

1. **SDK Level** – Avoid high-cardinality attributes in code
2. **Collector Filter Processor** – Drop unwanted metrics early in the pipeline
3. **Backend Filtering** – Use CloudWatch `dimension_rollup_option` or Prometheus relabeling rules

## End-to-End Implementation Flow

A typical skill execution flows through the observability stack as follows:

1. **Execution** – Skill handler emits EMF metrics and JSON logs to CloudWatch
2. **Instrumentation** – ADOT SDK creates spans, adds annotations, and forwards traces via OTLP to the local collector
3. **Processing** – Collector batches spans, applies filter processors, and exports to X-Ray or CloudWatch EMF
4. **Visualization** – Pre-generated dashboards from [`generate_dashboards.py`](https://github.com/aws/agent-toolkit-for-aws/blob/main/generate_dashboards.py) visualize metrics and trace-driven alarms

All components are **declarative** (YAML, CloudFormation) and **language-agnostic**, allowing incremental adoption starting with CloudWatch metrics and progressing to full distributed tracing.

## Summary

- **AWS Agent Toolkit** embeds observability via CloudWatch for metrics/logs and ADOT for tracing.
- **Metrics** use EMF (Embedded Metric Format) for zero-latency ingestion, with custom metrics like `AgentToolkitInvocationCount` defined in [`SKILL.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/SKILL.md).
- **Tracing** has migrated from X-Ray SDK to ADOT (OpenTelemetry), using the collector config in [`otel-config.yaml`](https://github.com/aws/agent-toolkit-for-aws/blob/main/otel-config.yaml) to process spans.
- **Automation** via [`scripts/generate_dashboards.py`](https://github.com/aws/agent-toolkit-for-aws/blob/main/scripts/generate_dashboards.py) generates CloudFormation templates for alarms and dashboards.
- **Correlation** between traces and logs is automatic, enabling rapid debugging across the serverless stack.

## Frequently Asked Questions

### What is the difference between annotations and metadata in AWS Agent Toolkit tracing?

**Annotations are indexed and searchable in X-Ray, with a limit of 50 per trace**, while metadata is unindexed contextual data. According to [`skills/core-skills/aws-observability/references/tracing.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/core-skills/aws-observability/references/tracing.md), you should place high-value filter keys like `order_id` or `customer_tier` in annotations, and debugging details in metadata.

### How does AWS Agent Toolkit handle high-cardinality metrics?

The toolkit implements a **three-layer defense**: avoid high-cardinality attributes at the SDK level, use the Collector's filter processor to drop unwanted metrics, and apply backend-specific filtering like CloudWatch's `dimension_rollup_option`. This prevents runaway costs while maintaining observability granularity.

### Can I use the observability features without modifying my skill code?

Yes. **Dashboards and alarms** can be generated entirely via the [`generate_dashboards.py`](https://github.com/aws/agent-toolkit-for-aws/blob/main/generate_dashboards.py) script using CloudFormation. For tracing, simply enable the **ADOT Lambda layer** or sidecar configuration; the auto-instrumentation captures spans without code changes. Custom metrics require EMF logging or SDK calls as shown in [`skills/core-skills/aws-observability/references/metrics.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/core-skills/aws-observability/references/metrics.md).

### What replaced the AWS X-Ray SDK in AWS Agent Toolkit?

**ADOT (AWS Distro for OpenTelemetry)** has replaced the X-Ray SDK for new development. The X-Ray SDK remains in maintenance mode for backward compatibility only. ADOT provides broader backend support (X-Ray, CloudWatch, Prometheus, OpenSearch) and standardized OTLP transport, configured via [`skills/core-skills/aws-observability/assets/otel-config.yaml`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/core-skills/aws-observability/assets/otel-config.yaml).