How Observability is Implemented in AWS Agent Toolkit: CloudWatch, X-Ray, and OpenTelemetry

AWS Agent Toolkit implements observability through a three-layer stack: Amazon CloudWatch for metrics and logs, AWS X-Ray and ADOT (OpenTelemetry) for distributed tracing, and the ADOT Collector for centralized processing and export.

The aws/agent-toolkit-for-aws repository provides a full-stack observability framework that covers every skill and plugin in the toolkit. This implementation is declarative, language-agnostic, and designed to work across Lambda, Fargate, and EC2 environments. The system automatically generates dashboards, manages alarm lifecycles, and correlates traces with logs to give operators a single pane of glass.

Metrics and Logging with Amazon CloudWatch

The foundation of observability in AWS Agent Toolkit rests on Amazon CloudWatch, handling both numeric metrics and structured log data.

Custom Metrics and EMF

Skills emit metrics via the Embedded Metric Format (EMF) or direct SDK calls. The toolkit defines standard metrics such as AgentToolkitInvocationCount and SkillExecutionLatency in skills/core-skills/aws-observability/SKILL.md.

From Python, you emit EMF by writing a structured JSON object to stdout:

import json
import logging

logger = logging.getLogger()
logger.setLevel(logging.INFO)

def handler(event, context):
    metric = {
        "_aws": {
            "Timestamp": int(context.get_remaining_time_in_millis() / 1000),
            "CloudWatchMetrics": [
                {
                    "Namespace": "AgentToolkit",
                    "Dimensions": [["Skill"]],
                    "Metrics": [{"Name": "InvocationCount", "Unit": "Count"}]
                }
            ]
        },
        "Skill": "my-skill",
        "InvocationCount": 1
    }
    logger.info(json.dumps(metric))  # CloudWatch extracts metric automatically

This approach eliminates the need for synchronous PutMetricData calls, reducing latency and cost.

Structured Logging and Insights

Every skill writes structured JSON logs to CloudWatch Logs. The repository includes ready-to-use CloudWatch Logs Insights query libraries in skills/core-skills/aws-observability/references/log-insights.md.

Queries filter by trace ID, skill name, or error type, enabling rapid root-cause analysis across distributed components.

Automated Dashboards and Alarms

The scripts/generate_dashboards.py utility automates infrastructure provisioning. It generates CloudFormation templates containing standard alarm packs and per-skill dashboards.

Deploy a complete observability stack for a serverless skill:

python3 scripts/generate_dashboards.py \
  --serverless my-skill \
  --output observability.json

aws cloudformation deploy \
  --template-file observability.json \
  --stack-name my-skill-observability

This script references skills/core-skills/aws-observability/references/dashboards.md for widget layouts and skills/core-skills/aws-observability/references/alarms.md for threshold best practices.

Distributed Tracing with X-Ray and ADOT

The tracing layer captures end-to-end request flows, supporting both legacy AWS X-Ray and modern OpenTelemetry implementations.

Legacy X-Ray vs. Modern OpenTelemetry

AWS Agent Toolkit maintains the X-Ray SDK in maintenance mode for backward compatibility only. All new projects use the AWS Distro for OpenTelemetry (ADOT).

The ADOT Collector runs as a Lambda layer, sidecar container, or EKS DaemonSet. It receives telemetry via OTLP (OpenTelemetry Protocol) and exports to X-Ray, CloudWatch, Prometheus, or OpenSearch.

Enable active tracing in AWS CDK:

import { Tracing } from 'aws-cdk-lib/aws-lambda';
import * as lambda from 'aws-cdk-lib/aws-lambda';

const fn = new lambda.Function(this, 'MyFunction', {
  runtime: lambda.Runtime.NODEJS_20_X,
  handler: 'index.handler',
  code: lambda.Code.fromAsset('lambda'),
  tracing: Tracing.ACTIVE,  // Enables ADOT auto-instrumentation layer
});

Annotations, Metadata, and Sampling

According to skills/core-skills/aws-observability/references/tracing.md, the toolkit distinguishes between annotations and metadata:

• Annotations are indexed (maximum 50 per trace) and searchable in X-Ray
• Metadata is unindexed and stored for context only

Set annotations programmatically:

from opentelemetry import trace

span = trace.get_current_span()
span.set_attribute("aws.xray.annotations", ["order_id", "customer_tier"])
span.set_attribute("order_id", "12345")          # Searchable

span.set_attribute("customer_tier", "gold")

Sampling defaults to 1 request per second plus 5% of additional traffic. Override this via centralized sampling rules using the awsproxy extension.

Trace-Log Correlation

Span IDs are automatically injected into application logs. This correlation, documented in tracing.md, allows you to pivot from a metric anomaly in CloudWatch to the specific distributed trace in X-Ray without manual context switching.

The ADOT Collector Pipeline

The ADOT Collector serves as the central nervous system for telemetry, configured via skills/core-skills/aws-observability/assets/otel-config.yaml.

Collector Configuration

The configuration defines receivers, processors, and exporters in a declarative YAML structure:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 30s
    send_batch_size: 8192

exporters:
  awsxray:
    region: us-east-1
  awsemf:
    namespace: MyApplication
    region: us-east-1

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [awsxray]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [awsemf]

This pipeline batches telemetry to reduce API calls and supports multiple export destinations simultaneously.

Cardinality Defense

The toolkit implements a three-layer cardinality defense to prevent cost explosions:

1. SDK Level – Avoid high-cardinality attributes in code
2. Collector Filter Processor – Drop unwanted metrics early in the pipeline
3. Backend Filtering – Use CloudWatch dimension_rollup_option or Prometheus relabeling rules

End-to-End Implementation Flow

A typical skill execution flows through the observability stack as follows:

1. Execution – Skill handler emits EMF metrics and JSON logs to CloudWatch
2. Instrumentation – ADOT SDK creates spans, adds annotations, and forwards traces via OTLP to the local collector
3. Processing – Collector batches spans, applies filter processors, and exports to X-Ray or CloudWatch EMF
4. Visualization – Pre-generated dashboards from generate_dashboards.py visualize metrics and trace-driven alarms

All components are declarative (YAML, CloudFormation) and language-agnostic, allowing incremental adoption starting with CloudWatch metrics and progressing to full distributed tracing.

Summary

• AWS Agent Toolkit embeds observability via CloudWatch for metrics/logs and ADOT for tracing.
• Metrics use EMF (Embedded Metric Format) for zero-latency ingestion, with custom metrics like AgentToolkitInvocationCount defined in SKILL.md.
• Tracing has migrated from X-Ray SDK to ADOT (OpenTelemetry), using the collector config in otel-config.yaml to process spans.
• Automation via scripts/generate_dashboards.py generates CloudFormation templates for alarms and dashboards.
• Correlation between traces and logs is automatic, enabling rapid debugging across the serverless stack.

Frequently Asked Questions

What is the difference between annotations and metadata in AWS Agent Toolkit tracing?

Annotations are indexed and searchable in X-Ray, with a limit of 50 per trace, while metadata is unindexed contextual data. According to skills/core-skills/aws-observability/references/tracing.md, you should place high-value filter keys like order_id or customer_tier in annotations, and debugging details in metadata.

How does AWS Agent Toolkit handle high-cardinality metrics?

The toolkit implements a three-layer defense: avoid high-cardinality attributes at the SDK level, use the Collector's filter processor to drop unwanted metrics, and apply backend-specific filtering like CloudWatch's dimension_rollup_option. This prevents runaway costs while maintaining observability granularity.

Can I use the observability features without modifying my skill code?

Yes. Dashboards and alarms can be generated entirely via the generate_dashboards.py script using CloudFormation. For tracing, simply enable the ADOT Lambda layer or sidecar configuration; the auto-instrumentation captures spans without code changes. Custom metrics require EMF logging or SDK calls as shown in skills/core-skills/aws-observability/references/metrics.md.

What replaced the AWS X-Ray SDK in AWS Agent Toolkit?

ADOT (AWS Distro for OpenTelemetry) has replaced the X-Ray SDK for new development. The X-Ray SDK remains in maintenance mode for backward compatibility only. ADOT provides broader backend support (X-Ray, CloudWatch, Prometheus, OpenSearch) and standardized OTLP transport, configured via skills/core-skills/aws-observability/assets/otel-config.yaml.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}