AWS Agent Toolkit Observability Metrics: A Complete Guide to CloudWatch, X-Ray, and ADOT Telemetry

The AWS Agent Toolkit aggregates native CloudWatch, X-Ray, and ADOT telemetry emitted by AWS services—rather than generating custom metrics—providing curated reference files, alarm templates, and collector configurations to help developers discover and monitor critical infrastructure signals.

The aws/agent-toolkit-for-aws repository bundles a dedicated aws-observability skill that serves as a centralized catalog for AWS observability data. According to the source code in skills/core-skills/aws-observability/SKILL.md, the toolkit does not create proprietary metrics; instead, it compiles the canonical CloudWatch metrics that services like EC2, Lambda, and RDS already emit, organizing them into actionable reference documents with recommended thresholds and alerting strategies.

Core Observability Metrics by Service

The toolkit’s references/metrics.md file catalogs the primary metric families across major AWS services, providing namespaces, typical dimensions, and default alerting guidance.

EC2 and Auto-Scaling Metrics

For compute instances, the toolkit references standard CloudWatch metrics including CPUUtilization, NetworkIn, NetworkOut, DiskReadOps, DiskWriteOps, and StatusCheckFailed. These metrics appear in the EC2 section of the metrics reference file and form the basis for the EC2 alarm templates provided in references/alarms.md.

Lambda Function Metrics

Serverless monitoring centers on Invocations, Duration, Errors, Throttles, IteratorAge, ConcurrentExecutions, and UnreservedConcurrentExecutions. The Lambda section in references/metrics.md documents these alongside recommended resolution periods and threshold defaults, such as alerting on error rates exceeding 1% for two consecutive minutes.

ElastiCache (Redis and Valkey) Metrics

For in-memory caching, the toolkit highlights EngineCPUUtilization as the preferred metric over host-level CPUUtilization, alongside CurrConnections, NetworkBytesOut, DatabaseMemoryUsagePercentage, and FreeableMemory. These metrics help identify memory pressure and connection saturation before they impact application performance.

RDS and Aurora Metrics

Database observability relies on CPUUtilization, DatabaseConnections, ReadLatency, WriteLatency, FreeableMemory, BufferCacheHitRatio, and DiskQueueDepth. The toolkit’s reference documentation notes specific thresholds, such as triggering alarms when BufferCacheHitRatio drops below 95% for MySQL instances or when DiskQueueDepth exceeds 1 for sustained periods.

DocumentDB Metrics

MongoDB-compatible workloads use CPUUtilization, DatabaseConnections, ReadLatency, WriteLatency, and FreeableMemory. The DocumentDB section follows the same structure as RDS, providing consistent dimensional keys for cluster-wide monitoring.

EKS and ECS Container Metrics

Container orchestration monitoring includes CPUUtilization and MemoryUtilization at the cluster level, plus ECSServiceAverageCPUUtilization and ECSServiceAverageMemoryUtilization for Fargate and EC2 launch types. When using FireLens for log routing, the toolkit also surfaces containerCPUUtilization and containerMemoryUtilization as available dimensions.

OpenSearch Service Metrics

Search and analytics workloads track CPUUtilization, JVMMemoryPressure, FreeStorageSpace, ClusterStatusRed, SearchLatency, and IndexingLatency. The toolkit specifically calls out JVMMemoryPressure as a critical early indicator of cluster instability, recommending alerts at 80% utilization.

Managed Flink Metrics

Stream processing observability uses containerCPUUtilization, containerMemoryUtilization, heapMemoryUtilization, and oldGenerationGCTime. These metrics, documented in the Flink section of references/metrics.md, help diagnose memory pressure and garbage collection bottlenecks in long-running streaming applications.

X-Ray and ADOT Trace Metrics

Distributed tracing metrics such as awsxray_segment_count and awsxray_error_rate are exposed via the ADOT (AWS Distro for OpenTelemetry) collector. The toolkit provides the otel-config.yaml configuration file in skills/core-skills/aws-observability/assets/otel-config.yaml to enable these metrics, bridging trace data with CloudWatch metric streams.

Custom and EMF Metrics

Any user-defined metric emitted via PutMetricData or the Embedded Metric Format (EMF) is supported. The toolkit ships an example alarm-template.ts demonstrating how to publish custom EMF metrics from Lambda functions, allowing applications to emit business-specific telemetry alongside infrastructure metrics.

Key Source Files in the Observability Skill

The observability capability is organized into specific files that define the metric catalog and implementation patterns:

• skills/core-skills/aws-observability/SKILL.md — Defines the skill’s scope and routing table, pointing users to the metric reference files and alarm templates.
• skills/core-skills/aws-observability/references/metrics.md — The canonical catalog enumerating each metric, its namespace, dimensions, and recommended resolution.
• skills/core-skills/aws-observability/references/alarms.md — Builds on the metric list to provide ready-to-use alarm templates for each service with predefined thresholds.
• skills/core-skills/aws-observability/assets/alarm-template.ts — Executable CDK asset demonstrating Lambda monitoring alarms and dashboard creation.
• skills/core-skills/aws-observability/assets/otel-config.yaml — ADOT collector configuration enabling X-Ray tracing and custom EMF metric forwarding.
• skills/core-skills/aws-observability/references/tracing.md — Guidance on enabling X-Ray, configuring ADOT, and interpreting trace-related metadata.

Practical Examples: Querying and Alerting on Metrics

Below are concrete implementations showing how to interact with the AWS Agent Toolkit’s observability metrics.

Querying EC2 CPU Metrics

Use the built-in call_aws helper or standard AWS CLI to retrieve specific instance metrics:

call_aws cloudwatch list-metrics \
  --namespace AWS/EC2 \
  --metric-name CPUUtilization \
  --dimensions Name=InstanceId,Value=i-0123456789abcdef0

Creating RDS CPU Alarms with CDK

The alarm-template.ts asset demonstrates creating threshold-based alarms:

import * as cloudwatch from 'aws-cdk-lib/aws-cloudwatch';
import * as rds from 'aws-cdk-lib/aws-rds';

const db = new rds.DatabaseInstance(this, 'MyDB', { /* … */ });

new cloudwatch.Alarm(this, 'CPUAlarm', {
  metric: db.metricCPUUtilization(),
  threshold: 80,
  evaluationPeriods: 3,
  comparisonOperator: cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
});

Emitting Custom EMF Metrics from Lambda

Publish business metrics using the AWS SDK, as shown in the toolkit’s Lambda examples:

const { putMetricData } = require('aws-sdk').CloudWatch;

exports.handler = async () => {
  await putMetricData({
    Namespace: 'MyApp',
    MetricData: [{
      MetricName: 'OrdersProcessed',
      Value: 42,
      Unit: 'Count',
    }],
  }).promise();
};

Configuring ADOT for X-Ray Traces

Enable trace-derived metrics using the collector configuration from otel-config.yaml:

receivers:
  awsprometheus:
    endpoint: 0.0.0.0:2020
exporters:
  awsxray:
processors:
  batch:
service:
  pipelines:
    traces:
      receivers: [awsprometheus]
      processors: [batch]
      exporters: [awsxray]

Summary

• The AWS Agent Toolkit aggregates native CloudWatch metrics rather than generating custom telemetry, covering EC2, Lambda, RDS, ElastiCache, DocumentDB, EKS, ECS, OpenSearch, and Flink.
• Metric definitions and alerting thresholds reside in skills/core-skills/aws-observability/references/metrics.md and alarms.md.
• The toolkit provides executable CDK assets (alarm-template.ts) and ADOT configurations (otel-config.yaml) to operationalize monitoring quickly.
• X-Ray and ADOT trace metrics are supported via the OpenTelemetry collector configuration included in the repository.
• Custom EMF metrics are supported through Lambda examples demonstrating PutMetricData API usage.

Frequently Asked Questions

Does the AWS Agent Toolkit generate its own metrics?

No. According to the source code in skills/core-skills/aws-observability/SKILL.md, the toolkit exclusively aggregates telemetry that AWS services already emit to CloudWatch, X-Ray, and ADOT. It provides curated reference files that document these native metrics, their dimensions, and recommended alarm thresholds, but does not implement custom metric generation logic.

How do I access the complete metric catalog for a specific service?

Navigate to skills/core-skills/aws-observability/references/metrics.md in the repository. This file contains the canonical list of CloudWatch metrics organized by service, including namespace specifications, typical dimensions (such as InstanceId or FunctionName), and suggested resolution periods. For implementation examples, see assets/alarm-template.ts for CDK patterns or references/alarms.md for static alarm definitions.

Can I use the toolkit to monitor custom business metrics?

Yes. The toolkit supports custom metrics emitted via the CloudWatch PutMetricData API or the Embedded Metric Format (EMF). The alarm-template.ts file includes examples of publishing custom metrics from Lambda functions, while assets/otel-config.yaml demonstrates how to configure the ADOT collector to forward custom EMF metrics to CloudWatch.

What is the difference between the alarms.md reference and the alarm-template.ts asset?

The references/alarms.md file provides static documentation describing recommended alarm thresholds, evaluation periods, and comparison operators for each metric family. The assets/alarm-template.ts file is an executable TypeScript asset using AWS CDK that demonstrates programmatically creating these alarms and associated dashboards, allowing you to deploy production-ready monitoring infrastructure as code.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}