# AWS Agent Toolkit Observability Metrics: A Complete Guide to CloudWatch, X-Ray, and ADOT Telemetry

> Learn how the AWS Agent Toolkit uses CloudWatch, X-Ray, and ADOT telemetry for observability. Discover critical infrastructure signals with curated files, alarm templates, and configurations.

- Repository: [Amazon Web Services/agent-toolkit-for-aws](https://github.com/aws/agent-toolkit-for-aws)
- Tags: how-to-guide
- Published: 2026-06-28

---

**The AWS Agent Toolkit aggregates native CloudWatch, X-Ray, and ADOT telemetry emitted by AWS services—rather than generating custom metrics—providing curated reference files, alarm templates, and collector configurations to help developers discover and monitor critical infrastructure signals.**

The `aws/agent-toolkit-for-aws` repository bundles a dedicated **aws-observability** skill that serves as a centralized catalog for AWS observability data. According to the source code in [`skills/core-skills/aws-observability/SKILL.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/core-skills/aws-observability/SKILL.md), the toolkit does not create proprietary metrics; instead, it compiles the canonical CloudWatch metrics that services like EC2, Lambda, and RDS already emit, organizing them into actionable reference documents with recommended thresholds and alerting strategies.

## Core Observability Metrics by Service

The toolkit’s [`references/metrics.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/references/metrics.md) file catalogs the primary metric families across major AWS services, providing namespaces, typical dimensions, and default alerting guidance.

### EC2 and Auto-Scaling Metrics

For compute instances, the toolkit references standard CloudWatch metrics including `CPUUtilization`, `NetworkIn`, `NetworkOut`, `DiskReadOps`, `DiskWriteOps`, and `StatusCheckFailed`. These metrics appear in the EC2 section of the metrics reference file and form the basis for the EC2 alarm templates provided in [`references/alarms.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/references/alarms.md).

### Lambda Function Metrics

Serverless monitoring centers on `Invocations`, `Duration`, `Errors`, `Throttles`, `IteratorAge`, `ConcurrentExecutions`, and `UnreservedConcurrentExecutions`. The Lambda section in [`references/metrics.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/references/metrics.md) documents these alongside recommended resolution periods and threshold defaults, such as alerting on error rates exceeding 1% for two consecutive minutes.

### ElastiCache (Redis and Valkey) Metrics

For in-memory caching, the toolkit highlights `EngineCPUUtilization` as the preferred metric over host-level `CPUUtilization`, alongside `CurrConnections`, `NetworkBytesOut`, `DatabaseMemoryUsagePercentage`, and `FreeableMemory`. These metrics help identify memory pressure and connection saturation before they impact application performance.

### RDS and Aurora Metrics

Database observability relies on `CPUUtilization`, `DatabaseConnections`, `ReadLatency`, `WriteLatency`, `FreeableMemory`, `BufferCacheHitRatio`, and `DiskQueueDepth`. The toolkit’s reference documentation notes specific thresholds, such as triggering alarms when `BufferCacheHitRatio` drops below 95% for MySQL instances or when `DiskQueueDepth` exceeds 1 for sustained periods.

### DocumentDB Metrics

MongoDB-compatible workloads use `CPUUtilization`, `DatabaseConnections`, `ReadLatency`, `WriteLatency`, and `FreeableMemory`. The DocumentDB section follows the same structure as RDS, providing consistent dimensional keys for cluster-wide monitoring.

### EKS and ECS Container Metrics

Container orchestration monitoring includes `CPUUtilization` and `MemoryUtilization` at the cluster level, plus `ECSServiceAverageCPUUtilization` and `ECSServiceAverageMemoryUtilization` for Fargate and EC2 launch types. When using FireLens for log routing, the toolkit also surfaces `containerCPUUtilization` and `containerMemoryUtilization` as available dimensions.

### OpenSearch Service Metrics

Search and analytics workloads track `CPUUtilization`, `JVMMemoryPressure`, `FreeStorageSpace`, `ClusterStatusRed`, `SearchLatency`, and `IndexingLatency`. The toolkit specifically calls out `JVMMemoryPressure` as a critical early indicator of cluster instability, recommending alerts at 80% utilization.

### Managed Flink Metrics

Stream processing observability uses `containerCPUUtilization`, `containerMemoryUtilization`, `heapMemoryUtilization`, and `oldGenerationGCTime`. These metrics, documented in the Flink section of [`references/metrics.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/references/metrics.md), help diagnose memory pressure and garbage collection bottlenecks in long-running streaming applications.

### X-Ray and ADOT Trace Metrics

Distributed tracing metrics such as `awsxray_segment_count` and `awsxray_error_rate` are exposed via the ADOT (AWS Distro for OpenTelemetry) collector. The toolkit provides the [`otel-config.yaml`](https://github.com/aws/agent-toolkit-for-aws/blob/main/otel-config.yaml) configuration file in [`skills/core-skills/aws-observability/assets/otel-config.yaml`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/core-skills/aws-observability/assets/otel-config.yaml) to enable these metrics, bridging trace data with CloudWatch metric streams.

### Custom and EMF Metrics

Any user-defined metric emitted via `PutMetricData` or the Embedded Metric Format (EMF) is supported. The toolkit ships an example [`alarm-template.ts`](https://github.com/aws/agent-toolkit-for-aws/blob/main/alarm-template.ts) demonstrating how to publish custom EMF metrics from Lambda functions, allowing applications to emit business-specific telemetry alongside infrastructure metrics.

## Key Source Files in the Observability Skill

The observability capability is organized into specific files that define the metric catalog and implementation patterns:

- **[`skills/core-skills/aws-observability/SKILL.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/core-skills/aws-observability/SKILL.md)** — Defines the skill’s scope and routing table, pointing users to the metric reference files and alarm templates.
- **[`skills/core-skills/aws-observability/references/metrics.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/core-skills/aws-observability/references/metrics.md)** — The canonical catalog enumerating each metric, its namespace, dimensions, and recommended resolution.
- **[`skills/core-skills/aws-observability/references/alarms.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/core-skills/aws-observability/references/alarms.md)** — Builds on the metric list to provide ready-to-use alarm templates for each service with predefined thresholds.
- **[`skills/core-skills/aws-observability/assets/alarm-template.ts`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/core-skills/aws-observability/assets/alarm-template.ts)** — Executable CDK asset demonstrating Lambda monitoring alarms and dashboard creation.
- **[`skills/core-skills/aws-observability/assets/otel-config.yaml`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/core-skills/aws-observability/assets/otel-config.yaml)** — ADOT collector configuration enabling X-Ray tracing and custom EMF metric forwarding.
- **[`skills/core-skills/aws-observability/references/tracing.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/core-skills/aws-observability/references/tracing.md)** — Guidance on enabling X-Ray, configuring ADOT, and interpreting trace-related metadata.

## Practical Examples: Querying and Alerting on Metrics

Below are concrete implementations showing how to interact with the AWS Agent Toolkit’s observability metrics.

### Querying EC2 CPU Metrics

Use the built-in `call_aws` helper or standard AWS CLI to retrieve specific instance metrics:

```bash
call_aws cloudwatch list-metrics \
  --namespace AWS/EC2 \
  --metric-name CPUUtilization \
  --dimensions Name=InstanceId,Value=i-0123456789abcdef0

```

### Creating RDS CPU Alarms with CDK

The [`alarm-template.ts`](https://github.com/aws/agent-toolkit-for-aws/blob/main/alarm-template.ts) asset demonstrates creating threshold-based alarms:

```typescript
import * as cloudwatch from 'aws-cdk-lib/aws-cloudwatch';
import * as rds from 'aws-cdk-lib/aws-rds';

const db = new rds.DatabaseInstance(this, 'MyDB', { /* … */ });

new cloudwatch.Alarm(this, 'CPUAlarm', {
  metric: db.metricCPUUtilization(),
  threshold: 80,
  evaluationPeriods: 3,
  comparisonOperator: cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
});

```

### Emitting Custom EMF Metrics from Lambda

Publish business metrics using the AWS SDK, as shown in the toolkit’s Lambda examples:

```javascript
const { putMetricData } = require('aws-sdk').CloudWatch;

exports.handler = async () => {
  await putMetricData({
    Namespace: 'MyApp',
    MetricData: [{
      MetricName: 'OrdersProcessed',
      Value: 42,
      Unit: 'Count',
    }],
  }).promise();
};

```

### Configuring ADOT for X-Ray Traces

Enable trace-derived metrics using the collector configuration from [`otel-config.yaml`](https://github.com/aws/agent-toolkit-for-aws/blob/main/otel-config.yaml):

```yaml
receivers:
  awsprometheus:
    endpoint: 0.0.0.0:2020
exporters:
  awsxray:
processors:
  batch:
service:
  pipelines:
    traces:
      receivers: [awsprometheus]
      processors: [batch]
      exporters: [awsxray]

```

## Summary

- The AWS Agent Toolkit **aggregates native CloudWatch metrics** rather than generating custom telemetry, covering EC2, Lambda, RDS, ElastiCache, DocumentDB, EKS, ECS, OpenSearch, and Flink.
- Metric definitions and alerting thresholds reside in [`skills/core-skills/aws-observability/references/metrics.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/core-skills/aws-observability/references/metrics.md) and [`alarms.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/alarms.md).
- The toolkit provides **executable CDK assets** ([`alarm-template.ts`](https://github.com/aws/agent-toolkit-for-aws/blob/main/alarm-template.ts)) and **ADOT configurations** ([`otel-config.yaml`](https://github.com/aws/agent-toolkit-for-aws/blob/main/otel-config.yaml)) to operationalize monitoring quickly.
- **X-Ray and ADOT trace metrics** are supported via the OpenTelemetry collector configuration included in the repository.
- **Custom EMF metrics** are supported through Lambda examples demonstrating `PutMetricData` API usage.

## Frequently Asked Questions

### Does the AWS Agent Toolkit generate its own metrics?

No. According to the source code in [`skills/core-skills/aws-observability/SKILL.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/core-skills/aws-observability/SKILL.md), the toolkit exclusively aggregates telemetry that AWS services already emit to CloudWatch, X-Ray, and ADOT. It provides curated reference files that document these native metrics, their dimensions, and recommended alarm thresholds, but does not implement custom metric generation logic.

### How do I access the complete metric catalog for a specific service?

Navigate to [`skills/core-skills/aws-observability/references/metrics.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/core-skills/aws-observability/references/metrics.md) in the repository. This file contains the canonical list of CloudWatch metrics organized by service, including namespace specifications, typical dimensions (such as InstanceId or FunctionName), and suggested resolution periods. For implementation examples, see [`assets/alarm-template.ts`](https://github.com/aws/agent-toolkit-for-aws/blob/main/assets/alarm-template.ts) for CDK patterns or [`references/alarms.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/references/alarms.md) for static alarm definitions.

### Can I use the toolkit to monitor custom business metrics?

Yes. The toolkit supports custom metrics emitted via the CloudWatch `PutMetricData` API or the Embedded Metric Format (EMF). The [`alarm-template.ts`](https://github.com/aws/agent-toolkit-for-aws/blob/main/alarm-template.ts) file includes examples of publishing custom metrics from Lambda functions, while [`assets/otel-config.yaml`](https://github.com/aws/agent-toolkit-for-aws/blob/main/assets/otel-config.yaml) demonstrates how to configure the ADOT collector to forward custom EMF metrics to CloudWatch.

### What is the difference between the alarms.md reference and the alarm-template.ts asset?

The [`references/alarms.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/references/alarms.md) file provides static documentation describing recommended alarm thresholds, evaluation periods, and comparison operators for each metric family. The [`assets/alarm-template.ts`](https://github.com/aws/agent-toolkit-for-aws/blob/main/assets/alarm-template.ts) file is an executable TypeScript asset using AWS CDK that demonstrates programmatically creating these alarms and associated dashboards, allowing you to deploy production-ready monitoring infrastructure as code.