# AWS Observability Skill in the AWS Agent Toolkit: Architecture, Routing, and Implementation

> Explore the AWS Observability skill in Agent Toolkit. Learn how it routes CloudWatch, X-Ray, CloudTrail, and ADOT queries to CLI snippets, CDK examples, and troubleshooting guides.

- Repository: [Amazon Web Services/agent-toolkit-for-aws](https://github.com/aws/agent-toolkit-for-aws)
- Tags: architecture
- Published: 2026-06-28

---

**The AWS Observability skill is a reusable component in the aws/agent-toolkit-for-aws repository that routes user queries about CloudWatch, X-Ray, CloudTrail, and ADOT to specific reference files containing CLI snippets, CDK examples, and troubleshooting guidance.**

The `aws-observability` skill provides a structured approach to building, configuring, and debugging AWS observability implementations. This first-class skill (version `1`) bundles expertise across CloudWatch metrics, logs, alarms, dashboards, X-Ray tracing, CloudTrail audits, and the ADOT collector into a modular reference system that agents consult to provide precise, actionable guidance.

## Skill Architecture and Routing Table

### Intent-Based Routing System

The skill operates through a routing table defined in [`skills/core-skills/aws-observability/SKILL.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/core-skills/aws-observability/SKILL.md) that maps user intents to specific markdown reference files. When a user asks about "writing Log Insights queries" or "creating CloudWatch alarms," the agent matches the intent against the routing table and loads the corresponding reference document from the `references/` directory.

### Execution Flow

The skill follows a four-step execution model: **intent detection**, **reference lookup**, **guidance rendering**, and **optional execution**. When connected to the **AWS MCP server**, the skill can invoke `call_aws` to execute CLI commands directly, providing audit-logged, observable command execution.

## Core Reference Files and Observability Domains

The skill contains eight specialized reference files, each covering a distinct observability domain:

### CloudWatch Alarms ([`references/alarms.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/references/alarms.md))

Located at [`skills/core-skills/aws-observability/references/alarms.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/core-skills/aws-observability/references/alarms.md), this file defines metric alarms, composite alarms, and anomaly-detection alarms. It covers evaluation mechanics, missing-data handling (`treat-missing-data` options), and provides both CDK and CLI implementation patterns.

### Log Insights Queries ([`references/log-insights.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/references/log-insights.md))

The [`references/log-insights.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/references/log-insights.md) file provides the complete query syntax for CloudWatch Logs Insights, including a reusable query library and troubleshooting tips for common query failures.

### Custom Metrics and EMF ([`references/metrics.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/references/metrics.md))

Found in [`references/metrics.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/references/metrics.md), this reference covers custom metric publishing, the **Embedded Metric Format (EMF)**, high-resolution options (1-second granularity), and metric filter creation for CloudWatch Logs.

### Distributed Tracing ([`references/tracing.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/references/tracing.md))

The [`references/tracing.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/references/tracing.md) document explains X-Ray to ADOT (AWS Distro for OpenTelemetry) migration paths, sampling rule configuration, and collector setup for trace ingestion.

### Dashboards ([`references/dashboards.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/references/dashboards.md))

Located at [`references/dashboards.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/references/dashboards.md), this file details widget types, cross-account sharing configurations, dynamic label syntax, and dashboard template usage.

### Synthetic Canaries ([`references/synthetics.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/references/synthetics.md))

The [`references/synthetics.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/references/synthetics.md) reference covers CloudWatch Synthetics canary runtime constraints, VPC networking requirements, and common failure modes for canary scripts.

### CloudTrail Auditing ([`references/cloudtrail.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/references/cloudtrail.md))

Found in [`references/cloudtrail.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/references/cloudtrail.md), this reference describes event types (management, data, insight events), S3/Athena query patterns for log analysis, and audit-trail best practices.

### Troubleshooting ([`references/troubleshooting.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/references/troubleshooting.md))

The [`references/troubleshooting.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/references/troubleshooting.md) file catalogs the five most common observability failures and provides step-by-step remediation procedures for agent-assisted debugging.

## Reusable Assets and Templates

### CDK Alarm Template

The [`skills/core-skills/aws-observability/assets/alarm-template.ts`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/core-skills/aws-observability/assets/alarm-template.ts) file provides a TypeScript CDK starter that creates a Lambda-monitoring alarm set paired with a CloudWatch dashboard.

### ADOT Collector Configuration

Located at [`assets/otel-config.yaml`](https://github.com/aws/agent-toolkit-for-aws/blob/main/assets/otel-config.yaml), this file contains the default ADOT collector configuration for sending X-Ray traces and EMF metrics to CloudWatch.

## Execution Model and MCP Server Integration

While the AWS Observability skill functions with standard AWS CLI and Boto3 calls, it works best when paired with the **AWS MCP server**. This integration enables sandboxed command execution where the skill can invoke `call_aws` to run suggested CLI or CloudFormation commands directly, returning observable, audit-logged results.

## Practical Examples: Creating CloudWatch Alarms

### CLI Implementation

For creating a Lambda error-rate alarm using the AWS CLI:

```bash
aws cloudwatch put-metric-alarm --alarm-name MyFunc-ErrorRate \
  --metrics '[
    {"Id":"errors","MetricStat":{"Metric":{"Namespace":"AWS/Lambda","MetricName":"Errors","Dimensions":[{"Name":"FunctionName","Value":"MyFunc"}]},"Period":60,"Stat":"Sum"},"ReturnData":false},
    {"Id":"invocations","MetricStat":{"Metric":{"Namespace":"AWS/Lambda","MetricName":"Invocations","Dimensions":[{"Name":"FunctionName","Value":"MyFunc"}]},"Period":60,"Stat":"Sum"},"ReturnData":false},
    {"Id":"error_rate","Expression":"IF(invocations > 0, errors * 100 / invocations, 0)","Label":"Error Rate %"}
  ]' \
  --threshold 5 \
  --comparison-operator GreaterThanThreshold \
  --evaluation-periods 3 \
  --datapoints-to-alarm 2 \
  --treat-missing-data notBreaching

```

*Source:* [`references/alarms.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/references/alarms.md)

### CDK Implementation

The same alarm implemented in AWS CDK TypeScript:

```typescript
import { Alarm, ComparisonOperator, MathExpression, TreatMissingData } from 'aws-cdk-lib/aws-cloudwatch';
import { Duration } from 'aws-cdk-lib';

const errorRateAlarm = new Alarm(this, 'ErrorRateAlarm', {
  metric: new MathExpression({
    expression: 'IF(invocations > 0, errors * 100 / invocations, 0)',
    usingMetrics: {
      errors: fn.metricErrors({ period: Duration.minutes(1) }),
      invocations: fn.metricInvocations({ period: Duration.minutes(1) }),
    },
  }),
  threshold: 5,
  evaluationPeriods: 3,
  datapointsToAlarm: 2,
  comparisonOperator: ComparisonOperator.GREATER_THAN_THRESHOLD,
  treatMissingData: TreatMissingData.NOT_BREACHING,
});

```

*Source:* [`references/alarms.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/references/alarms.md)

## Summary

- The **AWS Observability skill** is located at `skills/core-skills/aws-observability/` in the `aws/agent-toolkit-for-aws` repository and provides modular expertise for CloudWatch, X-Ray, CloudTrail, and ADOT.
- An **intent-based routing table** in [`SKILL.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/SKILL.md) maps user queries to specific reference files including [`alarms.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/alarms.md), [`log-insights.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/log-insights.md), [`metrics.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/metrics.md), and [`tracing.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/tracing.md).
- **Practical assets** include [`alarm-template.ts`](https://github.com/aws/agent-toolkit-for-aws/blob/main/alarm-template.ts) for CDK implementations and [`otel-config.yaml`](https://github.com/aws/agent-toolkit-for-aws/blob/main/otel-config.yaml) for ADOT collector configuration.
- The skill supports **optional MCP server integration** for sandboxed, audit-logged command execution via `call_aws`.
- All reference files contain production-ready code snippets for both **AWS CLI and CDK** implementations.

## Frequently Asked Questions

### What services does the AWS Observability skill cover?

The skill covers **CloudWatch** (metrics, logs, alarms, dashboards, and EMF), **X-Ray** distributed tracing, **CloudTrail** audit logging, and the **ADOT** (AWS Distro for OpenTelemetry) collector. Each service domain has a dedicated reference file in the `references/` directory.

### How does the skill route user queries to the correct documentation?

The skill uses a routing table defined in [`SKILL.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/SKILL.md) that maps specific intents—such as "writing Log Insights queries" or "configuring alarms"—to corresponding markdown files. When a user asks a question, the agent matches the intent and loads the appropriate reference document containing the concrete guidance.

### What are the reusable assets included in the AWS Observability skill?

The skill includes two primary assets: [`alarm-template.ts`](https://github.com/aws/agent-toolkit-for-aws/blob/main/alarm-template.ts), a CDK TypeScript starter for Lambda monitoring alarms and dashboards, and [`otel-config.yaml`](https://github.com/aws/agent-toolkit-for-aws/blob/main/otel-config.yaml), a default configuration file for the ADOT collector that sends traces to X-Ray and metrics to CloudWatch.

### Can the AWS Observability skill execute commands directly?

Yes, when paired with the **AWS MCP server**, the skill can invoke `call_aws` to execute CLI commands or CloudFormation deployments directly. This provides sandboxed, audit-logged execution, though the skill also supports traditional AWS CLI and Boto3 usage patterns.