AWS Observability Skill in the AWS Agent Toolkit: Architecture, Routing, and Implementation

The AWS Observability skill is a reusable component in the aws/agent-toolkit-for-aws repository that routes user queries about CloudWatch, X-Ray, CloudTrail, and ADOT to specific reference files containing CLI snippets, CDK examples, and troubleshooting guidance.

The aws-observability skill provides a structured approach to building, configuring, and debugging AWS observability implementations. This first-class skill (version 1) bundles expertise across CloudWatch metrics, logs, alarms, dashboards, X-Ray tracing, CloudTrail audits, and the ADOT collector into a modular reference system that agents consult to provide precise, actionable guidance.

Skill Architecture and Routing Table

Intent-Based Routing System

The skill operates through a routing table defined in skills/core-skills/aws-observability/SKILL.md that maps user intents to specific markdown reference files. When a user asks about "writing Log Insights queries" or "creating CloudWatch alarms," the agent matches the intent against the routing table and loads the corresponding reference document from the references/ directory.

Execution Flow

The skill follows a four-step execution model: intent detection, reference lookup, guidance rendering, and optional execution. When connected to the AWS MCP server, the skill can invoke call_aws to execute CLI commands directly, providing audit-logged, observable command execution.

Core Reference Files and Observability Domains

The skill contains eight specialized reference files, each covering a distinct observability domain:

CloudWatch Alarms (references/alarms.md)

Located at skills/core-skills/aws-observability/references/alarms.md, this file defines metric alarms, composite alarms, and anomaly-detection alarms. It covers evaluation mechanics, missing-data handling (treat-missing-data options), and provides both CDK and CLI implementation patterns.

Log Insights Queries (references/log-insights.md)

The references/log-insights.md file provides the complete query syntax for CloudWatch Logs Insights, including a reusable query library and troubleshooting tips for common query failures.

Custom Metrics and EMF (references/metrics.md)

Found in references/metrics.md, this reference covers custom metric publishing, the Embedded Metric Format (EMF), high-resolution options (1-second granularity), and metric filter creation for CloudWatch Logs.

Distributed Tracing (references/tracing.md)

The references/tracing.md document explains X-Ray to ADOT (AWS Distro for OpenTelemetry) migration paths, sampling rule configuration, and collector setup for trace ingestion.

Dashboards (references/dashboards.md)

Located at references/dashboards.md, this file details widget types, cross-account sharing configurations, dynamic label syntax, and dashboard template usage.

Synthetic Canaries (references/synthetics.md)

The references/synthetics.md reference covers CloudWatch Synthetics canary runtime constraints, VPC networking requirements, and common failure modes for canary scripts.

CloudTrail Auditing (references/cloudtrail.md)

Found in references/cloudtrail.md, this reference describes event types (management, data, insight events), S3/Athena query patterns for log analysis, and audit-trail best practices.

Troubleshooting (references/troubleshooting.md)

The references/troubleshooting.md file catalogs the five most common observability failures and provides step-by-step remediation procedures for agent-assisted debugging.

Reusable Assets and Templates

CDK Alarm Template

The skills/core-skills/aws-observability/assets/alarm-template.ts file provides a TypeScript CDK starter that creates a Lambda-monitoring alarm set paired with a CloudWatch dashboard.

ADOT Collector Configuration

Located at assets/otel-config.yaml, this file contains the default ADOT collector configuration for sending X-Ray traces and EMF metrics to CloudWatch.

Execution Model and MCP Server Integration

While the AWS Observability skill functions with standard AWS CLI and Boto3 calls, it works best when paired with the AWS MCP server. This integration enables sandboxed command execution where the skill can invoke call_aws to run suggested CLI or CloudFormation commands directly, returning observable, audit-logged results.

Practical Examples: Creating CloudWatch Alarms

CLI Implementation

For creating a Lambda error-rate alarm using the AWS CLI:

aws cloudwatch put-metric-alarm --alarm-name MyFunc-ErrorRate \
  --metrics '[
    {"Id":"errors","MetricStat":{"Metric":{"Namespace":"AWS/Lambda","MetricName":"Errors","Dimensions":[{"Name":"FunctionName","Value":"MyFunc"}]},"Period":60,"Stat":"Sum"},"ReturnData":false},
    {"Id":"invocations","MetricStat":{"Metric":{"Namespace":"AWS/Lambda","MetricName":"Invocations","Dimensions":[{"Name":"FunctionName","Value":"MyFunc"}]},"Period":60,"Stat":"Sum"},"ReturnData":false},
    {"Id":"error_rate","Expression":"IF(invocations > 0, errors * 100 / invocations, 0)","Label":"Error Rate %"}
  ]' \
  --threshold 5 \
  --comparison-operator GreaterThanThreshold \
  --evaluation-periods 3 \
  --datapoints-to-alarm 2 \
  --treat-missing-data notBreaching

Source: references/alarms.md

CDK Implementation

The same alarm implemented in AWS CDK TypeScript:

import { Alarm, ComparisonOperator, MathExpression, TreatMissingData } from 'aws-cdk-lib/aws-cloudwatch';
import { Duration } from 'aws-cdk-lib';

const errorRateAlarm = new Alarm(this, 'ErrorRateAlarm', {
  metric: new MathExpression({
    expression: 'IF(invocations > 0, errors * 100 / invocations, 0)',
    usingMetrics: {
      errors: fn.metricErrors({ period: Duration.minutes(1) }),
      invocations: fn.metricInvocations({ period: Duration.minutes(1) }),
    },
  }),
  threshold: 5,
  evaluationPeriods: 3,
  datapointsToAlarm: 2,
  comparisonOperator: ComparisonOperator.GREATER_THAN_THRESHOLD,
  treatMissingData: TreatMissingData.NOT_BREACHING,
});

Source: references/alarms.md

Summary

• The AWS Observability skill is located at skills/core-skills/aws-observability/ in the aws/agent-toolkit-for-aws repository and provides modular expertise for CloudWatch, X-Ray, CloudTrail, and ADOT.
• An intent-based routing table in SKILL.md maps user queries to specific reference files including alarms.md, log-insights.md, metrics.md, and tracing.md.
• Practical assets include alarm-template.ts for CDK implementations and otel-config.yaml for ADOT collector configuration.
• The skill supports optional MCP server integration for sandboxed, audit-logged command execution via call_aws.
• All reference files contain production-ready code snippets for both AWS CLI and CDK implementations.

Frequently Asked Questions

What services does the AWS Observability skill cover?

The skill covers CloudWatch (metrics, logs, alarms, dashboards, and EMF), X-Ray distributed tracing, CloudTrail audit logging, and the ADOT (AWS Distro for OpenTelemetry) collector. Each service domain has a dedicated reference file in the references/ directory.

How does the skill route user queries to the correct documentation?

The skill uses a routing table defined in SKILL.md that maps specific intents—such as "writing Log Insights queries" or "configuring alarms"—to corresponding markdown files. When a user asks a question, the agent matches the intent and loads the appropriate reference document containing the concrete guidance.

What are the reusable assets included in the AWS Observability skill?

The skill includes two primary assets: alarm-template.ts, a CDK TypeScript starter for Lambda monitoring alarms and dashboards, and otel-config.yaml, a default configuration file for the ADOT collector that sends traces to X-Ray and metrics to CloudWatch.

Can the AWS Observability skill execute commands directly?

Yes, when paired with the AWS MCP server, the skill can invoke call_aws to execute CLI commands or CloudFormation deployments directly. This provides sandboxed, audit-logged execution, though the skill also supports traditional AWS CLI and Boto3 usage patterns.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}