Best Practices for Using the Observability Skill in AWS Agent Toolkit

Start every interaction with the SKILL.md entry point, execute commands through the AWS MCP server for sandboxed auditing, and rely on composite alarms with explicit TreatMissingData settings to eliminate alert fatigue.

The observability skill in AWS agent toolkit provides a curated guide for building, configuring, and optimizing monitoring across CloudWatch, X-Ray, and CloudTrail. Located in the aws/agent-toolkit-for-aws repository, this skill consolidates reference files for metrics, alarms, dashboards, and tracing into a single navigable structure. Following these best practices ensures low-noise alerting, cost-effective metric storage, and unified telemetry pipelines.

Start with the SKILL.md Entry Point

Always begin with skills/core-skills/aws-observability/SKILL.md to understand routing and reference file organization. This file serves as the skill's index, directing you to specific sub-topics like alarms, dashboards, or tracing. Starting here guarantees you reference the correct context before executing any AWS commands.

When available, run commands through the AWS MCP server (call_aws tool) rather than direct CLI calls. The MCP server provides sandboxed execution, audit logging, and built-in observability, falling back to AWS CLI only when the server is unreachable. This pattern is documented in SKILL.md at line 13.

Configure Alarms for Precision and Noise Reduction

The alarms.md reference file contains specific patterns for robust CloudWatch alerting that eliminate false positives.

Composite and Metric Alarm Patterns

Choose composite alarms or metric alarms instead of raw metric math for EC2 or Auto Scaling actions. Configure M-of-N logic using DatapointsToAlarm set to at least 2-3 datapoints to prevent flapping. Always explicitly set TreatMissingData to avoid the default missing state, which generates unwanted INSUFFICIENT_DATA notifications. Add an ActionsSuppressor alarm to silence notifications during CI/CD deployment windows.

Latency and Error-Rate Monitoring

Use p99 (or p90) statistics for latency thresholds, never Average. For error rates, build metric math expressions that divide Errors by Invocations to prevent single-error spikes from triggering alerts. This approach surfaces tail latency that averages mask while maintaining stable alerting thresholds.

aws cloudwatch put-metric-alarm \
  --alarm-name MyFunction-ErrorRate \
  --metrics '[
    {"Id":"errors","MetricStat":{"Metric":{"Namespace":"AWS/Lambda","MetricName":"Errors","Dimensions":[{"Name":"FunctionName","Value":"MyFunction"}]},"Period":60,"Stat":"Sum"},"ReturnData":false},
    {"Id":"invocations","MetricStat":{"Metric":{"Namespace":"AWS/Lambda","MetricName":"Invocations","Dimensions":[{"Name":"FunctionName","Value":"MyFunction"}]},"Period":60,"Stat":"Sum"},"ReturnData":false},
    {"Id":"error_rate","Expression":"IF(invocations > 0, errors * 100 / invocations, 0)","Label":"Error Rate %"}
  ]' \
  --threshold 5 \
  --comparison-operator GreaterThanThreshold \
  --evaluation-periods 3 \
  --datapoints-to-alarm 2 \
  --treat-missing-data notBreaching

Anomaly Detection Bands

Deploy ANOMALY_DETECTION_BAND alarms when baselines are unknown or workloads exhibit high seasonality. This feature provides automatic adaptive thresholds with low false-positive rates, ideal for variable traffic patterns.

Resources:
  AnomalyDetector:
    Type: AWS::CloudWatch::AnomalyDetector
    Properties:
      MetricName: Invocations
      Namespace: AWS/Lambda
      Stat: Sum

  AnomalyAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      ComparisonOperator: LessThanLowerOrGreaterThanUpperThreshold
      EvaluationPeriods: 1
      Metrics:
        - Expression: ANOMALY_DETECTION_BAND(m1, 2)
          Id: ad1
        - Id: m1
          MetricStat:
            Metric:
              MetricName: Invocations
              Namespace: AWS/Lambda
            Period: 86400
            Stat: Sum
      ThresholdMetricId: ad1
      TreatMissingData: breaching

Build Reusable Dashboards and Queries

Dashboard Templates and Widgets

Create reusable dashboard templates using the provided assets/alarm-template.ts CDK TypeScript file. Include cross-account/region widgets when monitoring multiple services, and use dynamic labels (e.g., {{function}}) for quick drill-down capabilities. These practices ensure consistent visibility across teams and reduce manual maintenance overhead.

import { Alarm, CompositeAlarm, AlarmRule, Duration } from 'aws-cdk-lib/aws-cloudwatch';
import { Function } from 'aws-cdk-lib/aws-lambda';

const errorRateAlarm = new Alarm(this, 'ErrorRate', {
  metric: fn.metricErrors({ period: Duration.minutes(1) })
    .addMetric(fn.metricInvocations({ period: Duration.minutes(1) })
    .with({ statistic: 'Sum' }))
    .createMathExpression('IF(invocations > 0, errors * 100 / invocations, 0)'),
  threshold: 5,
  evaluationPeriods: 3,
  datapointsToAlarm: 2,
  comparisonOperator: ComparisonOperator.GREATER_THAN_THRESHOLD,
  treatMissingData: TreatMissingData.NOT_BREACHING,
});

const latencyAlarm = new Alarm(this, 'LatencyP99', {
  metric: fn.metricDuration({ statistic: 'p99', period: Duration.minutes(1) }),
  threshold: 3000,
  evaluationPeriods: 3,
  datapointsToAlarm: 2,
  comparisonOperator: ComparisonOperator.GREATER_THAN_THRESHOLD,
  treatMissingData: TreatMissingData.NOT_BREACHING,
});

const serviceHealth = new CompositeAlarm(this, 'ServiceHealth', {
  alarmRule: AlarmRule.anyOf(
    AlarmRule.fromAlarm(errorRateAlarm, AlarmState.ALARM),
    AlarmRule.fromAlarm(latencyAlarm, AlarmState.ALARM)
  ),
});

Log Insights Query Library

Store common queries in a versioned library and reference them from the skill. Prefer field extraction (parse) over regex for performance, as seen in log-insights.md. Reusable queries speed up root-cause analysis and keep CloudWatch Logs Insights costs low by avoiding repeated ad-hoc parsing.

fields @timestamp, @message
| filter @message like /START/
| parse @message "*START*" as start
| stats count() as cold_starts by bin(5m)
| sort @timestamp desc

Instrument with Modern Telemetry Standards

Custom Metrics and EMF

Push custom metrics via PutMetricData or Embedded Metric Format (EMF) JSON. Keep high-resolution metrics (10-second granularity) only for truly fast-changing data, as high-resolution is expensive. EMF lets you emit multiple related metrics in a single call, reducing API overhead.

X-Ray to ADOT Migration

Migrate X-Ray instrumentation to AWS Distro for OpenTelemetry (ADOT) for unified OpenTelemetry pipelines. Use the supplied assets/otel-config.yaml as a baseline configuration, customizing sampling rules for production workloads. ADOT enables consolidated metrics and traces while reducing operational overhead and aligning with industry standards.

receivers:
  awsxray:
    endpoint: 0.0.0.0:2000
exporters:
  awsemf:
    namespace: "MyApp"
service:
  pipelines:
    traces:
      receivers: [awsxray]
      exporters: [awsemf]

Secure and Audit with CloudTrail

Enable EventBridge or Athena queries on CloudTrail logs for security and compliance dashboards, as documented in cloudtrail.md. This provides a single source of truth for which principals performed specific AWS actions, essential for forensic analysis and compliance reporting.

Validate End-to-End with Synthetic Canaries

Use synthetic canaries to validate end-to-end service health, keeping runtime dependencies minimal. Follow the "Common failures" table in synthetics.md to avoid networking or VPC misconfigurations. Canaries detect outage-level issues that metrics alone might miss, providing early warning for user-facing problems.

Cross-Account Observability Strategy

Deploy a monitoring account that watches alarms in source accounts using CloudWatch Observability Access Manager (OAM). Note that alarms and the monitoring account must reside in the same region for this configuration to work. This centralizes alerting for large organizations while respecting account isolation boundaries.

Summary

• Start with SKILL.md in skills/core-skills/aws-observability/ to ensure correct routing and context.
• Use the AWS MCP server for sandboxed execution and audit logging when available.
• Configure composite alarms with explicit TreatMissingData and M-of-N (2-3) datapoints to reduce alert fatigue.
• Monitor p99 latency and error rates via metric math, not averages or raw counters.
• Adopt ADOT over native X-Ray and use EMF for efficient custom metric ingestion.
• Centralize cross-account monitoring with OAM, ensuring regional alignment between monitoring and source accounts.

Frequently Asked Questions

How do I prevent alarm flapping during deployments?

Add an ActionsSuppressor alarm to your primary alarm configuration. This suppressor silences notifications during CI/CD windows, preventing deployment-induced noise from triggering pages while maintaining monitoring coverage.

What is the recommended statistic for Lambda latency alarms?

Always use p99 (or p90) for latency thresholds. The Average statistic masks tail latency issues that affect user experience, while p99 surfaces the worst-case performance that actually matters for SLA compliance.

Should I use high-resolution metrics for all custom data?

No. Reserve high-resolution metrics (10-second granularity) only for fast-changing data that requires immediate visibility. Standard 60-second granularity is sufficient for most workloads and significantly reduces CloudWatch costs.

How do I enable cross-account alarm visibility?

Deploy a monitoring account using CloudWatch Observability Access Manager (OAM) to link source accounts. Ensure the monitoring account and the alarms being monitored are in the same region, as cross-region monitoring is not supported for this feature.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}