# Best Practices for Using the Observability Skill in AWS Agent Toolkit > Master AWS Agent Toolkit's observability skill. Follow best practices like using SKILL.md, MCP server commands, and composite alarms to reduce alert fatigue and improve debugging. - Repository: [Amazon Web Services/agent-toolkit-for-aws](https://github.com/aws/agent-toolkit-for-aws) - Tags: best-practices - Published: 2026-06-28 --- **Start every interaction with the SKILL.md entry point, execute commands through the AWS MCP server for sandboxed auditing, and rely on composite alarms with explicit TreatMissingData settings to eliminate alert fatigue.** The observability skill in AWS agent toolkit provides a curated guide for building, configuring, and optimizing monitoring across CloudWatch, X-Ray, and CloudTrail. Located in the `aws/agent-toolkit-for-aws` repository, this skill consolidates reference files for metrics, alarms, dashboards, and tracing into a single navigable structure. Following these best practices ensures low-noise alerting, cost-effective metric storage, and unified telemetry pipelines. ## Start with the SKILL.md Entry Point Always begin with [`skills/core-skills/aws-observability/SKILL.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/core-skills/aws-observability/SKILL.md) to understand routing and reference file organization. This file serves as the skill's index, directing you to specific sub-topics like alarms, dashboards, or tracing. Starting here guarantees you reference the correct context before executing any AWS commands. When available, run commands through the **AWS MCP server** (`call_aws` tool) rather than direct CLI calls. The MCP server provides sandboxed execution, audit logging, and built-in observability, falling back to AWS CLI only when the server is unreachable. This pattern is documented in SKILL.md at line 13. ## Configure Alarms for Precision and Noise Reduction The [`alarms.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/alarms.md) reference file contains specific patterns for robust CloudWatch alerting that eliminate false positives. ### Composite and Metric Alarm Patterns Choose **composite alarms** or **metric alarms** instead of raw metric math for EC2 or Auto Scaling actions. Configure **M-of-N** logic using `DatapointsToAlarm` set to at least 2-3 datapoints to prevent flapping. Always explicitly set `TreatMissingData` to avoid the default `missing` state, which generates unwanted INSUFFICIENT_DATA notifications. Add an **ActionsSuppressor** alarm to silence notifications during CI/CD deployment windows. ### Latency and Error-Rate Monitoring Use **p99** (or p90) statistics for latency thresholds, never `Average`. For error rates, build metric math expressions that divide `Errors` by `Invocations` to prevent single-error spikes from triggering alerts. This approach surfaces tail latency that averages mask while maintaining stable alerting thresholds. ```bash aws cloudwatch put-metric-alarm \ --alarm-name MyFunction-ErrorRate \ --metrics '[ {"Id":"errors","MetricStat":{"Metric":{"Namespace":"AWS/Lambda","MetricName":"Errors","Dimensions":[{"Name":"FunctionName","Value":"MyFunction"}]},"Period":60,"Stat":"Sum"},"ReturnData":false}, {"Id":"invocations","MetricStat":{"Metric":{"Namespace":"AWS/Lambda","MetricName":"Invocations","Dimensions":[{"Name":"FunctionName","Value":"MyFunction"}]},"Period":60,"Stat":"Sum"},"ReturnData":false}, {"Id":"error_rate","Expression":"IF(invocations > 0, errors * 100 / invocations, 0)","Label":"Error Rate %"} ]' \ --threshold 5 \ --comparison-operator GreaterThanThreshold \ --evaluation-periods 3 \ --datapoints-to-alarm 2 \ --treat-missing-data notBreaching ``` ### Anomaly Detection Bands Deploy **ANOMALY_DETECTION_BAND** alarms when baselines are unknown or workloads exhibit high seasonality. This feature provides automatic adaptive thresholds with low false-positive rates, ideal for variable traffic patterns. ```yaml Resources: AnomalyDetector: Type: AWS::CloudWatch::AnomalyDetector Properties: MetricName: Invocations Namespace: AWS/Lambda Stat: Sum AnomalyAlarm: Type: AWS::CloudWatch::Alarm Properties: ComparisonOperator: LessThanLowerOrGreaterThanUpperThreshold EvaluationPeriods: 1 Metrics: - Expression: ANOMALY_DETECTION_BAND(m1, 2) Id: ad1 - Id: m1 MetricStat: Metric: MetricName: Invocations Namespace: AWS/Lambda Period: 86400 Stat: Sum ThresholdMetricId: ad1 TreatMissingData: breaching ``` ## Build Reusable Dashboards and Queries ### Dashboard Templates and Widgets Create reusable **dashboard templates** using the provided [`assets/alarm-template.ts`](https://github.com/aws/agent-toolkit-for-aws/blob/main/assets/alarm-template.ts) CDK TypeScript file. Include **cross-account/region widgets** when monitoring multiple services, and use **dynamic labels** (e.g., `{{function}}`) for quick drill-down capabilities. These practices ensure consistent visibility across teams and reduce manual maintenance overhead. ```typescript import { Alarm, CompositeAlarm, AlarmRule, Duration } from 'aws-cdk-lib/aws-cloudwatch'; import { Function } from 'aws-cdk-lib/aws-lambda'; const errorRateAlarm = new Alarm(this, 'ErrorRate', { metric: fn.metricErrors({ period: Duration.minutes(1) }) .addMetric(fn.metricInvocations({ period: Duration.minutes(1) }) .with({ statistic: 'Sum' })) .createMathExpression('IF(invocations > 0, errors * 100 / invocations, 0)'), threshold: 5, evaluationPeriods: 3, datapointsToAlarm: 2, comparisonOperator: ComparisonOperator.GREATER_THAN_THRESHOLD, treatMissingData: TreatMissingData.NOT_BREACHING, }); const latencyAlarm = new Alarm(this, 'LatencyP99', { metric: fn.metricDuration({ statistic: 'p99', period: Duration.minutes(1) }), threshold: 3000, evaluationPeriods: 3, datapointsToAlarm: 2, comparisonOperator: ComparisonOperator.GREATER_THAN_THRESHOLD, treatMissingData: TreatMissingData.NOT_BREACHING, }); const serviceHealth = new CompositeAlarm(this, 'ServiceHealth', { alarmRule: AlarmRule.anyOf( AlarmRule.fromAlarm(errorRateAlarm, AlarmState.ALARM), AlarmRule.fromAlarm(latencyAlarm, AlarmState.ALARM) ), }); ``` ### Log Insights Query Library Store common queries in a versioned library and reference them from the skill. Prefer **field extraction** (`parse`) over regex for performance, as seen in [`log-insights.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/log-insights.md). Reusable queries speed up root-cause analysis and keep CloudWatch Logs Insights costs low by avoiding repeated ad-hoc parsing. ```sql fields @timestamp, @message | filter @message like /START/ | parse @message "*START*" as start | stats count() as cold_starts by bin(5m) | sort @timestamp desc ``` ## Instrument with Modern Telemetry Standards ### Custom Metrics and EMF Push custom metrics via **PutMetricData** or **Embedded Metric Format (EMF)** JSON. Keep high-resolution metrics (10-second granularity) only for truly fast-changing data, as high-resolution is expensive. EMF lets you emit multiple related metrics in a single call, reducing API overhead. ### X-Ray to ADOT Migration Migrate X-Ray instrumentation to **AWS Distro for OpenTelemetry (ADOT)** for unified OpenTelemetry pipelines. Use the supplied [`assets/otel-config.yaml`](https://github.com/aws/agent-toolkit-for-aws/blob/main/assets/otel-config.yaml) as a baseline configuration, customizing sampling rules for production workloads. ADOT enables consolidated metrics and traces while reducing operational overhead and aligning with industry standards. ```yaml receivers: awsxray: endpoint: 0.0.0.0:2000 exporters: awsemf: namespace: "MyApp" service: pipelines: traces: receivers: [awsxray] exporters: [awsemf] ``` ## Secure and Audit with CloudTrail Enable **EventBridge** or Athena queries on CloudTrail logs for security and compliance dashboards, as documented in [`cloudtrail.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/cloudtrail.md). This provides a single source of truth for which principals performed specific AWS actions, essential for forensic analysis and compliance reporting. ## Validate End-to-End with Synthetic Canaries Use **synthetic canaries** to validate end-to-end service health, keeping runtime dependencies minimal. Follow the "Common failures" table in [`synthetics.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/synthetics.md) to avoid networking or VPC misconfigurations. Canaries detect outage-level issues that metrics alone might miss, providing early warning for user-facing problems. ## Cross-Account Observability Strategy Deploy a **monitoring account** that watches alarms in source accounts using CloudWatch Observability Access Manager (OAM). Note that alarms and the monitoring account must reside in the **same region** for this configuration to work. This centralizes alerting for large organizations while respecting account isolation boundaries. ## Summary - **Start with SKILL.md** in `skills/core-skills/aws-observability/` to ensure correct routing and context. - **Use the AWS MCP server** for sandboxed execution and audit logging when available. - **Configure composite alarms** with explicit `TreatMissingData` and M-of-N (2-3) datapoints to reduce alert fatigue. - **Monitor p99 latency** and error rates via metric math, not averages or raw counters. - **Adopt ADOT** over native X-Ray and use EMF for efficient custom metric ingestion. - **Centralize cross-account monitoring** with OAM, ensuring regional alignment between monitoring and source accounts. ## Frequently Asked Questions ### How do I prevent alarm flapping during deployments? Add an **ActionsSuppressor** alarm to your primary alarm configuration. This suppressor silences notifications during CI/CD windows, preventing deployment-induced noise from triggering pages while maintaining monitoring coverage. ### What is the recommended statistic for Lambda latency alarms? Always use **p99** (or p90) for latency thresholds. The `Average` statistic masks tail latency issues that affect user experience, while p99 surfaces the worst-case performance that actually matters for SLA compliance. ### Should I use high-resolution metrics for all custom data? No. Reserve **high-resolution metrics** (10-second granularity) only for fast-changing data that requires immediate visibility. Standard 60-second granularity is sufficient for most workloads and significantly reduces CloudWatch costs. ### How do I enable cross-account alarm visibility? Deploy a **monitoring account** using CloudWatch Observability Access Manager (OAM) to link source accounts. Ensure the monitoring account and the alarms being monitored are in the **same region**, as cross-region monitoring is not supported for this feature.