# How to Troubleshoot Issues with the Observability Skill in AWS Agent Toolkit

> Troubleshoot AWS Agent Toolkit observability skill failures. Learn to diagnose CloudWatch, X-Ray, and CloudTrail issues with structured guidance and AWS CLI commands.

- Repository: [Amazon Web Services/agent-toolkit-for-aws](https://github.com/aws/agent-toolkit-for-aws)
- Tags: how-to-guide
- Published: 2026-06-28

---

**The AWS Observability Skill diagnoses CloudWatch, X-Ray, and CloudTrail failures by routing through a structured matrix in [`troubleshooting.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/troubleshooting.md) and validating hypotheses with specific AWS CLI commands.**

The `aws-observability` skill in the `aws/agent-toolkit-for-aws` repository helps agents build, configure, and optimize AWS monitoring stacks. When you need to troubleshoot issues with the observability skill in AWS Agent Toolkit, the system follows a systematic workflow defined in [`skills/core-skills/aws-observability/SKILL.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/core-skills/aws-observability/SKILL.md) and references detailed symptom-cause-fix tables. This guide reproduces that exact diagnostic logic so you can resolve alarm, log, metric, tracing, and audit issues efficiently.

## Understanding the Troubleshooting Architecture

The skill implements a five-step diagnostic flow that mirrors how AWS support engineers approach observability failures.

First, the **routing table** in [`skills/core-skills/aws-observability/SKILL.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/core-skills/aws-observability/SKILL.md) (lines 17-27) identifies the specific service category—whether the issue involves CloudWatch alarms, Log Insights queries, custom metrics, X-Ray traces, or CloudTrail events.

Next, the skill presents the **"Top 5" fixes**—a concise checklist covering the most common failure patterns located at the opening section of [`skills/core-skills/aws-observability/references/troubleshooting.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/core-skills/aws-observability/references/troubleshooting.md) (lines 5-12).

Then it drills down into **symptom-cause-fix matrices** (e.g., lines 19-25 for alarms, lines 46-55 for missing logs, lines 92-100 for X-Ray traces). Each matrix entry maps a specific symptom to its root cause and corrective action.

Finally, the skill guides users to run **validation CLI commands** to confirm the hypothesis before iterating to the next potential cause if the fix fails.

## Diagnosing Common Failure Patterns

The troubleshooting reference file organizes solutions by service category. Below are the exact failure patterns the skill recognizes and how to resolve them.

### Fixing CloudWatch Alarms Stuck in INSUFFICIENT_DATA

An alarm stuck in `INSUFFICIENT_DATA` usually indicates a **namespace or dimension mismatch**. According to [`troubleshooting.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/troubleshooting.md) (lines 21-23), the metric identifier in the alarm must match the publishing source exactly—including case sensitivity (`AWS/Lambda` vs. `aws/lambda`).

If the alarm fails to trigger despite breaching thresholds, verify the **M-of-N configuration** and **missing-data treatment**. The skill recommends lowering the **M** (evaluation periods to breach), raising the **N** (total periods), or switching missing-data treatment from `missing` to `notBreaching` (lines 30-32).

### Resolving Missing CloudWatch Logs

When logs fail to appear, check three components in sequence: the **log group existence**, **IAM permissions**, and **retention settings**. The skill directs users to verify the log group exists (creating it if necessary), ensure the resource has the `AWSLambdaBasicExecutionRole` or equivalent, and confirm the retention period hasn't expired (lines 50-55).

### Repairing Empty Log Insights Queries

Empty query results typically stem from **incorrect time ranges**, **wrong log group specifications**, or **Infrequent Access (IA) class log groups** used with pattern commands. Expand the query time window, confirm the exact log group name, and avoid querying IA class groups for `pattern` commands (lines 60-63).

### Preventing Custom Metric Disappearance

Custom metrics become **inactive** if they receive no data points for two weeks or more. To prevent disappearance, emit at least one datapoint weekly using `put-metric-data`, or query with `get-metric-statistics` using exact identifiers including namespace and dimensions (lines 74-77).

### Recovering Missing X-Ray Traces

Missing traces usually indicate **disabled active tracing**, **restrictive sampling rules**, or **uninstrumented downstream services**. Enable active tracing on Lambda functions or API Gateway stages, increase the `ReservoirCount` or `FixedRate` in X-Ray sampling rules, and add AWS Distro for OpenTelemetry (ADOT) instrumentation to downstream calls (lines 95-100).

### Locating CloudTrail Data Events

When CloudTrail events are missing, verify that **data events are enabled** for the specific resources (S3 buckets or Lambda functions), confirm the event occurred within the **90-day CloudTrail console limit**, and ensure you're using an **organization-wide trail** for cross-account visibility rather than a single-account trail (lines 15-22).

## Validating Fixes with AWS CLI Commands

After identifying the potential cause, run these commands to validate the fix before applying changes.

### Verify CloudWatch Alarm Namespace and Dimensions

```bash

# Replace <AlarmName> with your alarm identifier

ALARM=$(aws cloudwatch describe-alarms --alarm-names <AlarmName> --query "MetricAlarms[0]")
NAMESPACE=$(echo $ALARM | jq -r '.Namespace')
DIMENSIONS=$(echo $ALARM | jq -r '.Dimensions | map("\(.Name)=\(.Value)") | join(",")')

# List matching metrics

aws cloudwatch list-metrics \
  --namespace "$NAMESPACE" \
  --dimensions $(printf "--dimensions %s " $(echo $DIMENSIONS | tr ',' '\n')) \
  --output table

```

If this returns an empty list, the alarm references a non-existent metric. Correct the namespace or dimension values to match the actual publishing source.

### Check Log Group Existence and IAM Permissions

```bash
LOG_GROUP=/aws/lambda/my-function
aws logs describe-log-groups --log-group-name-prefix "$LOG_GROUP"

# Verify the Lambda execution role has CloudWatch Logs permissions

aws lambda get-function-configuration --function-name my-function \
  --query 'Role' | xargs -I{} aws iam get-role --role-name {} \
  --query 'Role.AssumeRolePolicyDocument.Statement[?Action==`logs:*`]' --output table

```

Missing log groups or the absence of `logs:PutLogEvents` permission triggers the "Missing logs" fix in the skill's reference matrix.

### Force Custom Metric Activity

```bash
aws cloudwatch put-metric-data \
  --namespace "MyApp/Metrics" \
  --metric-name "ActiveSessions" \
  --value 1 \
  --unit Count \
  --dimensions InstanceId=$(aws sts get-caller-identity --query 'Account' --output text)

```

Emitting data at least once every two weeks keeps custom metrics active and queryable.

### Inspect X-Ray Sampling Configuration

```bash
aws xray get-sampling-rules \
  --rule-name "Default" \
  --query 'SamplingRuleRecord.SamplingRule.{ReservoirCount,FixedRate,ServiceName}'

```

If `ReservoirCount` is too low for your traffic volume, increase it:

```bash
aws xray update-sampling-rule \
  --rule-name "Default" \
  --sampling-rule '{"FixedRate":0.2,"ReservoirCount":5}'

```

### Enable CloudTrail Data Events for S3

```bash
aws cloudtrail put-event-selectors \
  --trail-name my-trail \
  --event-selectors '[{
      "ReadWriteType": "All",
      "IncludeManagementEvents": true,
      "DataResources": [{
          "Type": "AWS::S3::Object",
          "Values": ["arn:aws:s3:::my-bucket/"]
      }]
  }]'

```

This resolves "Can't find events" errors by capturing S3 object-level API calls.

## Key Reference Files

When troubleshooting manually, consult these specific files in the `aws/agent-toolkit-for-aws` repository:

- **[`skills/core-skills/aws-observability/SKILL.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/core-skills/aws-observability/SKILL.md)** – Entry point defining the skill's routing logic and high-level capabilities (lines 17-27).
- **[`skills/core-skills/aws-observability/references/troubleshooting.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/core-skills/aws-observability/references/troubleshooting.md)** – Comprehensive error-cause-fix matrices for alarms, logs, metrics, tracing, and CloudTrail.
- **[`skills/core-skills/aws-observability/assets/alarm-template.ts`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/core-skills/aws-observability/assets/alarm-template.ts)** – CDK snippets for rebuilding broken alarms with correct configurations.
- **[`skills/core-skills/aws-observability/assets/otel-config.yaml`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/core-skills/aws-observability/assets/otel-config.yaml)** – Default ADOT collector configuration for X-Ray and EMF metrics.

## Summary

- Start diagnostics with the **"Top 5" checklist** in [`troubleshooting.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/troubleshooting.md) to identify the failure category quickly.
- Verify **namespace and dimension exactness** using `aws cloudwatch list-metrics` before assuming an alarm is broken.
- Check **log group existence and IAM roles** when logs are missing; create groups and attach `AWSLambdaBasicExecutionRole` if necessary.
- Emit **custom metric datapoints weekly** to prevent them from becoming inactive and unqueryable.
- Adjust **X-Ray sampling rules** (ReservoirCount and FixedRate) when traces appear sporadically or not at all.
- Enable **CloudTrail data events** and use organization-wide trails to capture S3 and Lambda API calls across accounts.

## Frequently Asked Questions

### Why does my CloudWatch alarm show INSUFFICIENT_DATA?

The alarm likely references a metric with a **wrong namespace or dimension values**. According to [`troubleshooting.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/troubleshooting.md) (lines 21-23), the namespace is case-sensitive and must match exactly (e.g., `AWS/Lambda` not `aws/lambda`). Run `aws cloudwatch list-metrics` with the alarm's namespace and dimensions to verify the metric exists and is actively publishing.

### How do I prevent custom metrics from disappearing?

**Emit at least one datapoint every two weeks**. As noted in [`troubleshooting.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/troubleshooting.md) (lines 74-77), CloudWatch metrics become inactive after 14 days without data. Use `aws cloudwatch put-metric-data` to publish a dummy value weekly, or ensure your application sends regular heartbeats to keep the metric visible in the console and API.

### Why can't I see X-Ray traces for my Lambda function?

**Check active tracing and sampling rules**. The skill identifies three common causes: active tracing is disabled on the Lambda function, X-Ray sampling rules are too restrictive (low `ReservoirCount` or `FixedRate`), or downstream services lack instrumentation (lines 95-100). Enable active tracing in the Lambda console and increase the sampling rate using `aws xray update-sampling-rule`.

### How do I troubleshoot missing CloudTrail events for S3 object access?

**Enable data events for the specific S3 bucket**. By default, CloudTrail only logs management events. To capture object-level operations like `GetObject` or `PutObject`, run `aws cloudtrail put-event-selectors` with `DataResources` configured for your bucket ARN (lines 15-18). Also verify you're looking within the 90-day console retention window or querying archived S3 logs for older events.