How to Troubleshoot Issues with the Observability Skill in AWS Agent Toolkit

The AWS Observability Skill diagnoses CloudWatch, X-Ray, and CloudTrail failures by routing through a structured matrix in troubleshooting.md and validating hypotheses with specific AWS CLI commands.

The aws-observability skill in the aws/agent-toolkit-for-aws repository helps agents build, configure, and optimize AWS monitoring stacks. When you need to troubleshoot issues with the observability skill in AWS Agent Toolkit, the system follows a systematic workflow defined in skills/core-skills/aws-observability/SKILL.md and references detailed symptom-cause-fix tables. This guide reproduces that exact diagnostic logic so you can resolve alarm, log, metric, tracing, and audit issues efficiently.

Understanding the Troubleshooting Architecture

The skill implements a five-step diagnostic flow that mirrors how AWS support engineers approach observability failures.

First, the routing table in skills/core-skills/aws-observability/SKILL.md (lines 17-27) identifies the specific service category—whether the issue involves CloudWatch alarms, Log Insights queries, custom metrics, X-Ray traces, or CloudTrail events.

Next, the skill presents the "Top 5" fixes—a concise checklist covering the most common failure patterns located at the opening section of skills/core-skills/aws-observability/references/troubleshooting.md (lines 5-12).

Then it drills down into symptom-cause-fix matrices (e.g., lines 19-25 for alarms, lines 46-55 for missing logs, lines 92-100 for X-Ray traces). Each matrix entry maps a specific symptom to its root cause and corrective action.

Finally, the skill guides users to run validation CLI commands to confirm the hypothesis before iterating to the next potential cause if the fix fails.

Diagnosing Common Failure Patterns

The troubleshooting reference file organizes solutions by service category. Below are the exact failure patterns the skill recognizes and how to resolve them.

Fixing CloudWatch Alarms Stuck in INSUFFICIENT_DATA

An alarm stuck in INSUFFICIENT_DATA usually indicates a namespace or dimension mismatch. According to troubleshooting.md (lines 21-23), the metric identifier in the alarm must match the publishing source exactly—including case sensitivity (AWS/Lambda vs. aws/lambda).

If the alarm fails to trigger despite breaching thresholds, verify the M-of-N configuration and missing-data treatment. The skill recommends lowering the M (evaluation periods to breach), raising the N (total periods), or switching missing-data treatment from missing to notBreaching (lines 30-32).

Resolving Missing CloudWatch Logs

When logs fail to appear, check three components in sequence: the log group existence, IAM permissions, and retention settings. The skill directs users to verify the log group exists (creating it if necessary), ensure the resource has the AWSLambdaBasicExecutionRole or equivalent, and confirm the retention period hasn't expired (lines 50-55).

Repairing Empty Log Insights Queries

Empty query results typically stem from incorrect time ranges, wrong log group specifications, or Infrequent Access (IA) class log groups used with pattern commands. Expand the query time window, confirm the exact log group name, and avoid querying IA class groups for pattern commands (lines 60-63).

Preventing Custom Metric Disappearance

Custom metrics become inactive if they receive no data points for two weeks or more. To prevent disappearance, emit at least one datapoint weekly using put-metric-data, or query with get-metric-statistics using exact identifiers including namespace and dimensions (lines 74-77).

Recovering Missing X-Ray Traces

Missing traces usually indicate disabled active tracing, restrictive sampling rules, or uninstrumented downstream services. Enable active tracing on Lambda functions or API Gateway stages, increase the ReservoirCount or FixedRate in X-Ray sampling rules, and add AWS Distro for OpenTelemetry (ADOT) instrumentation to downstream calls (lines 95-100).

Locating CloudTrail Data Events

When CloudTrail events are missing, verify that data events are enabled for the specific resources (S3 buckets or Lambda functions), confirm the event occurred within the 90-day CloudTrail console limit, and ensure you're using an organization-wide trail for cross-account visibility rather than a single-account trail (lines 15-22).

Validating Fixes with AWS CLI Commands

After identifying the potential cause, run these commands to validate the fix before applying changes.

Verify CloudWatch Alarm Namespace and Dimensions

# Replace <AlarmName> with your alarm identifier

ALARM=$(aws cloudwatch describe-alarms --alarm-names <AlarmName> --query "MetricAlarms[0]")
NAMESPACE=$(echo $ALARM | jq -r '.Namespace')
DIMENSIONS=$(echo $ALARM | jq -r '.Dimensions | map("\(.Name)=\(.Value)") | join(",")')

# List matching metrics

aws cloudwatch list-metrics \
  --namespace "$NAMESPACE" \
  --dimensions $(printf "--dimensions %s " $(echo $DIMENSIONS | tr ',' '\n')) \
  --output table

If this returns an empty list, the alarm references a non-existent metric. Correct the namespace or dimension values to match the actual publishing source.

Check Log Group Existence and IAM Permissions

LOG_GROUP=/aws/lambda/my-function
aws logs describe-log-groups --log-group-name-prefix "$LOG_GROUP"

# Verify the Lambda execution role has CloudWatch Logs permissions

aws lambda get-function-configuration --function-name my-function \
  --query 'Role' | xargs -I{} aws iam get-role --role-name {} \
  --query 'Role.AssumeRolePolicyDocument.Statement[?Action==`logs:*`]' --output table

Missing log groups or the absence of logs:PutLogEvents permission triggers the "Missing logs" fix in the skill's reference matrix.

Force Custom Metric Activity

aws cloudwatch put-metric-data \
  --namespace "MyApp/Metrics" \
  --metric-name "ActiveSessions" \
  --value 1 \
  --unit Count \
  --dimensions InstanceId=$(aws sts get-caller-identity --query 'Account' --output text)

Emitting data at least once every two weeks keeps custom metrics active and queryable.

Inspect X-Ray Sampling Configuration

aws xray get-sampling-rules \
  --rule-name "Default" \
  --query 'SamplingRuleRecord.SamplingRule.{ReservoirCount,FixedRate,ServiceName}'

If ReservoirCount is too low for your traffic volume, increase it:

aws xray update-sampling-rule \
  --rule-name "Default" \
  --sampling-rule '{"FixedRate":0.2,"ReservoirCount":5}'

Enable CloudTrail Data Events for S3

aws cloudtrail put-event-selectors \
  --trail-name my-trail \
  --event-selectors '[{
      "ReadWriteType": "All",
      "IncludeManagementEvents": true,
      "DataResources": [{
          "Type": "AWS::S3::Object",
          "Values": ["arn:aws:s3:::my-bucket/"]
      }]
  }]'

This resolves "Can't find events" errors by capturing S3 object-level API calls.

Key Reference Files

When troubleshooting manually, consult these specific files in the aws/agent-toolkit-for-aws repository:

• skills/core-skills/aws-observability/SKILL.md – Entry point defining the skill's routing logic and high-level capabilities (lines 17-27).
• skills/core-skills/aws-observability/references/troubleshooting.md – Comprehensive error-cause-fix matrices for alarms, logs, metrics, tracing, and CloudTrail.
• skills/core-skills/aws-observability/assets/alarm-template.ts – CDK snippets for rebuilding broken alarms with correct configurations.
• skills/core-skills/aws-observability/assets/otel-config.yaml – Default ADOT collector configuration for X-Ray and EMF metrics.

Summary

• Start diagnostics with the "Top 5" checklist in troubleshooting.md to identify the failure category quickly.
• Verify namespace and dimension exactness using aws cloudwatch list-metrics before assuming an alarm is broken.
• Check log group existence and IAM roles when logs are missing; create groups and attach AWSLambdaBasicExecutionRole if necessary.
• Emit custom metric datapoints weekly to prevent them from becoming inactive and unqueryable.
• Adjust X-Ray sampling rules (ReservoirCount and FixedRate) when traces appear sporadically or not at all.
• Enable CloudTrail data events and use organization-wide trails to capture S3 and Lambda API calls across accounts.

Frequently Asked Questions

Why does my CloudWatch alarm show INSUFFICIENT_DATA?

The alarm likely references a metric with a wrong namespace or dimension values. According to troubleshooting.md (lines 21-23), the namespace is case-sensitive and must match exactly (e.g., AWS/Lambda not aws/lambda). Run aws cloudwatch list-metrics with the alarm's namespace and dimensions to verify the metric exists and is actively publishing.

How do I prevent custom metrics from disappearing?

Emit at least one datapoint every two weeks. As noted in troubleshooting.md (lines 74-77), CloudWatch metrics become inactive after 14 days without data. Use aws cloudwatch put-metric-data to publish a dummy value weekly, or ensure your application sends regular heartbeats to keep the metric visible in the console and API.

Why can't I see X-Ray traces for my Lambda function?

Check active tracing and sampling rules. The skill identifies three common causes: active tracing is disabled on the Lambda function, X-Ray sampling rules are too restrictive (low ReservoirCount or FixedRate), or downstream services lack instrumentation (lines 95-100). Enable active tracing in the Lambda console and increase the sampling rate using aws xray update-sampling-rule.

How do I troubleshoot missing CloudTrail events for S3 object access?

Enable data events for the specific S3 bucket. By default, CloudTrail only logs management events. To capture object-level operations like GetObject or PutObject, run aws cloudtrail put-event-selectors with DataResources configured for your bucket ARN (lines 15-18). Also verify you're looking within the 90-day console retention window or querying archived S3 logs for older events.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}