AWS Observability Skill Examples in the Agent Toolkit for AWS

The AWS Observability skill provides ready-to-use code examples for instrumenting agents with CloudWatch alarms, X-Ray tracing, and structured logging through reference files located in the skills/core-skills/aws-observability directory.

The Agent Toolkit for AWS (aws/agent-toolkit-for-aws) includes a comprehensive observability skill that enables developers to monitor and troubleshoot agent workloads using AWS-native services. This core skill routes user requests for metrics, logs, and traces to specialized reference implementations written in Markdown. Whether you are building Python-based agents or infrastructure-as-code with CDK, the repository contains concrete examples you can copy and adapt immediately.

Core Capabilities of the Observability Skill

The observability skill is defined in skills/core-skills/aws-observability/SKILL.md and supports six primary operational domains:

• Log Insights – Execute CloudWatch Logs Insights queries for ad-hoc analysis.
• Alarms – Deploy metric, composite, and anomaly-detection alarms with sensible defaults.
• Custom Metrics – Publish data using the Embedded Metric Format (EMF) or standard CloudWatch APIs.
• Tracing – Enable automatic X-Ray instrumentation via the OpenTelemetry wrapper and ADOT collector.
• Dashboards – Generate CloudWatch dashboards that unify logs, metrics, and traces.
• Cross-account observability – Link multiple source accounts to a central monitoring account.

Creating CloudWatch Alarms with CDK

The skills/core-skills/aws-observability/references/alarms.md file contains production-ready TypeScript snippets for creating error-rate alarms. This example calculates a percentage-based error rate and triggers when it exceeds 5% across three evaluation periods:

import {
  Alarm,
  ComparisonOperator,
  MathExpression,
  TreatMissingData,
} from 'aws-cdk-lib/aws-cloudwatch';
import { Duration } from 'aws-cdk-lib';

const errorRateAlarm = new Alarm(this, 'ErrorRateAlarm', {
  metric: new MathExpression({
    expression: 'IF(invocations > 0, errors * 100 / invocations, 0)',
    usingMetrics: {
      errors: fn.metricErrors({ period: Duration.minutes(1) }),
      invocations: fn.metricInvocations({ period: Duration.minutes(1) }),
    },
  }),
  threshold: 5,                               // 5% error rate
  evaluationPeriods: 3,
  datapointsToAlarm: 2,
  comparisonOperator: ComparisonOperator.GREATER_THAN_THRESHOLD,
  treatMissingData: TreatMissingData.NOT_BREACHING,
});

Enabling X-Ray Tracing for Python Agents

According to the plugins/aws-agents/skills/agents-optimize/references/observability.md guide, you enable automatic tracing by wrapping the application entrypoint with the OpenTelemetry instrumentation agent.

Update your Dockerfile to use the opentelemetry-instrument command:

# Dockerfile snippet

CMD ["opentelemetry-instrument", "python", "-m", "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8080"]

Attach the following IAM policy to the agent’s execution role to permit telemetry submission:

{
  "Effect": "Allow",
  "Action": [
    "logs:CreateLogGroup",
    "logs:CreateLogStream",
    "logs:PutLogEvents",
    "xray:PutTraceSegments",
    "xray:PutTelemetryRecords"
  ],
  "Resource": "*"
}

Structured Logging Examples

The same observability reference document demonstrates Python logging practices that integrate with the CloudWatch pipeline. Use the standard logging module with structured extra parameters rather than print statements:

import logging
logger = logging.getLogger(__name__)

# Properly captured by the observability pipeline

logger.info("User request processed", extra={"session_id": session_id})

Querying Traces via the Agent Toolkit CLI

Once deployed, the observability skill exposes CLI commands to inspect distributed traces. The agentcore CLI routes these commands to the X-Ray API and formats the output for the console:

# List recent traces for a specific runtime

agentcore traces list --runtime MyAgent --since 1h --limit 10

# Retrieve detailed information for a specific trace ID

agentcore traces get <traceId> --runtime MyAgent

Similarly, running agentcore logs … triggers the skill to query CloudWatch Logs Insights using the patterns defined in log-insights.md.

Building CloudWatch Dashboards

The skills/core-skills/aws-observability/references/dashboards.md file and the assets/alarm-template.ts asset provide a reusable CDK construct for monitoring dashboards. Import the template and include its widgets in a new dashboard:

import * as cw from 'aws-cdk-lib/aws-cloudwatch';
import { AlarmTemplate } from '../../assets/alarm-template';

const dashboard = new cw.Dashboard(this, 'MyDashboard', {
  widgets: AlarmTemplate.widgets,
});

Cross-Account Observability Setup

For multi-account architectures, the observability skill references the AgentCore documentation on linking source accounts to a centralized monitoring account. This configuration aggregates metrics and traces into a single pane of glass, reducing operational overhead for distributed agent deployments.

Summary

• The AWS Observability skill is defined in skills/core-skills/aws-observability/SKILL.md and routes requests to specialized reference files.
• CDK examples in alarms.md and dashboards.md provide copy-paste constructs for error-rate alarms and unified dashboards.
• Python agents enable tracing by wrapping the entrypoint with opentelemetry-instrument and attaching the correct IAM permissions.
• The agentcore CLI commands (traces list, traces get, logs) invoke the skill to query X-Ray and CloudWatch Logs Insights.
• Structured logging uses standard Python logging with context-rich extra fields for automatic ingestion.

Frequently Asked Questions

How do I invoke the AWS Observability skill?

The skill is invoked implicitly when you run agentcore CLI commands such as agentcore logs or agentcore traces list. The CLI reads the skill definition from SKILL.md and routes the request to the appropriate reference file (e.g., tracing.md or log-insights.md) to execute the underlying AWS API calls.

Where are the observability skill examples located?

Reference implementations are stored in skills/core-skills/aws-observability/references/, with specific files for alarms, dashboards, log insights, and tracing. Additional AgentCore-specific guidance lives in plugins/aws-agents/skills/agents-optimize/references/observability.md.

What tracing options does the observability skill support?

The skill supports X-Ray tracing through the AWS Distro for OpenTelemetry (ADOT) collector. You can enable automatic instrumentation by using the opentelemetry-instrument wrapper in your container entrypoint, or configure the ADOT collector manually for advanced sampling and cross-account trace aggregation.

Can I use the observability skill for cross-account monitoring?

Yes. The skill references the multi-account observability patterns documented in AgentCore, allowing you to link multiple source accounts to a central monitoring account. This setup enables unified CloudWatch dashboards and X-Ray service maps across organizational boundaries.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}