# AWS Observability Skill Examples in the Agent Toolkit for AWS

> Explore AWS Observability skill examples in the Agent Toolkit repo. Instrument agents with CloudWatch alarms, X-Ray tracing, and structured logging easily.

- Repository: [Amazon Web Services/agent-toolkit-for-aws](https://github.com/aws/agent-toolkit-for-aws)
- Tags: examples
- Published: 2026-06-28

---

**The AWS Observability skill provides ready-to-use code examples for instrumenting agents with CloudWatch alarms, X-Ray tracing, and structured logging through reference files located in the `skills/core-skills/aws-observability` directory.**

The **Agent Toolkit for AWS** (`aws/agent-toolkit-for-aws`) includes a comprehensive observability skill that enables developers to monitor and troubleshoot agent workloads using AWS-native services. This core skill routes user requests for metrics, logs, and traces to specialized reference implementations written in Markdown. Whether you are building Python-based agents or infrastructure-as-code with CDK, the repository contains concrete examples you can copy and adapt immediately.

## Core Capabilities of the Observability Skill

The observability skill is defined in **[`skills/core-skills/aws-observability/SKILL.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/core-skills/aws-observability/SKILL.md)** and supports six primary operational domains:

- **Log Insights** – Execute CloudWatch Logs Insights queries for ad-hoc analysis.
- **Alarms** – Deploy metric, composite, and anomaly-detection alarms with sensible defaults.
- **Custom Metrics** – Publish data using the Embedded Metric Format (EMF) or standard CloudWatch APIs.
- **Tracing** – Enable automatic X-Ray instrumentation via the OpenTelemetry wrapper and ADOT collector.
- **Dashboards** – Generate CloudWatch dashboards that unify logs, metrics, and traces.
- **Cross-account observability** – Link multiple source accounts to a central monitoring account.

## Creating CloudWatch Alarms with CDK

The **[`skills/core-skills/aws-observability/references/alarms.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/core-skills/aws-observability/references/alarms.md)** file contains production-ready TypeScript snippets for creating error-rate alarms. This example calculates a percentage-based error rate and triggers when it exceeds 5% across three evaluation periods:

```typescript
import {
  Alarm,
  ComparisonOperator,
  MathExpression,
  TreatMissingData,
} from 'aws-cdk-lib/aws-cloudwatch';
import { Duration } from 'aws-cdk-lib';

const errorRateAlarm = new Alarm(this, 'ErrorRateAlarm', {
  metric: new MathExpression({
    expression: 'IF(invocations > 0, errors * 100 / invocations, 0)',
    usingMetrics: {
      errors: fn.metricErrors({ period: Duration.minutes(1) }),
      invocations: fn.metricInvocations({ period: Duration.minutes(1) }),
    },
  }),
  threshold: 5,                               // 5% error rate
  evaluationPeriods: 3,
  datapointsToAlarm: 2,
  comparisonOperator: ComparisonOperator.GREATER_THAN_THRESHOLD,
  treatMissingData: TreatMissingData.NOT_BREACHING,
});

```

## Enabling X-Ray Tracing for Python Agents

According to the **[`plugins/aws-agents/skills/agents-optimize/references/observability.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/plugins/aws-agents/skills/agents-optimize/references/observability.md)** guide, you enable automatic tracing by wrapping the application entrypoint with the OpenTelemetry instrumentation agent.

Update your Dockerfile to use the `opentelemetry-instrument` command:

```dockerfile

# Dockerfile snippet

CMD ["opentelemetry-instrument", "python", "-m", "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8080"]

```

Attach the following IAM policy to the agent’s execution role to permit telemetry submission:

```json
{
  "Effect": "Allow",
  "Action": [
    "logs:CreateLogGroup",
    "logs:CreateLogStream",
    "logs:PutLogEvents",
    "xray:PutTraceSegments",
    "xray:PutTelemetryRecords"
  ],
  "Resource": "*"
}

```

## Structured Logging Examples

The same observability reference document demonstrates Python logging practices that integrate with the CloudWatch pipeline. Use the standard `logging` module with structured `extra` parameters rather than `print` statements:

```python
import logging
logger = logging.getLogger(__name__)

# Properly captured by the observability pipeline

logger.info("User request processed", extra={"session_id": session_id})

```

## Querying Traces via the Agent Toolkit CLI

Once deployed, the observability skill exposes CLI commands to inspect distributed traces. The **`agentcore`** CLI routes these commands to the X-Ray API and formats the output for the console:

```bash

# List recent traces for a specific runtime

agentcore traces list --runtime MyAgent --since 1h --limit 10

# Retrieve detailed information for a specific trace ID

agentcore traces get <traceId> --runtime MyAgent

```

Similarly, running `agentcore logs …` triggers the skill to query CloudWatch Logs Insights using the patterns defined in **[`log-insights.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/log-insights.md)**.

## Building CloudWatch Dashboards

The **[`skills/core-skills/aws-observability/references/dashboards.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/core-skills/aws-observability/references/dashboards.md)** file and the **[`assets/alarm-template.ts`](https://github.com/aws/agent-toolkit-for-aws/blob/main/assets/alarm-template.ts)** asset provide a reusable CDK construct for monitoring dashboards. Import the template and include its widgets in a new dashboard:

```typescript
import * as cw from 'aws-cdk-lib/aws-cloudwatch';
import { AlarmTemplate } from '../../assets/alarm-template';

const dashboard = new cw.Dashboard(this, 'MyDashboard', {
  widgets: AlarmTemplate.widgets,
});

```

## Cross-Account Observability Setup

For multi-account architectures, the observability skill references the AgentCore documentation on linking source accounts to a centralized monitoring account. This configuration aggregates metrics and traces into a single pane of glass, reducing operational overhead for distributed agent deployments.

## Summary

- The **AWS Observability skill** is defined in [`skills/core-skills/aws-observability/SKILL.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/core-skills/aws-observability/SKILL.md) and routes requests to specialized reference files.
- **CDK examples** in [`alarms.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/alarms.md) and [`dashboards.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/dashboards.md) provide copy-paste constructs for error-rate alarms and unified dashboards.
- **Python agents** enable tracing by wrapping the entrypoint with `opentelemetry-instrument` and attaching the correct IAM permissions.
- The **`agentcore` CLI** commands (`traces list`, `traces get`, `logs`) invoke the skill to query X-Ray and CloudWatch Logs Insights.
- **Structured logging** uses standard Python logging with context-rich `extra` fields for automatic ingestion.

## Frequently Asked Questions

### How do I invoke the AWS Observability skill?

The skill is invoked implicitly when you run **`agentcore`** CLI commands such as `agentcore logs` or `agentcore traces list`. The CLI reads the skill definition from [`SKILL.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/SKILL.md) and routes the request to the appropriate reference file (e.g., [`tracing.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/tracing.md) or [`log-insights.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/log-insights.md)) to execute the underlying AWS API calls.

### Where are the observability skill examples located?

Reference implementations are stored in **`skills/core-skills/aws-observability/references/`**, with specific files for alarms, dashboards, log insights, and tracing. Additional AgentCore-specific guidance lives in **[`plugins/aws-agents/skills/agents-optimize/references/observability.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/plugins/aws-agents/skills/agents-optimize/references/observability.md)**.

### What tracing options does the observability skill support?

The skill supports **X-Ray tracing** through the AWS Distro for OpenTelemetry (ADOT) collector. You can enable automatic instrumentation by using the `opentelemetry-instrument` wrapper in your container entrypoint, or configure the ADOT collector manually for advanced sampling and cross-account trace aggregation.

### Can I use the observability skill for cross-account monitoring?

Yes. The skill references the **multi-account observability** patterns documented in AgentCore, allowing you to link multiple source accounts to a central monitoring account. This setup enables unified CloudWatch dashboards and X-Ray service maps across organizational boundaries.