# How to Monitor Agents Built with the AWS Agent Toolkit: Complete Observability Guide

> Learn to monitor agents built with the AWS Agent Toolkit. Get complete observability with X-Ray, CloudWatch Logs, and Metrics for your distributed traces and structured logs.

- Repository: [Amazon Web Services/agent-toolkit-for-aws](https://github.com/aws/agent-toolkit-for-aws)
- Tags: how-to-guide
- Published: 2026-06-28

---

**The AWS Agent Toolkit provides automatic observability through AWS X-Ray, CloudWatch Logs, and CloudWatch Metrics for every AgentCore agent, requiring only standard AWS credentials and IAM permissions to capture distributed traces and structured logs without additional instrumentation code.**

The `aws/agent-toolkit-for-aws` repository ships with a built-in observability stack that activates the moment you deploy an agent. Whether you are running agents locally with `agentcore dev` or in production with `agentcore deploy`, the toolkit automatically instruments your application using OpenTelemetry and streams telemetry to AWS monitoring services.

## Auto-Enabled Observability Components

Every **AgentCore** agent receives automatic instrumentation for distributed tracing and log aggregation. According to the observability reference in [`plugins/aws-agents/skills/agents-optimize/references/observability.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/plugins/aws-agents/skills/agents-optimize/references/observability.md), two critical services are active by default:

- **AWS X-Ray tracing** – Each agent invocation creates a trace segment that captures the entire request lifecycle, including subsegments for tool calls and LLM interactions.
- **Amazon CloudWatch Logs** – The AgentCore runtime captures structured log entries and ships them to a dedicated CloudWatch log group without requiring custom log handlers.

These components function whether you are testing locally or running containerized workloads in AWS, provided the container uses the `opentelemetry-instrument` entrypoint wrapper.

## Prerequisites for Full-Stack Telemetry

To successfully export spans and logs to AWS, your environment must satisfy four specific requirements outlined in the observability documentation:

| Requirement | Implementation Detail |
|-------------|----------------------|
| **AWS credentials** | The OpenTelemetry exporter requires valid credentials to push data to CloudWatch and X-Ray. |
| **CloudWatch Transaction Search** | Must be enabled in the AWS Console for trace search visibility; otherwise traces are ingested but remain invisible. |
| **OTEL entrypoint wrapper** | Your Dockerfile must start the application with `opentelemetry-instrument` to activate automatic instrumentation. |
| **IAM permissions** | The execution role needs `logs:*` and `xray:*` actions to write telemetry data. |

A wildcard resource (`*`) is sufficient for both logs and X-Ray permissions, though you should scope policies to specific resource ARNs when possible for least-privilege compliance.

## Instrumenting Your Agent Code

The AgentCore pipeline automatically captures standard Python `logging` module output as structured CloudWatch logs. Statements using `print()` are ignored by the telemetry pipeline.

Use the standard library logger with structured extras to ensure searchable, contextual log entries:

```python
import logging

logger = logging.getLogger(__name__)

# This creates a structured log entry in CloudWatch

logger.info("Processing request", extra={"session_id": session_id, "user_id": user_id})

```

This approach requires no additional dependencies or configuration changes; the AgentCore runtime handles log formatting and transport automatically.

## Querying Telemetry with the CLI

The `agentcore` CLI provides native commands for inspecting live telemetry without navigating the AWS Console.

### Viewing Traces

List recent traces for a specific runtime or retrieve detailed span information:

```bash

# List traces from the last 30 minutes

agentcore traces list --runtime MyAgent --since 30m

# Get detailed trace information by ID

agentcore traces get <traceId> --runtime MyAgent

```

### Streaming Logs

Filter logs by severity level and time range directly from the terminal:

```bash

# Stream error logs from the past hour

agentcore logs --runtime MyAgent --level error --since 1h

```

### CloudWatch Dashboard

The toolkit automatically provisions a CloudWatch dashboard under the `AWS/BedrockAgentCore` namespace. This dashboard displays invocation counts, error rates, latency percentiles (p50, p95, p99), CPU and memory utilization, and error log counts.

## Production Monitoring and Continuous Evaluation

For production workloads, the **agents-optimize** skill provides a continuous quality-gate workflow that adds an online evaluator. This evaluator streams live telemetry to CloudWatch while running automated checks against your agent's outputs.

Enable continuous monitoring using the online-eval subcommand:

```bash

# Add a continuous online evaluator with 10% sampling

agentcore add online-eval --name production_monitor \
  --eval my_quality_monitor --sample-rate 0.1

```

Alternatively, the **agents-harden** skill provides a built-in **production_monitor** evaluator that can be enabled directly for standardized production telemetry.

Control the evaluator state without redeploying:

```bash
agentcore pause online-eval production_monitor
agentcore resume online-eval production_monitor

```

## Cross-Account Observability for Multi-Environment Setups

When agents operate across multiple AWS accounts (development, staging, production), you can aggregate all telemetry into a single monitoring account for unified visibility. The configuration requires five steps as documented in [`plugins/aws-agents/skills/agents-optimize/references/observability.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/plugins/aws-agents/skills/agents-optimize/references/observability.md):

1. **Designate a monitoring account** – Select the central account where you will view aggregated dashboards.
2. **Enable CloudWatch cross-account settings** – Configure both Metrics and Logs sharing in the monitoring account; X-Ray traces are shared via the X-Ray service directly.
3. **Link source accounts** – Connect dev, staging, and production accounts via AWS Organizations or manual linking.
4. **Deploy agents** – No additional IAM configuration is required; telemetry flows automatically to the monitoring account.
5. **View unified data** – The CloudWatch console displays metrics, logs, and traces side-by-side, identified by source account ID.

This architecture eliminates the need to switch between AWS accounts for debugging multi-environment issues.

## Troubleshooting and Best Practices

Understanding the operational characteristics of the observability stack prevents false negatives during incident response.

- **Trace delay** – Expect approximately 10 seconds between an invocation and its trace appearing in the console; legacy documentation may cite 30–60 seconds, but current latency is significantly lower.
- **Dashboard granularity** – Metrics appear at 1-minute granularity if you enable detailed monitoring on underlying compute resources, as referenced in the EC2 skills documentation ([`skills/specialized-skills/ec2-skills/launching-ec2-instance-with-best-practices/references/launch-ec2-instance-with-best-practices.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/specialized-skills/ec2-skills/launching-ec2-instance-with-best-practices/references/launch-ec2-instance-with-best-practices.md)).
- **Logging scope** – Always use the standard Python `logging` module rather than `print()` statements, as the AgentCore instrumentation specifically hooks into the logging framework to capture structured data.

## Summary

- The AWS Agent Toolkit automatically enables **AWS X-Ray** and **CloudWatch Logs** for every AgentCore agent without code changes.
- Successful telemetry export requires **AWS credentials**, the **OTEL entrypoint wrapper**, **CloudWatch Transaction Search** enabled, and proper **IAM permissions**.
- Use the standard Python `logging` module with structured extras to ensure logs are captured and searchable.
- Query live telemetry using `agentcore traces list`, `agentcore traces get`, and `agentcore logs` commands.
- Enable **continuous production monitoring** via the online evaluator in the agents-optimize or agents-harden skills.
- Configure **cross-account observability** by linking accounts to a central monitoring account for unified multi-environment dashboards.

## Frequently Asked Questions

### How do I enable X-Ray tracing for my AgentCore agent?

X-Ray tracing is automatically enabled for all AgentCore agents when you use the `opentelemetry-instrument` entrypoint wrapper in your container. No code changes are required. Ensure your IAM execution role includes `xray:*` permissions and that CloudWatch Transaction Search is enabled in your AWS account to view traces in the console.

### What is the difference between basic and detailed monitoring in the CloudWatch dashboard?

Basic monitoring provides metrics at 5-minute intervals, while detailed monitoring provides 1-minute granularity. Detailed monitoring requires enabling the option on your underlying compute resources, such as EC2 instances or Fargate services, as documented in the EC2 and container skills references.

### Can I monitor agents running in multiple AWS accounts from a single dashboard?

Yes. Configure cross-account observability by designating a monitoring account, enabling CloudWatch cross-account settings for metrics and logs, and linking your source accounts via AWS Organizations. X-Ray traces are shared automatically through the X-Ray service. Once linked, the CloudWatch console displays telemetry from all accounts side-by-side.

### Why are my print statements not showing up in CloudWatch Logs?

The AgentCore telemetry pipeline specifically captures output from the standard Python `logging` module and ignores `print()` statements. To ensure your output appears in CloudWatch, use `logger.info()`, `logger.error()`, or other logging methods with the standard library logger, optionally including structured extras for better searchability.