How to Monitor Agents Built with the AWS Agent Toolkit: Complete Observability Guide

The AWS Agent Toolkit provides automatic observability through AWS X-Ray, CloudWatch Logs, and CloudWatch Metrics for every AgentCore agent, requiring only standard AWS credentials and IAM permissions to capture distributed traces and structured logs without additional instrumentation code.

The aws/agent-toolkit-for-aws repository ships with a built-in observability stack that activates the moment you deploy an agent. Whether you are running agents locally with agentcore dev or in production with agentcore deploy, the toolkit automatically instruments your application using OpenTelemetry and streams telemetry to AWS monitoring services.

Auto-Enabled Observability Components

Every AgentCore agent receives automatic instrumentation for distributed tracing and log aggregation. According to the observability reference in plugins/aws-agents/skills/agents-optimize/references/observability.md, two critical services are active by default:

• AWS X-Ray tracing – Each agent invocation creates a trace segment that captures the entire request lifecycle, including subsegments for tool calls and LLM interactions.
• Amazon CloudWatch Logs – The AgentCore runtime captures structured log entries and ships them to a dedicated CloudWatch log group without requiring custom log handlers.

These components function whether you are testing locally or running containerized workloads in AWS, provided the container uses the opentelemetry-instrument entrypoint wrapper.

Prerequisites for Full-Stack Telemetry

To successfully export spans and logs to AWS, your environment must satisfy four specific requirements outlined in the observability documentation:

Requirement	Implementation Detail
AWS credentials	The OpenTelemetry exporter requires valid credentials to push data to CloudWatch and X-Ray.
CloudWatch Transaction Search	Must be enabled in the AWS Console for trace search visibility; otherwise traces are ingested but remain invisible.
OTEL entrypoint wrapper	Your Dockerfile must start the application with opentelemetry-instrument to activate automatic instrumentation.
IAM permissions	The execution role needs logs:* and xray:* actions to write telemetry data.

A wildcard resource (*) is sufficient for both logs and X-Ray permissions, though you should scope policies to specific resource ARNs when possible for least-privilege compliance.

Instrumenting Your Agent Code

The AgentCore pipeline automatically captures standard Python logging module output as structured CloudWatch logs. Statements using print() are ignored by the telemetry pipeline.

Use the standard library logger with structured extras to ensure searchable, contextual log entries:

import logging

logger = logging.getLogger(__name__)

# This creates a structured log entry in CloudWatch

logger.info("Processing request", extra={"session_id": session_id, "user_id": user_id})

This approach requires no additional dependencies or configuration changes; the AgentCore runtime handles log formatting and transport automatically.

Querying Telemetry with the CLI

The agentcore CLI provides native commands for inspecting live telemetry without navigating the AWS Console.

Viewing Traces

List recent traces for a specific runtime or retrieve detailed span information:

# List traces from the last 30 minutes

agentcore traces list --runtime MyAgent --since 30m

# Get detailed trace information by ID

agentcore traces get <traceId> --runtime MyAgent

Streaming Logs

Filter logs by severity level and time range directly from the terminal:

# Stream error logs from the past hour

agentcore logs --runtime MyAgent --level error --since 1h

CloudWatch Dashboard

The toolkit automatically provisions a CloudWatch dashboard under the AWS/BedrockAgentCore namespace. This dashboard displays invocation counts, error rates, latency percentiles (p50, p95, p99), CPU and memory utilization, and error log counts.

Production Monitoring and Continuous Evaluation

For production workloads, the agents-optimize skill provides a continuous quality-gate workflow that adds an online evaluator. This evaluator streams live telemetry to CloudWatch while running automated checks against your agent's outputs.

Enable continuous monitoring using the online-eval subcommand:

# Add a continuous online evaluator with 10% sampling

agentcore add online-eval --name production_monitor \
  --eval my_quality_monitor --sample-rate 0.1

Alternatively, the agents-harden skill provides a built-in production_monitor evaluator that can be enabled directly for standardized production telemetry.

Control the evaluator state without redeploying:

agentcore pause online-eval production_monitor
agentcore resume online-eval production_monitor

Cross-Account Observability for Multi-Environment Setups

When agents operate across multiple AWS accounts (development, staging, production), you can aggregate all telemetry into a single monitoring account for unified visibility. The configuration requires five steps as documented in plugins/aws-agents/skills/agents-optimize/references/observability.md:

1. Designate a monitoring account – Select the central account where you will view aggregated dashboards.
2. Enable CloudWatch cross-account settings – Configure both Metrics and Logs sharing in the monitoring account; X-Ray traces are shared via the X-Ray service directly.
3. Link source accounts – Connect dev, staging, and production accounts via AWS Organizations or manual linking.
4. Deploy agents – No additional IAM configuration is required; telemetry flows automatically to the monitoring account.
5. View unified data – The CloudWatch console displays metrics, logs, and traces side-by-side, identified by source account ID.

This architecture eliminates the need to switch between AWS accounts for debugging multi-environment issues.

Troubleshooting and Best Practices

Understanding the operational characteristics of the observability stack prevents false negatives during incident response.

• Trace delay – Expect approximately 10 seconds between an invocation and its trace appearing in the console; legacy documentation may cite 30–60 seconds, but current latency is significantly lower.
• Dashboard granularity – Metrics appear at 1-minute granularity if you enable detailed monitoring on underlying compute resources, as referenced in the EC2 skills documentation (skills/specialized-skills/ec2-skills/launching-ec2-instance-with-best-practices/references/launch-ec2-instance-with-best-practices.md).
• Logging scope – Always use the standard Python logging module rather than print() statements, as the AgentCore instrumentation specifically hooks into the logging framework to capture structured data.

Summary

• The AWS Agent Toolkit automatically enables AWS X-Ray and CloudWatch Logs for every AgentCore agent without code changes.
• Successful telemetry export requires AWS credentials, the OTEL entrypoint wrapper, CloudWatch Transaction Search enabled, and proper IAM permissions.
• Use the standard Python logging module with structured extras to ensure logs are captured and searchable.
• Query live telemetry using agentcore traces list, agentcore traces get, and agentcore logs commands.
• Enable continuous production monitoring via the online evaluator in the agents-optimize or agents-harden skills.
• Configure cross-account observability by linking accounts to a central monitoring account for unified multi-environment dashboards.

Frequently Asked Questions

How do I enable X-Ray tracing for my AgentCore agent?

X-Ray tracing is automatically enabled for all AgentCore agents when you use the opentelemetry-instrument entrypoint wrapper in your container. No code changes are required. Ensure your IAM execution role includes xray:* permissions and that CloudWatch Transaction Search is enabled in your AWS account to view traces in the console.

What is the difference between basic and detailed monitoring in the CloudWatch dashboard?

Basic monitoring provides metrics at 5-minute intervals, while detailed monitoring provides 1-minute granularity. Detailed monitoring requires enabling the option on your underlying compute resources, such as EC2 instances or Fargate services, as documented in the EC2 and container skills references.

Can I monitor agents running in multiple AWS accounts from a single dashboard?

Yes. Configure cross-account observability by designating a monitoring account, enabling CloudWatch cross-account settings for metrics and logs, and linking your source accounts via AWS Organizations. X-Ray traces are shared automatically through the X-Ray service. Once linked, the CloudWatch console displays telemetry from all accounts side-by-side.

Why are my print statements not showing up in CloudWatch Logs?

The AgentCore telemetry pipeline specifically captures output from the standard Python logging module and ignores print() statements. To ensure your output appears in CloudWatch, use logger.info(), logger.error(), or other logging methods with the standard library logger, optionally including structured extras for better searchability.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}