How to Monitor Agents Running with AWS Agent Toolkit: X-Ray, CloudWatch, and Cross-Account Observability

The AWS Agent Toolkit ships with a built-in observability stack that automatically captures logs, metrics, and distributed traces for every AgentCore agent using AWS X-Ray and Amazon CloudWatch, requiring no manual instrumentation for basic telemetry.

The aws/agent-toolkit-for-aws repository provides a complete framework for building production-grade AI agents. When you need to monitor agents running with AWS Agent Toolkit, the framework delivers end-to-end visibility through auto-instrumented telemetry that activates whether you run locally with agentcore dev or in production with agentcore deploy.

Auto-Enabled Observability Components

According to the observability reference in plugins/aws-agents/skills/agents-optimize/references/observability.md, every AgentCore agent automatically emits telemetry data through three integrated AWS services.

X-Ray Distributed Tracing

Every agent invocation creates a complete distributed trace in AWS X-Ray without requiring code changes. The tracing captures the full request lifecycle, including upstream and downstream service calls, latency bottlenecks, and error propagation. These traces appear in the CloudWatch console within approximately 10 seconds of invocation completion.

CloudWatch Logs and Metrics

The AgentCore runtime automatically ships structured log entries to Amazon CloudWatch Logs using a dedicated log group. Metrics publish automatically to the AWS/BedrockAgentCore namespace, providing standard dimensions such as invocation count, error rates, and latency percentiles. A ready-made CloudWatch dashboard aggregates these metrics alongside CPU and memory utilization.

Prerequisites for Full-Stack Telemetry

To enable the complete observability pipeline, ensure your environment meets these requirements documented in plugins/aws-agents/skills/agents-optimize/references/observability.md:

• AWS credentials locally – The OpenTelemetry exporter requires active credentials to push spans and logs to CloudWatch and X-Ray.
• CloudWatch Transaction Search enabled – Required for trace search visibility in the console; without it, traces ingest but remain invisible in the UI.
• OTEL entrypoint wrapper – Your container Dockerfile must start with opentelemetry-instrument to ensure automatic runtime instrumentation.
• IAM permissions on the execution role – The agent's execution role needs permissions for logs:* and xray:* actions to write telemetry data.

Instrumenting Your Agent Code

While traces and infrastructure metrics are automatic, application logs require specific implementation. Use the standard Python logging module rather than print() statements, as the AgentCore pipeline captures structured logs from the logging framework while ignoring stdout prints.

import logging

logger = logging.getLogger(__name__)

# Structured logging with custom dimensions

logger.info("Processing request", extra={"session_id": session_id, "user_id": user_id})

Viewing Telemetry via CLI

The agentcore CLI provides direct access to telemetry without navigating the AWS Console.

Querying Traces

List recent traces for a specific runtime:

agentcore traces list --runtime <AgentName> --since 30m

Retrieve detailed span information for a specific trace:

agentcore traces get <traceId> --runtime <AgentName>

Streaming Logs

Filter logs by level and timeframe:

agentcore logs --runtime <AgentName> --level error --since 1h

CloudWatch Dashboard

Access the pre-built dashboard in the CloudWatch console under the AWS/BedrockAgentCore namespace to view invocation counts, error rates, latency percentiles (p50, p99), and resource utilization in a unified view.

Continuous Production Monitoring

For production workloads, enable continuous evaluation workflows that stream live telemetry to CloudWatch. The agents-optimize skill provides an online evaluator as detailed in plugins/aws-agents/skills/agents-optimize/SKILL.md:

# Add a continuous online evaluator with 10% sampling

agentcore add online-eval --name production_monitor \
  --eval my_quality_monitor --sample-rate 0.1

Alternatively, enable the built-in production_monitor evaluator directly from the agents-harden skill for standardized production monitoring, as referenced in plugins/aws-agents/skills/agents-harden/SKILL.md.

Control the evaluator state with:

agentcore pause online-eval production_monitor
agentcore resume online-eval production_monitor

Cross-Account Observability for Multi-Environment Setups

When agents run across multiple AWS accounts (development, staging, production), aggregate all telemetry into a central monitoring account following this workflow from plugins/aws-agents/skills/agents-optimize/references/observability.md:

1. Designate a monitoring account – Select the central account for unified dashboard viewing.
2. Configure CloudWatch cross-account settings – Enable Metrics and Logs sharing; traces share automatically via X-Ray.
3. Link source accounts – Connect dev, staging, and production accounts via AWS Organizations or manual linking.
4. Deploy agents – No additional IAM configuration required; telemetry flows automatically.
5. View unified data – The CloudWatch console displays metrics, logs, and traces side-by-side, identified by source account ID.

Best Practices and Edge Cases

Consider these implementation details from the source code to avoid common monitoring pitfalls:

• Trace delay – Expect approximately 10 seconds between invocation and trace availability; older documentation may cite 30–60 seconds.
• IAM scope – While wildcard * resources suffice for logs and X-Ray, apply least-privilege principles by restricting to specific log groups and trace resources when possible.
• Dashboard latency – Metrics appear at 1-minute granularity only if you enable detailed monitoring on underlying compute resources (see EC2 monitoring guidance in the EC2 skill references).

Summary

• The AWS Agent Toolkit provides zero-configuration observability through auto-enabled X-Ray tracing and CloudWatch logging for every AgentCore agent.
• Prerequisites include the opentelemetry-instrument wrapper, CloudWatch Transaction Search, and proper IAM permissions for logs and X-Ray.
• Use the Python logging module for structured logs; avoid print() statements which the pipeline ignores.
• Query telemetry directly via agentcore traces and agentcore logs CLI commands, or view the pre-built CloudWatch dashboard under AWS/BedrockAgentCore.
• Enable continuous production monitoring through the online-eval command in the agents-optimize and agents-harden skills.
• Implement cross-account observability by linking multiple AWS accounts to a central monitoring account for unified visibility.

Frequently Asked Questions

Do I need to manually instrument my code for X-Ray tracing?

No. Distributed tracing is automatic when using the AgentCore runtime. According to plugins/aws-agents/skills/agents-optimize/references/observability.md, every agent invocation creates an X-Ray trace without code changes, provided your container uses the opentelemetry-instrument entrypoint wrapper and has the necessary IAM permissions.

How do I view logs for a specific agent runtime?

Use the agentcore logs CLI command with your runtime name. For example, agentcore logs --runtime MyAgent --since 30m streams recent logs, and you can filter by severity level using --level error. Logs also appear in the CloudWatch Logs console under the automatically provisioned log group for your agent.

What IAM permissions are required for observability?

The agent's execution role requires permissions for logs:* and xray:* actions to write to CloudWatch Logs and X-Ray. For local development, ensure your AWS credentials have permissions to push spans and logs to these services. As noted in the observability documentation, a wildcard resource policy is functionally sufficient, though production environments should scope permissions to specific resources.

Can I monitor agents across multiple AWS accounts?

Yes. Configure CloudWatch cross-account observability by designating a central monitoring account and linking your source accounts (dev, staging, prod) via AWS Organizations. Once linked, deploy agents normally—the telemetry aggregates automatically into the monitoring account's CloudWatch console, displaying metrics, logs, and traces identified by source account ID without requiring additional agent configuration.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}