# How to Monitor Agents Running with AWS Agent Toolkit: X-Ray, CloudWatch, and Cross-Account Observability > Easily monitor your agents with AWS Agent Toolkit. Leverage X-Ray and CloudWatch for logs, metrics, and traces without manual instrumentation. Get started today. - Repository: [Amazon Web Services/agent-toolkit-for-aws](https://github.com/aws/agent-toolkit-for-aws) - Tags: how-to-guide - Published: 2026-06-30 --- **The AWS Agent Toolkit ships with a built-in observability stack that automatically captures logs, metrics, and distributed traces for every AgentCore agent using AWS X-Ray and Amazon CloudWatch, requiring no manual instrumentation for basic telemetry.** The `aws/agent-toolkit-for-aws` repository provides a complete framework for building production-grade AI agents. When you need to **monitor agents running with AWS Agent Toolkit**, the framework delivers end-to-end visibility through auto-instrumented telemetry that activates whether you run locally with `agentcore dev` or in production with `agentcore deploy`. ## Auto-Enabled Observability Components According to the observability reference in [`plugins/aws-agents/skills/agents-optimize/references/observability.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/plugins/aws-agents/skills/agents-optimize/references/observability.md), every AgentCore agent automatically emits telemetry data through three integrated AWS services. ### X-Ray Distributed Tracing Every agent invocation creates a complete distributed trace in **AWS X-Ray** without requiring code changes. The tracing captures the full request lifecycle, including upstream and downstream service calls, latency bottlenecks, and error propagation. These traces appear in the CloudWatch console within approximately 10 seconds of invocation completion. ### CloudWatch Logs and Metrics The AgentCore runtime automatically ships structured log entries to **Amazon CloudWatch Logs** using a dedicated log group. Metrics publish automatically to the `AWS/BedrockAgentCore` namespace, providing standard dimensions such as invocation count, error rates, and latency percentiles. A ready-made CloudWatch dashboard aggregates these metrics alongside CPU and memory utilization. ## Prerequisites for Full-Stack Telemetry To enable the complete observability pipeline, ensure your environment meets these requirements documented in [`plugins/aws-agents/skills/agents-optimize/references/observability.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/plugins/aws-agents/skills/agents-optimize/references/observability.md): - **AWS credentials locally** – The OpenTelemetry exporter requires active credentials to push spans and logs to CloudWatch and X-Ray. - **CloudWatch Transaction Search enabled** – Required for trace search visibility in the console; without it, traces ingest but remain invisible in the UI. - **OTEL entrypoint wrapper** – Your container Dockerfile must start with `opentelemetry-instrument` to ensure automatic runtime instrumentation. - **IAM permissions on the execution role** – The agent's execution role needs permissions for `logs:*` and `xray:*` actions to write telemetry data. ## Instrumenting Your Agent Code While traces and infrastructure metrics are automatic, application logs require specific implementation. Use the standard Python `logging` module rather than `print()` statements, as the AgentCore pipeline captures structured logs from the logging framework while ignoring stdout prints. ```python import logging logger = logging.getLogger(__name__) # Structured logging with custom dimensions logger.info("Processing request", extra={"session_id": session_id, "user_id": user_id}) ``` ## Viewing Telemetry via CLI The `agentcore` CLI provides direct access to telemetry without navigating the AWS Console. ### Querying Traces List recent traces for a specific runtime: ```bash agentcore traces list --runtime --since 30m ``` Retrieve detailed span information for a specific trace: ```bash agentcore traces get --runtime ``` ### Streaming Logs Filter logs by level and timeframe: ```bash agentcore logs --runtime --level error --since 1h ``` ### CloudWatch Dashboard Access the pre-built dashboard in the CloudWatch console under the `AWS/BedrockAgentCore` namespace to view invocation counts, error rates, latency percentiles (p50, p99), and resource utilization in a unified view. ## Continuous Production Monitoring For production workloads, enable continuous evaluation workflows that stream live telemetry to CloudWatch. The **agents-optimize** skill provides an online evaluator as detailed in [`plugins/aws-agents/skills/agents-optimize/SKILL.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/plugins/aws-agents/skills/agents-optimize/SKILL.md): ```bash # Add a continuous online evaluator with 10% sampling agentcore add online-eval --name production_monitor \ --eval my_quality_monitor --sample-rate 0.1 ``` Alternatively, enable the built-in **production_monitor** evaluator directly from the **agents-harden** skill for standardized production monitoring, as referenced in [`plugins/aws-agents/skills/agents-harden/SKILL.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/plugins/aws-agents/skills/agents-harden/SKILL.md). Control the evaluator state with: ```bash agentcore pause online-eval production_monitor agentcore resume online-eval production_monitor ``` ## Cross-Account Observability for Multi-Environment Setups When agents run across multiple AWS accounts (development, staging, production), aggregate all telemetry into a central monitoring account following this workflow from [`plugins/aws-agents/skills/agents-optimize/references/observability.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/plugins/aws-agents/skills/agents-optimize/references/observability.md): 1. **Designate a monitoring account** – Select the central account for unified dashboard viewing. 2. **Configure CloudWatch cross-account settings** – Enable Metrics and Logs sharing; traces share automatically via X-Ray. 3. **Link source accounts** – Connect dev, staging, and production accounts via AWS Organizations or manual linking. 4. **Deploy agents** – No additional IAM configuration required; telemetry flows automatically. 5. **View unified data** – The CloudWatch console displays metrics, logs, and traces side-by-side, identified by source account ID. ## Best Practices and Edge Cases Consider these implementation details from the source code to avoid common monitoring pitfalls: - **Trace delay** – Expect approximately 10 seconds between invocation and trace availability; older documentation may cite 30–60 seconds. - **IAM scope** – While wildcard `*` resources suffice for logs and X-Ray, apply least-privilege principles by restricting to specific log groups and trace resources when possible. - **Dashboard latency** – Metrics appear at 1-minute granularity only if you enable *detailed monitoring* on underlying compute resources (see EC2 monitoring guidance in the EC2 skill references). ## Summary - The AWS Agent Toolkit provides **zero-configuration observability** through auto-enabled X-Ray tracing and CloudWatch logging for every AgentCore agent. - **Prerequisites** include the `opentelemetry-instrument` wrapper, CloudWatch Transaction Search, and proper IAM permissions for logs and X-Ray. - Use the **Python `logging` module** for structured logs; avoid `print()` statements which the pipeline ignores. - Query telemetry directly via **`agentcore traces`** and **`agentcore logs`** CLI commands, or view the pre-built CloudWatch dashboard under `AWS/BedrockAgentCore`. - Enable **continuous production monitoring** through the `online-eval` command in the agents-optimize and agents-harden skills. - Implement **cross-account observability** by linking multiple AWS accounts to a central monitoring account for unified visibility. ## Frequently Asked Questions ### Do I need to manually instrument my code for X-Ray tracing? No. Distributed tracing is automatic when using the AgentCore runtime. According to [`plugins/aws-agents/skills/agents-optimize/references/observability.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/plugins/aws-agents/skills/agents-optimize/references/observability.md), every agent invocation creates an X-Ray trace without code changes, provided your container uses the `opentelemetry-instrument` entrypoint wrapper and has the necessary IAM permissions. ### How do I view logs for a specific agent runtime? Use the `agentcore logs` CLI command with your runtime name. For example, `agentcore logs --runtime MyAgent --since 30m` streams recent logs, and you can filter by severity level using `--level error`. Logs also appear in the CloudWatch Logs console under the automatically provisioned log group for your agent. ### What IAM permissions are required for observability? The agent's execution role requires permissions for `logs:*` and `xray:*` actions to write to CloudWatch Logs and X-Ray. For local development, ensure your AWS credentials have permissions to push spans and logs to these services. As noted in the observability documentation, a wildcard resource policy is functionally sufficient, though production environments should scope permissions to specific resources. ### Can I monitor agents across multiple AWS accounts? Yes. Configure CloudWatch cross-account observability by designating a central monitoring account and linking your source accounts (dev, staging, prod) via AWS Organizations. Once linked, deploy agents normally—the telemetry aggregates automatically into the monitoring account's CloudWatch console, displaying metrics, logs, and traces identified by source account ID without requiring additional agent configuration.