# Debug Observability Issues in AWS: Fixing Missing Metrics and Traces in Agent Toolkit

> Fix missing AWS metrics and traces in Agent Toolkit. Learn how to resolve common observability issues caused by entry point wrappers, IAM permissions, or disabled CloudWatch Transaction Search.

- Repository: [Amazon Web Services/agent-toolkit-for-aws](https://github.com/aws/agent-toolkit-for-aws)
- Tags: how-to-guide
- Published: 2026-07-03

---

**Missing AWS metrics and traces in Agent Toolkit for AWS usually stem from three root causes: missing the OpenTelemetry wrapper in your container entry point, insufficient IAM permissions for CloudWatch and X-Ray, or disabled CloudWatch Transaction Search in your account.**

The Agent Toolkit for AWS (`agentcore`) relies on OpenTelemetry (OTEL) to automatically instrument agents and ship telemetry to **Amazon CloudWatch** and **AWS X-Ray**. When you debug observability issues in AWS and find that metrics, logs, or traces are not appearing, the failure almost always traces back to one of three prerequisites defined in the repository’s observability configuration.

## Prerequisites for Observability in Agent Toolkit

According to [`plugins/aws-agents/skills/agents-optimize/references/observability.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/plugins/aws-agents/skills/agents-optimize/references/observability.md), three components must be correctly configured before telemetry flows. Omitting any one results in silent data loss.

### OpenTelemetry Entry-Point Wrapper

The Docker image must start the agent through the OTEL wrapper (`opentelemetry-instrument`). Without this wrapper, the agent runs normally but exports no spans. In [`observability.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/observability.md) (lines 25-31), the reference implementation shows the required `CMD` instruction:

```dockerfile

# Example Dockerfile fragment

FROM python:3.11-slim
WORKDIR /app
COPY . .
RUN pip install -r requirements.txt

# OTEL wrapper ensures logs and X-Ray traces are emitted

CMD ["opentelemetry-instrument", "python", "-m", "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8080"]

```

### IAM Permissions for CloudWatch and X-Ray

The execution role requires explicit permissions for `logs:*` and `xray:*` actions. Missing permissions silently drop telemetry without error messages. The required JSON policy is defined in [`observability.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/observability.md) (lines 39-49). Ensure your role includes:

- `logs:PutLogEvents`
- `xray:PutTraceSegments`
- `xray:PutTelemetryRecords`

### CloudWatch Transaction Search Configuration

X-Ray traces are ingested regardless of this setting, but they are **not searchable** until Transaction Search is enabled in the account. When disabled, `agentcore traces list` returns empty results even though the data exists. See [`observability.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/observability.md) (lines 12-18) for cross-account linking instructions.

## Common Symptoms and Diagnostic Fixes

When you debug observability issues in AWS deployments, match your symptom to the specific root cause and apply the targeted fix.

### No Logs Appearing in CloudWatch

**Symptom:** Containers run but CloudWatch log streams remain empty.

**Root Cause:** Using `print()` instead of Python’s `logging` module. As documented in [`observability.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/observability.md) (lines 53-66), the OTEL collector only captures structured logs from the standard logging framework.

**Fix:** Replace `print` calls with `logger.info/debug/error`:

```python
import logging
logger = logging.getLogger(__name__)

def handler(event, context):
    # Good – captured by CloudWatch

    logger.info("Processing request", extra={"session_id": event.get("session_id")})
    # Bad – not captured

    # print(f"Processing request {event.get('session_id')}")

```

### Traces Missing After Fresh Deployment

**Symptom:** First deployment shows no traces in X-Ray.

**Root Cause:** IAM role missing `logs:PutLogEvents` or `xray:PutTraceSegments`. Verify the policy snippet from lines 39-49 of the observability reference is attached to the execution role.

### Observability Works Locally But Fails in Containers

**Symptom:** Local tests emit traces, but deployed containers do not.

**Root Cause:** Dockerfile lacking the OTEL wrapper entry point. Check that your container uses `opentelemetry-instrument` as the entry command, not just `python` or `uvicorn` directly.

### Cross-Account Agents Show No Data

**Symptom:** Agents in linked accounts appear in the console but show no telemetry.

**Root Cause:** Monitoring account not linked or Transaction Search not enabled in source accounts. Follow the cross-account setup steps in [`observability.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/observability.md) (lines 12-18) and verify the monitoring link is active.

## Querying Telemetry with agentcore CLI

After invoking an agent, wait approximately **10 seconds** (the current delay documented at line 80, not the older 30-60 second guidance) before querying. Use these commands to verify data ingestion:

```bash

# List recent traces (≈10 s after invocation)

agentcore traces list --runtime <AgentName> --since 1h --limit 10

# Get a specific trace

agentcore traces get <traceId> --runtime <AgentName>

# Stream recent logs

agentcore logs --runtime <AgentName> --since 30m

# Filter logs by level

agentcore logs --runtime <AgentName> --level error --since 1h

```

## Generating CloudWatch Dashboards

For database-specific observability, the toolkit provides a helper script to generate CloudWatch dashboards. Located at [`skills/specialized-skills/database-skills/amazon-elasticache/scripts/generate_dashboards.py`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/specialized-skills/database-skills/amazon-elasticache/scripts/generate_dashboards.py) (lines 22-524), this script creates CloudFormation templates for ElastiCache monitoring:

```bash
python3 scripts/generate_dashboards.py --serverless my-cache --output elasticache-observability.json
aws cloudformation deploy --template-file elasticache-observability.json \
  --stack-name my-cache-observability --region us-east-1

```

## Summary

To debug observability issues in AWS when using Agent Toolkit for AWS, verify these key points:

- **Container entry point** must use `opentelemetry-instrument` to wrap the application start command.
- **IAM execution role** requires `logs:*` and `xray:*` permissions as specified in [`observability.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/observability.md) lines 39-49.
- **CloudWatch Transaction Search** must be enabled to query X-Ray traces, even when data is ingested.
- **Logging** must use Python’s `logging` module; `print()` statements are not captured.
- **Telemetry delay** is approximately 10 seconds, not the legacy 30-60 second window.

## Frequently Asked Questions

### Why don't my traces appear immediately after invocation?

The current observability pipeline in `agentcore` introduces a delay of approximately **10 seconds** between invocation and trace availability (line 80 of [`observability.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/observability.md)). Previous documentation referenced 30-60 second delays, but the updated OpenTelemetry exporter has reduced this latency.

### Why does print() output not appear in CloudWatch?

The Agent Toolkit for AWS collector intercepts logs from the standard Python `logging` framework only. Calls to `print()` write to stdout but bypass the OTEL log pipeline. Replace all `print` statements with `logger.info()` or equivalent as shown in [`observability.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/observability.md) lines 53-66.

### What IAM permissions are required for observability?

The execution role needs permissions for CloudWatch Logs (`logs:PutLogEvents`, `logs:CreateLogGroup`) and AWS X-Ray (`xray:PutTraceSegments`, `xray:PutTelemetryRecords`). The exact JSON policy is defined in [`plugins/aws-agents/skills/agents-optimize/references/observability.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/plugins/aws-agents/skills/agents-optimize/references/observability.md) at lines 39-49.

### Why can't I search X-Ray traces even though they exist?

X-Ray traces are ingested into the service regardless of account settings, but they are only searchable when **CloudWatch Transaction Search** is enabled. If `agentcore traces list` returns empty results while the X-Ray console shows data, enable Transaction Search in the source account or verify the cross-account monitoring link is active (lines 12-18).