Debug Observability Issues in AWS: Fixing Missing Metrics and Traces in Agent Toolkit

Missing AWS metrics and traces in Agent Toolkit for AWS usually stem from three root causes: missing the OpenTelemetry wrapper in your container entry point, insufficient IAM permissions for CloudWatch and X-Ray, or disabled CloudWatch Transaction Search in your account.

The Agent Toolkit for AWS (agentcore) relies on OpenTelemetry (OTEL) to automatically instrument agents and ship telemetry to Amazon CloudWatch and AWS X-Ray. When you debug observability issues in AWS and find that metrics, logs, or traces are not appearing, the failure almost always traces back to one of three prerequisites defined in the repository’s observability configuration.

Prerequisites for Observability in Agent Toolkit

According to plugins/aws-agents/skills/agents-optimize/references/observability.md, three components must be correctly configured before telemetry flows. Omitting any one results in silent data loss.

OpenTelemetry Entry-Point Wrapper

The Docker image must start the agent through the OTEL wrapper (opentelemetry-instrument). Without this wrapper, the agent runs normally but exports no spans. In observability.md (lines 25-31), the reference implementation shows the required CMD instruction:

# Example Dockerfile fragment

FROM python:3.11-slim
WORKDIR /app
COPY . .
RUN pip install -r requirements.txt

# OTEL wrapper ensures logs and X-Ray traces are emitted

CMD ["opentelemetry-instrument", "python", "-m", "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8080"]

IAM Permissions for CloudWatch and X-Ray

The execution role requires explicit permissions for logs:* and xray:* actions. Missing permissions silently drop telemetry without error messages. The required JSON policy is defined in observability.md (lines 39-49). Ensure your role includes:

• logs:PutLogEvents
• xray:PutTraceSegments
• xray:PutTelemetryRecords

CloudWatch Transaction Search Configuration

X-Ray traces are ingested regardless of this setting, but they are not searchable until Transaction Search is enabled in the account. When disabled, agentcore traces list returns empty results even though the data exists. See observability.md (lines 12-18) for cross-account linking instructions.

Common Symptoms and Diagnostic Fixes

When you debug observability issues in AWS deployments, match your symptom to the specific root cause and apply the targeted fix.

No Logs Appearing in CloudWatch

Symptom: Containers run but CloudWatch log streams remain empty.

Root Cause: Using print() instead of Python’s logging module. As documented in observability.md (lines 53-66), the OTEL collector only captures structured logs from the standard logging framework.

Fix: Replace print calls with logger.info/debug/error:

import logging
logger = logging.getLogger(__name__)

def handler(event, context):
    # Good – captured by CloudWatch

    logger.info("Processing request", extra={"session_id": event.get("session_id")})
    # Bad – not captured

    # print(f"Processing request {event.get('session_id')}")

Traces Missing After Fresh Deployment

Symptom: First deployment shows no traces in X-Ray.

Root Cause: IAM role missing logs:PutLogEvents or xray:PutTraceSegments. Verify the policy snippet from lines 39-49 of the observability reference is attached to the execution role.

Observability Works Locally But Fails in Containers

Symptom: Local tests emit traces, but deployed containers do not.

Root Cause: Dockerfile lacking the OTEL wrapper entry point. Check that your container uses opentelemetry-instrument as the entry command, not just python or uvicorn directly.

Cross-Account Agents Show No Data

Symptom: Agents in linked accounts appear in the console but show no telemetry.

Root Cause: Monitoring account not linked or Transaction Search not enabled in source accounts. Follow the cross-account setup steps in observability.md (lines 12-18) and verify the monitoring link is active.

Querying Telemetry with agentcore CLI

After invoking an agent, wait approximately 10 seconds (the current delay documented at line 80, not the older 30-60 second guidance) before querying. Use these commands to verify data ingestion:

# List recent traces (≈10 s after invocation)

agentcore traces list --runtime <AgentName> --since 1h --limit 10

# Get a specific trace

agentcore traces get <traceId> --runtime <AgentName>

# Stream recent logs

agentcore logs --runtime <AgentName> --since 30m

# Filter logs by level

agentcore logs --runtime <AgentName> --level error --since 1h

Generating CloudWatch Dashboards

For database-specific observability, the toolkit provides a helper script to generate CloudWatch dashboards. Located at skills/specialized-skills/database-skills/amazon-elasticache/scripts/generate_dashboards.py (lines 22-524), this script creates CloudFormation templates for ElastiCache monitoring:

python3 scripts/generate_dashboards.py --serverless my-cache --output elasticache-observability.json
aws cloudformation deploy --template-file elasticache-observability.json \
  --stack-name my-cache-observability --region us-east-1

Summary

To debug observability issues in AWS when using Agent Toolkit for AWS, verify these key points:

• Container entry point must use opentelemetry-instrument to wrap the application start command.
• IAM execution role requires logs:* and xray:* permissions as specified in observability.md lines 39-49.
• CloudWatch Transaction Search must be enabled to query X-Ray traces, even when data is ingested.
• Logging must use Python’s logging module; print() statements are not captured.
• Telemetry delay is approximately 10 seconds, not the legacy 30-60 second window.

Frequently Asked Questions

Why don't my traces appear immediately after invocation?

The current observability pipeline in agentcore introduces a delay of approximately 10 seconds between invocation and trace availability (line 80 of observability.md). Previous documentation referenced 30-60 second delays, but the updated OpenTelemetry exporter has reduced this latency.

Why does print() output not appear in CloudWatch?

The Agent Toolkit for AWS collector intercepts logs from the standard Python logging framework only. Calls to print() write to stdout but bypass the OTEL log pipeline. Replace all print statements with logger.info() or equivalent as shown in observability.md lines 53-66.

What IAM permissions are required for observability?

The execution role needs permissions for CloudWatch Logs (logs:PutLogEvents, logs:CreateLogGroup) and AWS X-Ray (xray:PutTraceSegments, xray:PutTelemetryRecords). The exact JSON policy is defined in plugins/aws-agents/skills/agents-optimize/references/observability.md at lines 39-49.

Why can't I search X-Ray traces even though they exist?

X-Ray traces are ingested into the service regardless of account settings, but they are only searchable when CloudWatch Transaction Search is enabled. If agentcore traces list returns empty results while the X-Ray console shows data, enable Transaction Search in the source account or verify the cross-account monitoring link is active (lines 12-18).

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}