How to Integrate Custom Observability Tools with the AWS Agent Toolkit

You can integrate custom observability tools with the AWS Agent Toolkit by leveraging the built-in OpenTelemetry wrapper to emit structured logs and traces, then forwarding that telemetry to external platforms like Datadog, Splunk, or Prometheus via CloudWatch Logs subscriptions or custom exporters.

The AWS Agent Toolkit (AAT) automatically emits X-Ray traces and CloudWatch logs for every AgentCore invocation, but teams often need to consolidate this data into existing monitoring stacks. To integrate custom observability tools with the AWS Agent Toolkit, you extend the default telemetry pipeline using the OpenTelemetry (OTEL) wrapper injected into the container entrypoint, allowing you to route metrics, logs, and traces to third-party platforms without modifying core toolkit components.

Architecture Overview

The AWS Agent Toolkit uses an OpenTelemetry instrumentation layer that intercepts all telemetry before it reaches AWS-native services. You can tap into this pipeline at multiple points to redirect data to external observability platforms.

+-------------------+      +-------------------+      +-------------------+
|  AgentCore Code   | ---> | OTEL Wrapper (run | ---> | CloudWatch Logs   |
| (Python)          |      | opentelemetry-    |      |   & Metrics       |
|                   |      | instrument)       |      +-------------------+
+-------------------+      +-------------------+              |
         |                         |                          |
         |   (custom spans/metrics)                          |
         v                         v                          v
+-------------------+      +-------------------+      +-------------------+
|   X-Ray Tracing   |      |   Custom Exporter | ---> | External Observ.  |
| (trace data)      |      | (Lambda/Kinesis)  |      | (Datadog, etc.)   |
+-------------------+      +-------------------+      +-------------------+

The opentelemetry-instrument wrapper auto-instruments the Python process, capturing both automatic telemetry and custom instrumentation you add. From CloudWatch Logs, you deploy subscription filters that stream data to Lambda functions or Kinesis Data Firehose, which then transform and forward data to your external observability backend.

Prerequisites: IAM and Container Configuration

Before implementing custom integrations, ensure your execution role has the required permissions. According to the observability.md reference in the repository, you must attach a policy allowing CloudWatch and X-Ray operations.

Copy the JSON policy fragment from lines 39-50 of plugins/aws-agents/skills/agents-optimize/references/observability.md:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "cloudwatch:PutMetricData",
        "xray:PutTraceSegments",
        "xray:PutTelemetryRecords"
      ],
      "Resource": "*"
    }
  ]
}

Attach this policy to the IAM role assigned to your AgentCore container runtime.

Step-by-Step Integration Guide

Configure the OpenTelemetry Entrypoint

The toolkit requires the OTEL wrapper to initialize instrumentation before your application code runs. In your Dockerfile, use the opentelemetry-instrument command as the entrypoint.

As documented in observability.md (lines 25-31), update your container configuration:

FROM public.ecr.aws/python/python:3.11-slim

COPY requirements.txt .
RUN pip install -r requirements.txt

COPY . /app
WORKDIR /app

# Required: OTEL wrapper for automatic instrumentation

CMD ["opentelemetry-instrument", "python", "-m", "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8080"]

The tools/validate.py script in the repository verifies that this entrypoint is present during the build process.

Implement Structured Logging

Replace print statements with Python's logging module to ensure OTEL captures your log events. According to lines 53-65 of observability.md, structured logging is required for telemetry correlation.

import logging

logger = logging.getLogger(__name__)

def process_request(session_id: str):
    # Good: Captured by CloudWatch and OTEL

    logger.info("Processing request", extra={"session_id": session_id})
    
    # Bad: Plain print is ignored by OTEL instrumentation

    # print(f"Processing request {session_id}")

This approach ensures logs contain trace context and appear in both CloudWatch and any external sinks you configure.

Create Custom OpenTelemetry Spans

Add custom spans to trace specific business operations within your agent. These spans automatically merge into the X-Ray traces that AgentCore already emits.

from opentelemetry import trace
from opentelemetry.trace import SpanKind

tracer = trace.get_tracer(__name__)

def my_custom_operation():
    with tracer.start_as_current_span(
        "my_custom_operation", 
        kind=SpanKind.INTERNAL
    ) as span:
        span.set_attribute("custom.attribute", "value")
        span.add_event("operation_step_1")
        # Your business logic here

Publish Custom CloudWatch Metrics

For metrics that need to appear in external dashboards, publish directly to CloudWatch using the boto3 client. The IAM policy already includes cloudwatch:PutMetricData.

import boto3
import time

cloudwatch = boto3.client("cloudwatch")

def publish_custom_metric(value: float):
    cloudwatch.put_metric_data(
        Namespace="MyAgentToolkit/Custom",
        MetricData=[
            {
                "MetricName": "ProcessingLatency",
                "Timestamp": time.time(),
                "Value": value,
                "Unit": "Milliseconds",
                "Dimensions": [
                    {"Name": "AgentName", "Value": "MyAgent"}
                ],
            }
        ],
    )

Export to External Observability Platforms

Deploy a CloudWatch Logs subscription filter that invokes a Lambda function to forward data to your external platform. The Lambda reads log events from CloudWatch and pushes them to HTTP endpoints like Datadog's log intake API.

import json
import urllib3

http = urllib3.PoolManager()
DATADOG_ENDPOINT = "https://http-intake.logs.datadoghq.com/v1/input/<API_KEY>"

def lambda_handler(event, context):
    for record in event["records"]:
        payload = json.loads(record["message"])
        http.request(
            "POST", 
            DATADOG_ENDPOINT, 
            body=json.dumps(payload).encode(),
            headers={"Content-Type": "application/json"}
        )
    return {"statusCode": 200}

Attach this Lambda to the CloudWatch log group created by AgentCore, which follows the pattern /<service>/<environment>/....

Configure Cross-Account Observability (Optional)

If you maintain separate monitoring and workload accounts, follow the 5-step cross-account configuration procedure documented in lines 108-118 of observability.md. This involves configuring CloudWatch cross-account sharing and X-Ray daemon settings to centralize telemetry in a dedicated monitoring account.

Key Repository Files

Understanding these source files helps you customize the integration:

• plugins/aws-agents/skills/agents-optimize/references/observability.md — Contains the complete reference for Docker entrypoints, IAM policies (lines 39-50), logging best practices (lines 53-65), and cross-account setup (lines 108-118).
• skills/core-skills/aws-observability/SKILL.md — Defines the observability skill that triggers when users query CloudWatch, X-Ray, or custom metrics.
• tools/validate.py — Build-time validation script that verifies opentelemetry-instrument is present in the container image.
• plugins/aws-data-analytics/skills/amazon-opensearch-service/references/observability.md — Reference playbook for ingesting traces into OpenSearch when you need searchable log backends.

Summary

• Use the OpenTelemetry wrapper (opentelemetry-instrument) as your container entrypoint to enable automatic telemetry capture.
• Write structured logs using Python's logging module instead of print to ensure OTEL correlation.
• Add custom spans via the OpenTelemetry API to trace business-specific operations alongside X-Ray data.
• Grant IAM permissions using the policy template from lines 39-50 of observability.md to allow CloudWatch and X-Ray publishing.
• Forward telemetry by attaching CloudWatch Logs subscription filters to Lambda functions that push to external platforms like Datadog or Splunk.
• Validate integration using the agentcore logs and agentcore traces CLI commands to confirm data flows to both AWS and external systems.

Frequently Asked Questions

How do I verify that OpenTelemetry is instrumenting my agent correctly?

Use the agentcore logs --runtime <AgentName> and agentcore traces list --runtime <AgentName> commands to verify emissions. Check that custom spans appear in the X-Ray console and that structured logs contain trace IDs. If data is missing, confirm the Dockerfile uses opentelemetry-instrument as the entrypoint and that the IAM role includes the permissions from lines 39-50 of observability.md.

Can I send traces directly to Datadog without going through CloudWatch?

While the AWS Agent Toolkit natively emits to X-Ray and CloudWatch, you can configure the OpenTelemetry Collector as a sidecar to export directly to OTLP-compatible endpoints. However, the documented approach uses CloudWatch Logs subscriptions because they provide durable buffering and transformation capabilities before reaching external systems.

What is the performance impact of custom OpenTelemetry spans?

The OpenTelemetry wrapper uses asynchronous batch processing with minimal overhead. According to the toolkit's implementation, spans are aggregated in memory and exported in batches, typically adding less than 5% latency to request processing. For high-throughput agents, consider sampling strategies configured in the OTEL_TRACES_SAMPLER environment variable.

How do I handle cross-account observability for centralized monitoring?

Follow the 5-step procedure documented in lines 108-118 of observability.md. This involves configuring CloudWatch cross-account observability in your central monitoring account, updating the X-Ray daemon configuration to send traces to a central account, and ensuring the execution role in your workload account has permissions to write to the monitoring account's CloudWatch resources.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}