How to Configure Logging for Agents Built with the AWS Agent Toolkit

The AWS Agent Toolkit enables comprehensive logging through two integrated layers: SDK-level wire debugging via Boto3 configuration and automatic CloudWatch provisioning for Lambda and API Gateway agents through specialized skills in the plugins/aws-core/skills/ directory.

The AWS Agent Toolkit provides an opinionated observability framework for agents deployed across AWS services. Whether you are debugging API calls during development or monitoring production workloads in Amazon CloudWatch, the toolkit standardizes logging configuration through concrete skill implementations. This guide references the actual source code in the aws/agent-toolkit-for-aws repository to demonstrate how to configure both Boto3 SDK diagnostics and application-level CloudWatch logging.

SDK-Level Logging with Boto3 and Botocore

The AWS SDK for Python (Boto3) and its underlying Botocore library use the standard Python logging module to emit wire-level request and response details. The toolkit documents these patterns in plugins/aws-core/skills/aws-sdk-python-usage/SKILL.md, providing specific helpers for different debugging scenarios.

Stream Logging for Debug Output

To quickly dump all Botocore traffic to stderr during development, use the set_stream_logger method. This approach is ideal for troubleshooting permission errors or API latency issues in real time.

import boto3

# Enable logging for all SDK components

boto3.set_stream_logger("")

# Or target only botocore internals

boto3.set_stream_logger("botocore")

File Logging for Persistent Diagnostics

For long-running agents where you need to review logs after execution, the set_file_logger method writes debug output to a specified file path. This implementation is documented in the SDK usage skill and prevents log loss in ephemeral compute environments.

from botocore.session import Session
import logging

session = Session()
session.set_file_logger(logging.DEBUG, "/tmp/botocore.log")

Programmatic Configuration with botocore.config.Config

You can combine logging configuration with retry and timeout policies by using the botocore.config.Config class. This method allows you to register event hooks that adjust log levels dynamically for specific client instances.

from botocore.config import Config
import boto3
import logging

cfg = Config(
    retries={"total_max_attempts": 2, "mode": "adaptive"},
    connect_timeout=5,
    read_timeout=10,
)

client = boto3.client("s3", config=cfg)

# Enable debug logging when requests are created

client.meta.events.register(
    "request-created", 
    lambda **_: logging.getLogger("botocore").setLevel(logging.DEBUG)
)

Application-Level Logging to Amazon CloudWatch

When agents run as Lambda functions, Fargate tasks, or behind API Gateway, the toolkit automatically provisions the necessary CloudWatch infrastructure. The relevant skills handle IAM role attachment, log group creation, and retention policy configuration.

Lambda Function Logging with aws-lambda-managed-instances

The aws-lambda-managed-instances skill automates CloudWatch setup for Lambda-based agents. According to plugins/aws-core/skills/aws-lambda-managed-instances/SKILL.md, this skill creates the log group /aws/lambda/<function-name>, attaches the AWSLambdaBasicExecutionRole (which includes logs:CreateLogGroup, logs:CreateLogStream, and logs:PutLogEvents), and configures a default 30-day retention period.

The skill accepts a log_retention parameter that allows you to customize how long logs persist before automatic deletion.

API Gateway Access and Execution Logging

For agents exposed via REST APIs, the creating-api-gateway-stage skill configures both access logs and execution logs. As documented in skills/specialized-skills/serverless-skills/creating-api-gateway-stage/references/create-api-gateway-stage.md, this skill emits the exact CLI commands required to enable CloudWatch logging and define structured JSON log formats for API requests.

IAM Permissions and Log Retention Policies

All CloudWatch logging implementations in the toolkit follow a consistent three-step pattern:

1. Create or validate the CloudWatch log group using aws logs create-log-group
2. Attach IAM permissions via the AWSLambdaBasicExecutionRole or a custom policy with CloudWatch Logs permissions
3. Set the retention period (commonly 30 days) to control storage costs

The debugging-lambda-timeouts skill in skills/specialized-skills/serverless-skills/debugging-lambda-timeouts/SKILL.md explicitly reminds developers to verify that the Lambda function's log group exists and that logging is not disabled, ensuring observability during troubleshooting sessions.

End-to-End Logging Configuration Example

Below is a complete implementation that combines SDK-level debugging with CloudWatch application logging for a Lambda-based agent. The toolkit handles the CloudWatch infrastructure, so you only need to configure the Python logger and Boto3 debug settings.

import logging
import boto3
from botocore.config import Config

# SDK-level logging: capture wire traffic to stderr

boto3.set_stream_logger("botocore")

# Configure client with retry logic

cfg = Config(
    retries={"total_max_attempts": 2, "mode": "adaptive"},
    connect_timeout=5,
    read_timeout=10,
)
s3_client = boto3.client("s3", config=cfg)

def lambda_handler(event, context):
    """
    Handler function with CloudWatch logging enabled.
    The aws-lambda-managed-instances skill automatically creates:
    - Log group: /aws/lambda/<function-name>
    - IAM role with CloudWatch permissions
    - 30-day retention policy
    """
    logger = logging.getLogger()
    logger.setLevel(logging.INFO)
    
    logger.info("Agent processing started", extra={"event_id": event.get("id")})
    
    try:
        response = s3_client.list_buckets()
        logger.debug("Retrieved %d buckets", len(response.get("Buckets", [])))
        return {"status": "success", "bucket_count": len(response["Buckets"])}
    except Exception as exc:
        logger.error("S3 operation failed: %s", exc, exc_info=True)
        raise

Summary

• SDK-level logging is configured via boto3.set_stream_logger() or set_file_logger() and documented in plugins/aws-core/skills/aws-sdk-python-usage/SKILL.md.
• CloudWatch logging for Lambda agents is automated by the aws-lambda-managed-instances skill, which provisions log groups, IAM roles, and retention policies.
• API Gateway logging is handled by the creating-api-gateway-stage skill, which configures access and execution logs through standardized CLI commands.
• All logging configurations require the AWSLambdaBasicExecutionRole or equivalent IAM permissions to write to CloudWatch Logs.
• The toolkit standardizes log retention (default 30 days) to balance observability needs with storage costs.

Frequently Asked Questions

How do I enable debug logging for Boto3 calls only?

Use boto3.set_stream_logger("botocore") to limit output to Botocore internals, or register an event hook on a specific client instance via client.meta.events.register() as shown in plugins/aws-core/skills/aws-sdk-python-usage/SKILL.md. This targets only the AWS SDK rather than your entire application.

What IAM permissions are required for CloudWatch logging?

The toolkit automatically attaches the AWSLambdaBasicExecutionRole managed policy, which includes logs:CreateLogGroup, logs:CreateLogStream, and logs:PutLogEvents. For custom roles, ensure these permissions are granted to allow the agent to write to CloudWatch Logs.

How do I configure log retention for Lambda functions?

When using the aws-lambda-managed-instances skill, pass the log_retention parameter (in days) to override the default 30-day retention period. The skill executes aws logs put-retention-policy automatically during deployment.

Can I log API Gateway requests to CloudWatch?

Yes. The creating-api-gateway-stage skill configures both access logging (full request/response details) and execution logging (API Gateway internal operations) by creating dedicated CloudWatch log groups and setting the appropriate stage variables, as documented in the skill's reference file.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}