AWS SDK Best Practices for Python (boto3): Production Patterns from the Agent Toolkit

Use low-level boto3 clients with explicit botocore.config.Config objects, reuse clients across your process, catch specific service exceptions, and leverage paginators and waiters instead of manual loops.

The Agent Toolkit for AWS repository codifies enterprise-grade patterns for writing resilient Python applications with the AWS SDK. Following the guidance in plugins/aws-core/skills/aws-sdk-python-usage/SKILL.md and its supporting reference documents ensures your boto3 code handles retries, pagination, and error scenarios correctly while maintaining high performance.

Client vs. Resource: Choosing the Right Abstraction

The AWS SDK for Python offers two distinct interfaces. Clients provide complete API coverage with direct mapping to AWS service operations, while Resources offer a higher-level object-oriented interface for specific services.

Use clients when you need access to the full API surface or when working with services that lack resource support. Switch to resources only for supported services like S3 or DynamoDB where you benefit from simplified operations like multipart uploads or batch writes. As documented in plugins/aws-core/skills/aws-sdk-python-usage/SKILL.md, prefer clients for most automation and infrastructure tasks to ensure you can access every API parameter.

Session Management and Client Configuration

Create an explicit boto3.Session only when you need to customize the profile, region, or shared configuration. Reuse a single client per process rather than constructing clients inside loops, which avoids connection pool exhaustion and reduces latency.

Configure clients using botocore.config.Config to control retries, timeouts, and connection pools. The Agent Toolkit recommends the newer total_max_attempts field (which counts attempts inclusively) rather than the legacy max_attempts.

import boto3
from botocore.config import Config

shared_cfg = Config(
    retries={"total_max_attempts": 3, "mode": "adaptive"},
    connect_timeout=5,
    read_timeout=10,
    max_pool_connections=50,
)

session = boto3.Session(profile_name="production", region_name="us-west-2")
s3 = session.client("s3", config=shared_cfg)
dynamodb = session.client("dynamodb", config=shared_cfg)

This pattern is detailed in plugins/aws-core/skills/aws-sdk-python-usage/references/configuration.md, which also covers service-specific settings like S3 addressing styles.

Error Handling with Specific Exceptions

Catch only specific botocore exceptions using the client.exceptions namespace. This prevents masking unexpected errors and allows targeted handling of known failure modes.

def get_bucket_location(bucket_name: str) -> str | None:
    try:
        resp = s3.get_bucket_location(Bucket=bucket_name)
        return resp.get("LocationConstraint")
    except s3.exceptions.NoSuchBucket:
        logger.warning("Bucket %s not found.", bucket_name)
        return None
    # Let unexpected exceptions propagate to the caller

Reserve generic ClientError handling for top-level exception boundaries only. The specific exception patterns are documented in plugins/aws-core/skills/aws-sdk-python-usage/references/error-handling.md.

Pagination with JMESPath

Never manually loop over NextToken or Marker fields. Instead, use client.get_paginator() combined with JMESPath search expressions to extract exactly the data you need.

iam = session.client("iam")
paginator = iam.get_paginator("list_users")

# Extract only usernames across all pages

for user_name in paginator.paginate().search("Users[].UserName"):
    print(user_name)

This approach handles the underlying token management automatically while reducing memory footprint. See plugins/aws-core/skills/aws-sdk-python-usage/references/pagination.md for complete paginator coverage.

Waiters for Resource State Transitions

Use built-in waiters via client.get_waiter() instead of polling manually with time.sleep(). Waiters implement intelligent backoff strategies and handle transient errors.

bucket_waiter = s3.get_waiter("bucket_exists")
bucket_waiter.wait(
    Bucket="my-new-bucket",
    WaiterConfig={"Delay": 5, "MaxAttempts": 12},
)

Available waiter names and configuration options are listed in plugins/aws-core/skills/aws-sdk-python-usage/references/waiters.md.

Service-Specific Patterns

For S3 operations, consult plugins/aws-core/skills/aws-sdk-python-usage/references/s3.md to implement multipart uploads, presigned URLs, and transfer acceleration correctly. For DynamoDB, reference plugins/aws-core/skills/aws-sdk-python-usage/references/dynamodb.md for batch write patterns and conditional update handling.

When working with large S3 uploads, use the high-level resource interface with automatic multipart switching:

s3_res = session.resource("s3")
obj = s3_res.Bucket("my-bucket").Object("large-file.bin")

with open("large-file.bin", "rb") as data:
    obj.upload_fileobj(data, Config=shared_cfg)

Logging and Observability

Activate debug logging via boto3.set_stream_logger("botocore") or configure the standard library logging module to capture API calls, retries, and response metadata. This is essential for troubleshooting throttling or signature errors in production environments.

Script Structure for Testability

Structure scripts with a single if __name__ == "__main__" entry point that delegates to a main() function. Keep argument parsing and exit-code handling confined to main(). This pattern, enforced in the Agent Toolkit skill guidelines, makes scripts testable and reusable as library modules.

Summary

• Use clients for complete API coverage; use resources only for S3 and DynamoDB convenience operations
• Reuse clients created with explicit botocore.config.Config objects; avoid creating clients in loops
• Handle specific exceptions from client.exceptions rather than generic ClientError where possible
• Use paginators with JMESPath instead of manual NextToken loops
• ** Leverage waiters** for resource state transitions instead of custom polling logic
• Reference service-specific guides for S3 and DynamoDB advanced patterns

Frequently Asked Questions

Should I use boto3 client or resource?

Use low-level clients for most production automation. Clients map directly to AWS API operations and provide complete parameter access. Resources offer a simplified object-oriented interface but only support a subset of services. According to the Agent Toolkit guidance in SKILL.md, prefer clients for infrastructure code and switch to resources only when you specifically need high-level abstractions like S3 multipart uploads or DynamoDB batch operations.

How do I configure retries and timeouts in boto3?

Pass a botocore.config.Config object when creating the client. Set retries.total_max_attempts (not the legacy max_attempts) and specify connect_timeout and read_timeout in seconds. For production workloads, also set max_pool_connections to at least 50 to prevent connection pool exhaustion under concurrent load. These settings are documented in plugins/aws-core/skills/aws-sdk-python-usage/references/configuration.md.

What is the best way to handle pagination in boto3?

Use client.get_paginator(operation_name) combined with JMESPath search. Paginators automatically handle NextToken fields across API calls. Use .search() to extract specific fields from response pages, which reduces memory usage compared to loading full response objects. This pattern is implemented in plugins/aws-core/skills/aws-sdk-python-usage/references/pagination.md.

How do I wait for AWS resources to be ready without writing polling loops?

Use client.get_waiter() with the appropriate waiter name. Waiters like bucket_exists or instance_running implement exponential backoff and handle transient failures. Configure Delay and MaxAttempts in the WaiterConfig dictionary to control the polling interval. See plugins/aws-core/skills/aws-sdk-python-usage/references/waiters.md for the complete list of available waiters per service.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}