# AWS SDK Best Practices for Python (boto3): Production Patterns from the Agent Toolkit

> Master AWS SDK best practices for Python with boto3. Learn production patterns for efficient, robust applications using low-level clients, explicit configs, and error handling.

- Repository: [Amazon Web Services/agent-toolkit-for-aws](https://github.com/aws/agent-toolkit-for-aws)
- Tags: best-practices
- Published: 2026-06-26

---

**Use low-level boto3 clients with explicit `botocore.config.Config` objects, reuse clients across your process, catch specific service exceptions, and leverage paginators and waiters instead of manual loops.**

The Agent Toolkit for AWS repository codifies enterprise-grade patterns for writing resilient Python applications with the AWS SDK. Following the guidance in [`plugins/aws-core/skills/aws-sdk-python-usage/SKILL.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/plugins/aws-core/skills/aws-sdk-python-usage/SKILL.md) and its supporting reference documents ensures your boto3 code handles retries, pagination, and error scenarios correctly while maintaining high performance.

## Client vs. Resource: Choosing the Right Abstraction

The AWS SDK for Python offers two distinct interfaces. **Clients** provide complete API coverage with direct mapping to AWS service operations, while **Resources** offer a higher-level object-oriented interface for specific services.

Use **clients** when you need access to the full API surface or when working with services that lack resource support. Switch to **resources** only for supported services like S3 or DynamoDB where you benefit from simplified operations like multipart uploads or batch writes. As documented in [`plugins/aws-core/skills/aws-sdk-python-usage/SKILL.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/plugins/aws-core/skills/aws-sdk-python-usage/SKILL.md), prefer clients for most automation and infrastructure tasks to ensure you can access every API parameter.

## Session Management and Client Configuration

Create an explicit `boto3.Session` only when you need to customize the profile, region, or shared configuration. Reuse a single client per process rather than constructing clients inside loops, which avoids connection pool exhaustion and reduces latency.

Configure clients using `botocore.config.Config` to control retries, timeouts, and connection pools. The Agent Toolkit recommends the newer `total_max_attempts` field (which counts attempts inclusively) rather than the legacy `max_attempts`.

```python
import boto3
from botocore.config import Config

shared_cfg = Config(
    retries={"total_max_attempts": 3, "mode": "adaptive"},
    connect_timeout=5,
    read_timeout=10,
    max_pool_connections=50,
)

session = boto3.Session(profile_name="production", region_name="us-west-2")
s3 = session.client("s3", config=shared_cfg)
dynamodb = session.client("dynamodb", config=shared_cfg)

```

This pattern is detailed in [`plugins/aws-core/skills/aws-sdk-python-usage/references/configuration.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/plugins/aws-core/skills/aws-sdk-python-usage/references/configuration.md), which also covers service-specific settings like S3 addressing styles.

## Error Handling with Specific Exceptions

Catch only specific botocore exceptions using the `client.exceptions` namespace. This prevents masking unexpected errors and allows targeted handling of known failure modes.

```python
def get_bucket_location(bucket_name: str) -> str | None:
    try:
        resp = s3.get_bucket_location(Bucket=bucket_name)
        return resp.get("LocationConstraint")
    except s3.exceptions.NoSuchBucket:
        logger.warning("Bucket %s not found.", bucket_name)
        return None
    # Let unexpected exceptions propagate to the caller

```

Reserve generic `ClientError` handling for top-level exception boundaries only. The specific exception patterns are documented in [`plugins/aws-core/skills/aws-sdk-python-usage/references/error-handling.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/plugins/aws-core/skills/aws-sdk-python-usage/references/error-handling.md).

## Pagination with JMESPath

Never manually loop over `NextToken` or `Marker` fields. Instead, use `client.get_paginator()` combined with JMESPath search expressions to extract exactly the data you need.

```python
iam = session.client("iam")
paginator = iam.get_paginator("list_users")

# Extract only usernames across all pages

for user_name in paginator.paginate().search("Users[].UserName"):
    print(user_name)

```

This approach handles the underlying token management automatically while reducing memory footprint. See [`plugins/aws-core/skills/aws-sdk-python-usage/references/pagination.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/plugins/aws-core/skills/aws-sdk-python-usage/references/pagination.md) for complete paginator coverage.

## Waiters for Resource State Transitions

Use built-in waiters via `client.get_waiter()` instead of polling manually with `time.sleep()`. Waiters implement intelligent backoff strategies and handle transient errors.

```python
bucket_waiter = s3.get_waiter("bucket_exists")
bucket_waiter.wait(
    Bucket="my-new-bucket",
    WaiterConfig={"Delay": 5, "MaxAttempts": 12},
)

```

Available waiter names and configuration options are listed in [`plugins/aws-core/skills/aws-sdk-python-usage/references/waiters.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/plugins/aws-core/skills/aws-sdk-python-usage/references/waiters.md).

## Service-Specific Patterns

For S3 operations, consult [`plugins/aws-core/skills/aws-sdk-python-usage/references/s3.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/plugins/aws-core/skills/aws-sdk-python-usage/references/s3.md) to implement multipart uploads, presigned URLs, and transfer acceleration correctly. For DynamoDB, reference [`plugins/aws-core/skills/aws-sdk-python-usage/references/dynamodb.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/plugins/aws-core/skills/aws-sdk-python-usage/references/dynamodb.md) for batch write patterns and conditional update handling.

When working with large S3 uploads, use the high-level resource interface with automatic multipart switching:

```python
s3_res = session.resource("s3")
obj = s3_res.Bucket("my-bucket").Object("large-file.bin")

with open("large-file.bin", "rb") as data:
    obj.upload_fileobj(data, Config=shared_cfg)

```

## Logging and Observability

Activate debug logging via `boto3.set_stream_logger("botocore")` or configure the standard library `logging` module to capture API calls, retries, and response metadata. This is essential for troubleshooting throttling or signature errors in production environments.

## Script Structure for Testability

Structure scripts with a single `if __name__ == "__main__"` entry point that delegates to a `main()` function. Keep argument parsing and exit-code handling confined to `main()`. This pattern, enforced in the Agent Toolkit skill guidelines, makes scripts testable and reusable as library modules.

## Summary

- **Use clients** for complete API coverage; use resources only for S3 and DynamoDB convenience operations
- **Reuse clients** created with explicit `botocore.config.Config` objects; avoid creating clients in loops
- **Handle specific exceptions** from `client.exceptions` rather than generic `ClientError` where possible
- **Use paginators** with JMESPath instead of manual `NextToken` loops
- ** Leverage waiters** for resource state transitions instead of custom polling logic
- **Reference service-specific guides** for S3 and DynamoDB advanced patterns

## Frequently Asked Questions

### Should I use boto3 client or resource?

**Use low-level clients for most production automation.** Clients map directly to AWS API operations and provide complete parameter access. Resources offer a simplified object-oriented interface but only support a subset of services. According to the Agent Toolkit guidance in [`SKILL.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/SKILL.md), prefer clients for infrastructure code and switch to resources only when you specifically need high-level abstractions like S3 multipart uploads or DynamoDB batch operations.

### How do I configure retries and timeouts in boto3?

**Pass a `botocore.config.Config` object when creating the client.** Set `retries.total_max_attempts` (not the legacy `max_attempts`) and specify `connect_timeout` and `read_timeout` in seconds. For production workloads, also set `max_pool_connections` to at least 50 to prevent connection pool exhaustion under concurrent load. These settings are documented in [`plugins/aws-core/skills/aws-sdk-python-usage/references/configuration.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/plugins/aws-core/skills/aws-sdk-python-usage/references/configuration.md).

### What is the best way to handle pagination in boto3?

**Use `client.get_paginator(operation_name)` combined with JMESPath search.** Paginators automatically handle `NextToken` fields across API calls. Use `.search()` to extract specific fields from response pages, which reduces memory usage compared to loading full response objects. This pattern is implemented in [`plugins/aws-core/skills/aws-sdk-python-usage/references/pagination.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/plugins/aws-core/skills/aws-sdk-python-usage/references/pagination.md).

### How do I wait for AWS resources to be ready without writing polling loops?

**Use `client.get_waiter()` with the appropriate waiter name.** Waiters like `bucket_exists` or `instance_running` implement exponential backoff and handle transient failures. Configure `Delay` and `MaxAttempts` in the `WaiterConfig` dictionary to control the polling interval. See [`plugins/aws-core/skills/aws-sdk-python-usage/references/waiters.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/plugins/aws-core/skills/aws-sdk-python-usage/references/waiters.md) for the complete list of available waiters per service.