# Performance Considerations for Parallel Workloads with AWS SDK: Best Practices from the Agent Toolkit

> Maximize throughput for parallel AWS SDK workloads. Discover best practices for reusing clients, managing thread pools, and implementing exponential backoff to avoid throttling.

- Repository: [Amazon Web Services/agent-toolkit-for-aws](https://github.com/aws/agent-toolkit-for-aws)
- Tags: performance
- Published: 2026-07-03

---

**Reuse a single boto3 client, bound your thread pool to 10–20 workers, and implement exponential backoff with jitter to maximize throughput while avoiding throttling when running parallel AWS SDK workloads.**

When agents need to issue many AWS API calls at once—such as scanning S3 objects, ingesting data into DynamoDB, or launching EC2 instances—understanding performance considerations for parallel workloads with AWS SDK becomes critical for latency, throughput, and cost optimization. The `aws/agent-toolkit-for-aws` repository provides production-ready skills and scripts that demonstrate battle-tested patterns for parallelism, throttling avoidance, and efficient client reuse. Below are the architectural patterns, implementation details, and concrete code examples drawn directly from the toolkit’s source code.

## Architecture and Core Concepts

### Client Reuse and Connection Pooling

Creating a single `boto3` client or `Session` and reusing it across threads is the foundation of high-performance parallel workloads. The `boto3` library maintains an HTTP connection pool per client; re-creating a client for every request discards this pool and adds unnecessary TLS handshake overhead.

According to the source code in [`skills/specialized-skills/serverless-skills/processing-s3-uploads-with-step-functions/scripts/lambda_function.py`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/specialized-skills/serverless-skills/processing-s3-uploads-with-step-functions/scripts/lambda_function.py), the recommended pattern instantiates the client once at the module level:

```python
s3_client = boto3.client('s3')

```

This pattern ensures that all subsequent API calls reuse the same underlying connection pool, minimizing socket churn and reducing latency.

### Thread-Based Parallelism and Pool Sizing

Use `concurrent.futures.ThreadPoolExecutor` to fire many API calls concurrently, but keep the pool size modest (10–20 workers) unless you know the service’s specific throttling limits. Threads allow the Python GIL to release during I/O operations, enabling true parallel HTTP calls. However, exceeding the service’s requests-per-second quota results in `ThrottlingException` errors that degrade performance.

The [`concurrent-operations.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/concurrent-operations.md) reference in the Durable Functions skill illustrates how to wrap calls with `context.parallel` while maintaining safe concurrency limits.

### Exponential Backoff with Jitter

Every parallel worker must implement a retry policy that backs off exponentially (e.g., `base_delay=0.5` seconds, `max_delay=30` seconds) and adds random jitter. AWS services return retryable exceptions when throttled; aggressive retries without backoff hammer the endpoint and reduce overall throughput.

The [`limits-and-patterns.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/limits-and-patterns.md) reference explicitly notes: "Use parallel workers with backoff on `ServiceUnavailableException`." Similarly, the [`rds_commitment_pricing_analyzer.py`](https://github.com/aws/agent-toolkit-for-aws/blob/main/rds_commitment_pricing_analyzer.py) script implements robust retry logic around `boto3` calls, serving as a template for production agents.

### Batch APIs and Pagination

Prefer batch operations (`batch_write_item`, multipart uploads) over one-by-one calls to reduce per-request overhead. When listing resources, use the SDK’s paginator helpers (`client.get_paginator('list_objects')`) instead of manual loops. Paginators handle continuation tokens efficiently and can be combined with parallel workers to process pages concurrently.

The S3 listing logic in [`lambda_function.py`](https://github.com/aws/agent-toolkit-for-aws/blob/main/lambda_function.py) demonstrates manual pagination, though the SDK’s built-in paginators are recommended for new implementations.

### Resource-Level Limits

Before scaling parallel workloads, consult the service’s rate-limit documentation. For example, S3 supports 3,500 GET requests per second per prefix. The [`msk_sizing.py`](https://github.com/aws/agent-toolkit-for-aws/blob/main/msk_sizing.py) script in the analytics skills demonstrates how to factor in "concurrent I/O operations" when sizing clusters, illustrating the necessity of aligning your worker pool size with actual service capacity.

## Implementation Patterns

### Shared Session Configuration

Initialize a single `boto3.Session` and derive service clients from it to ensure shared connection pools across different AWS services:

```python
import boto3

session = boto3.Session(region_name="us-east-1")
s3 = session.client("s3")
ec2 = session.client("ec2")

```

This guarantees that all service clients use the same underlying HTTP connection pool, reducing socket churn and memory footprint.

### Bounded Thread Pools for I/O Workloads

The default `ThreadPoolExecutor` configuration (`max_workers = CPU × 5`) is often too aggressive for I/O-bound AWS SDK calls. Start with a conservative limit and adjust based on CloudWatch metrics:

```python
from concurrent.futures import ThreadPoolExecutor, as_completed

MAX_WORKERS = 12  # Tune based on service limits

with ThreadPoolExecutor(max_workers=MAX_WORKERS) as executor:
    futures = [executor.submit(s3.list_objects_v2, Bucket=b) for b in buckets]
    for f in as_completed(futures):
        result = f.result()
        # process result...

```

### Retry Logic with Exponential Backoff

Implement a wrapper function that adds jitter and exponential delay for throttling errors:

```python
import random
import time
from botocore.exceptions import ClientError

def retryable_call(fn, *args, **kwargs):
    delay = 0.5
    for attempt in range(6):
        try:
            return fn(*args, **kwargs)
        except ClientError as e:
            if e.response["Error"]["Code"] not in {"Throttling", "ServiceUnavailable"}:
                raise
            time.sleep(delay + random.uniform(0, 0.1))
            delay = min(delay * 2, 30)
    raise RuntimeError("Exceeded retry attempts")

```

This pattern mirrors the implementation found in [`rds_commitment_pricing_analyzer.py`](https://github.com/aws/agent-toolkit-for-aws/blob/main/rds_commitment_pricing_analyzer.py) and aligns with the toolkit’s guidance on handling `ServiceUnavailableException` in parallel workers.

## Code Examples from the Repository

### Parallel S3 Object Listing with Retry Logic

This example from the Agent Toolkit patterns demonstrates client reuse, paginator utilization, and exponential backoff for S3 operations:

```python
import boto3
import random
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
from botocore.exceptions import ClientError

session = boto3.Session(region_name="us-east-1")
s3 = session.client("s3")
buckets = ["bucket-a", "bucket-b", "bucket-c"]

def safe_list(bucket):
    paginator = s3.get_paginator("list_objects_v2")
    try:
        for page in paginator.paginate(Bucket=bucket):
            for obj in page.get("Contents", []):
                print(bucket, obj["Key"])
    except ClientError as e:
        if e.response["Error"]["Code"] in {"Throttling", "ServiceUnavailable"}:
            time.sleep(random.uniform(0.5, 1.0))
            return safe_list(bucket)
        raise

MAX_WORKERS = 6
with ThreadPoolExecutor(max_workers=MAX_WORKERS) as pool:
    futures = {pool.submit(safe_list, b): b for b in buckets}
    for f in as_completed(futures):
        f.result()

```

*Source:* Client creation pattern in [`skills/specialized-skills/serverless-skills/processing-s3-uploads-with-step-functions/scripts/lambda_function.py`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/specialized-skills/serverless-skills/processing-s3-uploads-with-step-functions/scripts/lambda_function.py).

### DynamoDB Batch Writes with Backoff

For high-throughput ingestion, use `batch_write_item` with chunked requests and retry logic:

```python
import boto3
import time
import random
from botocore.exceptions import ClientError

session = boto3.Session()
dynamo = session.client("dynamodb")
TABLE = "MyTable"

def batch_write(items):
    """Write up to 25 items with exponential backoff."""
    delay = 0.5
    for attempt in range(5):
        try:
            dynamo.batch_write_item(RequestItems={TABLE: items})
            return
        except ClientError as e:
            if e.response["Error"]["Code"] not in {"ProvisionedThroughputExceededException", "ThrottlingException"}:
                raise
            time.sleep(delay + random.random() * 0.1)
            delay = min(delay * 2, 20)

# Split payload into 25-item chunks

big_payload = [{"PutRequest": {"Item": {"id": {"S": str(i)}}}} for i in range(200)]
chunks = [big_payload[i:i+25] for i in range(0, len(big_payload), 25)]

from concurrent.futures import ThreadPoolExecutor
with ThreadPoolExecutor(max_workers=8) as pool:
    pool.map(batch_write, chunks)

```

*Source:* Retry logic adapted from [`skills/specialized-skills/database-skills/rds-oss/scripts/rds_commitment_pricing_analyzer.py`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/specialized-skills/database-skills/rds-oss/scripts/rds_commitment_pricing_analyzer.py).

### Async Patterns for Maximum Concurrency

For Python applications requiring higher concurrency than threads allow, use `aiobotocore` to achieve async parallelism:

```python
import asyncio
import aiobotocore
from botocore.exceptions import ClientError

async def list_objects(bucket):
    session = aiobotocore.get_session()
    async with session.create_client('s3', region_name='us-east-1') as s3:
        paginator = s3.get_paginator('list_objects_v2')
        async for page in paginator.paginate(Bucket=bucket):
            for obj in page.get('Contents', []):
                print(bucket, obj['Key'])

async def main():
    buckets = ["b1", "b2", "b3"]
    await asyncio.gather(*(list_objects(b) for b in buckets))

asyncio.run(main())

```

*Note:* While `aiobotocore` is not bundled in the Agent Toolkit, this pattern aligns with the parallel-operation guidance in [`skills/specialized-skills/serverless-skills/aws-lambda-durable-functions/references/concurrent-operations.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/specialized-skills/serverless-skills/aws-lambda-durable-functions/references/concurrent-operations.md).

## Monitoring and Tuning

Enable **CloudWatch metrics** for the specific API operations (`<Service>.<Operation>`) to monitor request rates and throttling spikes. Adjust `MAX_WORKERS` or introduce circuit-breaker logic if throttling percentages exceed 5 percent. For high-throughput S3 transfers, leverage the **TransferManager** (`boto3.s3.transfer`), which automatically handles multipart parallel uploads and retries.

The [`msk_sizing.py`](https://github.com/aws/agent-toolkit-for-aws/blob/main/msk_sizing.py) script further demonstrates how to account for "concurrent I/O operations" when calculating infrastructure capacity, ensuring your parallel workload design aligns with underlying service limits.

## Summary

- **Reuse a single boto3 Session** across all workers to maintain HTTP connection pools and eliminate TLS overhead.
- **Bound thread pools to 10–20 workers** initially, scaling only after verifying service limits and monitoring throttling metrics.
- **Implement exponential backoff with jitter** for all `ThrottlingException` and `ServiceUnavailableException` errors.
- **Prefer batch APIs** like `batch_write_item` and multipart uploads to reduce round-trip latency.
- **Use paginator objects** for listing operations and process pages in parallel workers for efficiency.
- **Monitor CloudWatch** for throttling signals and adjust concurrency accordingly.

## Frequently Asked Questions

### How many threads should I use for parallel AWS SDK calls?

Start with 10–20 threads when using `ThreadPoolExecutor` for I/O-bound AWS SDK calls. The default `max_workers` (CPU count × 5) is typically too high for API-heavy workloads and can trigger throttling. Monitor CloudWatch metrics for your specific service and adjust based on observed `ThrottlingException` rates.

### Why is reusing the boto3 client important for performance?

Reusing a `boto3` client or `Session` maintains the underlying HTTP connection pool, avoiding the overhead of TCP handshakes and TLS negotiations for every request. As shown in [`lambda_function.py`](https://github.com/aws/agent-toolkit-for-aws/blob/main/lambda_function.py), creating the client once at the module level allows parallel threads to share existing connections, significantly reducing latency and CPU usage.

### What is the best way to handle throttling in parallel workloads?

Implement exponential backoff with random jitter (e.g., start at 0.5 seconds, double until 30 seconds maximum) specifically for `ThrottlingException` and `ServiceUnavailableException` errors. The [`rds_commitment_pricing_analyzer.py`](https://github.com/aws/agent-toolkit-for-aws/blob/main/rds_commitment_pricing_analyzer.py) script demonstrates this pattern, and the [`limits-and-patterns.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/limits-and-patterns.md) reference explicitly recommends backoff for parallel workers.

### Should I use threads or async/await for parallel AWS operations?

Both approaches work, but `concurrent.futures.ThreadPoolExecutor` is sufficient for most agent workloads since Python threads release the GIL during I/O. For extreme concurrency (thousands of concurrent operations), use `aiobotocore` with `asyncio` to avoid thread overhead, following the pattern outlined in the Durable Functions concurrent operations documentation.