How to Handle Paginated API Responses with the AWS SDK for JavaScript v3

Use the built-in paginate* async generator functions exported by each @aws-sdk/client-* package to automatically manage NextToken values, iterating over pages with for await...of loops instead of manual token tracking.

Handling paginated API responses with the AWS SDK for JavaScript v3 requires understanding the async generator pattern introduced in the modular v3 architecture. The aws/agent-toolkit-for-aws repository provides comprehensive guidance on using these paginators across services like DynamoDB, S3, and EC2. Rather than manually extracting and re-injecting NextToken or Marker values into subsequent requests, the SDK exposes dedicated functions that handle the entire request lifecycle automatically.

Understanding the Paginator Architecture

The v3 SDK generates paginator utilities for every service operation that supports pagination. According to the skill documentation in skills/core-skills/aws-sdk-js-v3-usage/SKILL.md, these functions follow a consistent naming convention: paginate<OperationName> (e.g., paginateListTables, paginateScan).

Each paginator accepts a configuration object containing the service client instance and returns an async iterator. The SDK manages all middleware, request signing, retry logic, and token extraction internally, yielding complete response pages until no continuation token remains.

Implementing Paginators in Your Code

Importing from Service Packages

Import the paginator directly from the service client package rather than instantiating command objects manually in a loop. The AWS SDK for JavaScript v3 exports these utilities from the root of each service package:

• @aws-sdk/client-dynamodb exports paginateListTables
• @aws-sdk/lib-dynamodb exports paginateScan for document client operations
• @aws-sdk/client-s3 exports paginateListObjectsV2
• @aws-sdk/client-ec2 exports paginateDescribeInstances

Basic Iteration Pattern

Pass an object containing the client instance and the command input parameters. The paginator returns an async generator compatible with for await...of syntax.

// List all DynamoDB tables using the paginator
import { DynamoDBClient, paginateListTables } from "@aws-sdk/client-dynamodb";

const client = new DynamoDBClient({ region: "us-east-1" });

const allTables = [];
for await (const page of paginateListTables({ client }, {})) {
  // page.TableNames contains up to 100 names per page
  allTables.push(...page.TableNames);
}
console.log("All tables:", allTables);

Service-Specific Pagination Examples

DynamoDB: Listing Tables and Scanning Data

The DynamoDB reference in skills/core-skills/aws-sdk-js-v3-usage/references/dynamodb.md demonstrates both low-level client and document client pagination patterns.

Listing tables with the base client:

import { DynamoDBClient, paginateListTables } from "@aws-sdk/client-dynamodb";

const client = new DynamoDBClient({ region: "us-east-1" });

const allTables = [];
for await (const page of paginateListTables({ client }, {})) {
  allTables.push(...page.TableNames);
}

Scanning large tables with the document client:

When working with the document client from @aws-sdk/lib-dynamodb, use paginateScan to handle large datasets automatically:

import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
import { paginateScan } from "@aws-sdk/lib-dynamodb";

const client = new DynamoDBClient({ region: "us-east-1" });

for await (const page of paginateScan(
  { client }, 
  { TableName: "BigTable", Limit: 200 }
)) {
  // page.Items holds the items for this page
  console.log("Page items:", page.Items);
}

S3: Listing Bucket Objects

The S3 reference documentation in skills/core-skills/aws-sdk-js-v3-usage/references/s3.md covers pagination alongside presigned URLs and multipart uploads. Use paginateListObjectsV2 to traverse large buckets:

import { S3Client, paginateListObjectsV2 } from "@aws-sdk/client-s3";

const s3 = new S3Client({ region: "us-east-1" });

for await (const page of paginateListObjectsV2(
  { client: s3 }, 
  { Bucket: "my-bucket" }
)) {
  for (const obj of page.Contents ?? []) {
    console.log("Object key:", obj.Key);
  }
}

EC2: Describing Instances

For EC2 operations that return paginated results, import paginateDescribeInstances from @aws-sdk/client-ec2:

import { EC2Client, paginateDescribeInstances } from "@aws-sdk/client-ec2";

const ec2 = new EC2Client({ region: "us-east-1" });

const instances = [];
for await (const page of paginateDescribeInstances({ client: ec2 }, {})) {
  instances.push(...page.Reservations.flatMap(r => r.Instances));
}
console.log(`Found ${instances.length} instances`);

Best Practices for Pagination Performance

Stream results instead of buffering. While the examples above collect results into arrays for demonstration, processing each page immediately reduces memory pressure for large datasets:

for await (const page of paginateScan({ client }, { TableName: "LargeTable" })) {
  await processBatch(page.Items); // Process and release memory per page
}

Respect service-specific limits. The paginator automatically handles the NextToken field, but you can still specify Limit parameters to control page size. For example, setting Limit: 200 in DynamoDB operations requests exactly 200 items per page from the service.

Reuse client instances. The skills/core-skills/aws-sdk-js-v3-usage/references/clients.md file emphasizes that paginators accept the client reference without modification, allowing you to maintain configured retry policies and connection pools across multiple paginated operations.

Summary

• Import paginate* functions from @aws-sdk/client-<service> packages to eliminate manual NextToken handling when working with paginated API responses with the AWS SDK for JavaScript v3.
• Iterate using for await...of loops over the async generator returned by paginator functions.
• Pass minimal configuration—only the client instance and command input object—to let the SDK manage middleware, signing, and automatic retries.
• Process pages individually rather than accumulating all data into memory for better performance with large datasets.
• Reference the aws/agent-toolkit-for-aws repository at skills/core-skills/aws-sdk-js-v3-usage/SKILL.md for the complete paginator pattern specification and service-specific examples.

Frequently Asked Questions

What is the difference between manual pagination and using paginate* functions?

Manual pagination requires you to extract the NextToken or Marker from each response and manually construct the subsequent request. The paginate* functions handle this automatically by inspecting the response, extracting the continuation token, and injecting it into the next request until no more pages remain. This eliminates boilerplate code and reduces bugs related to token management.

How do I handle errors when using AWS SDK paginators?

Errors propagate normally through the async iteration. Wrap your for await...of loop in a standard try-catch block to capture service exceptions, network timeouts, or retry exhaustion. The paginator respects the retry configuration defined on your client instance, as documented in skills/core-skills/aws-sdk-js-v3-usage/references/clients.md.

Can I set a maximum number of pages when using paginate* functions?

While the paginator itself continues until exhaustion, you can break out of the for await...of loop early using standard control flow. For example, count iterations and break after reaching your desired limit, or use array methods like take if converting the async iterator to a stream. The SDK does not provide a built-in maxPages parameter in the paginator configuration object.

Do paginators work with the DynamoDB Document Client?

Yes, but you must import paginators from @aws-sdk/lib-dynamodb rather than @aws-sdk/client-dynamodb. Functions like paginateScan and paginateQuery in the document client library handle the attribute value marshalling automatically, yielding native JavaScript objects instead of the DynamoDB wire format. See the DynamoDB reference at skills/core-skills/aws-sdk-js-v3-usage/references/dynamodb.md for concrete examples.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}