How to Handle Streaming Responses in AWS SDK for JavaScript v3 to Prevent Socket Exhaustion

Always consume, pipe, or explicitly discard streaming response bodies in AWS SDK for JavaScript v3 to prevent TCP sockets from remaining open in the connection pool, which causes EMFILE errors and process hangs in long-running Node.js applications.

The AWS SDK for JavaScript v3 streams response bodies for operations like GetObjectCommand to improve memory efficiency. However, if you leave these streams unread, the underlying TCP socket stays occupied in the connection pool. According to the aws/agent-toolkit-for-aws repository, this leads to socket exhaustion that can crash Lambda functions, EC2 services, and containerized applications. The following patterns from the toolkit's core skills documentation ensure you properly manage stream lifecycle and socket allocation.

Why Unconsumed Streams Exhaust Sockets

Each AWS service client maintains a pool of TCP connections to AWS endpoints. When the SDK returns a streaming Body, the connection remains attached to that stream until the data is fully read or the stream is explicitly destroyed. If you ignore the stream, the socket stays in ESTABLISHED state indefinitely, eventually triggering EMFILE (too many open files) errors. This is particularly dangerous in long-running processes or high-throughput Lambda containers where sockets accumulate across invocations.

Always Read or Discard Streaming Bodies

You must always drain the stream, pipe it to a destination, or explicitly cancel it. The aws/agent-toolkit-for-aws documentation in skills/core-skills/aws-sdk-js-v3-usage/references/effective-practices.md emphasizes that merely calling send() without handling the Body leaves the socket open.

Transform to Byte Array

For small objects, convert the stream to a buffer to ensure complete consumption:

import { S3Client, GetObjectCommand } from "@aws-sdk/client-s3";

const s3 = new S3Client({ region: "us-east-1" });

async function downloadObject(bucket, key) {
  const { Body } = await s3.send(new GetObjectCommand({ Bucket: bucket, Key: key }));
  // Fully drains the stream and releases the socket
  const data = await Body.transformToByteArray();
  console.log(`Downloaded ${data.length} bytes`);
}

Explicitly Destroy Unneeded Streams

If you only need metadata, cancel the stream immediately:

async function deleteObject(bucket, key) {
  const { Body } = await s3.send(new GetObjectCommand({ Bucket: bucket, Key: key }));
  // Explicitly destroy/cancel to free the socket before proceeding
  await (Body.destroy?.() ?? Body.cancel?.());
  // Continue with logic that doesn't need the body data
}

Configure the Socket Pool for Concurrency

The default NodeHttpHandler creates a new socket for every concurrent request. Configure maxSockets to match your application's concurrency level to prevent both serialization and over-allocation. As documented in skills/core-skills/aws-sdk-js-v3-usage/references/performance.md, use the requestHandler option to pass a custom HTTPS agent:

import { S3Client } from "@aws-sdk/client-s3";
import { NodeHttpHandler } from "@smithy/node-http-handler";
import https from "https";

const s3 = new S3Client({
  requestHandler: new NodeHttpHandler({
    httpsAgent: new https.Agent({
      keepAlive: true,
      maxSockets: 50,  // Match your expected concurrency
    }),
  }),
});

Key parameters:

• maxSockets: Limits parallel connections to the same host; too few causes serialization, too many risks EMFILE
• keepAlive: Enables TCP connection reuse across requests, reducing latency

Reuse Clients to Share Socket Pools

Creating a new client per request initializes a fresh socket pool each time, defeating connection reuse and quickly exhausting file descriptors. Instantiate one client per region and credential pair, then share it across all operations. The skills/core-skills/aws-sdk-js-v3-usage/references/performance.md file demonstrates this pattern for sharing both credentials and the underlying socket pool.

Avoid Streaming Deadlocks

When maxSockets is limited (especially to 1), awaiting a request before consuming its stream can deadlock your application. The client waits for a socket to send the request, but if a previous request's stream is still holding the only socket, you create a circular wait.

Incorrect pattern (causes deadlock):

// DON'T: Awaiting sequentially with limited sockets
const get = await s3.send(new GetObjectCommand({...}));  // Holds socket
const put = await s3.send(new PutObjectCommand({ Body: get.Body }));  // Waits for same socket

Correct pattern (piping without intermediate await):

async function copyObject(srcBucket, srcKey, dstBucket, dstKey) {
  // Start both requests without awaiting individually
  const get = s3.send(new GetObjectCommand({ Bucket: srcBucket, Key: srcKey }));
  const put = s3.send(new PutObjectCommand({ 
    Bucket: dstBucket, 
    Key: dstKey, 
    Body: get.Body  // Pipe directly without awaiting get first
  }));
  // Await together once pipes are established
  await Promise.all([get, put]);
}

Handle Large Uploads with Multipart Streams

For streams of unknown size or large files, use @aws-sdk/lib-storage Upload instead of PutObjectCommand. This helper manages multipart uploads with configurable concurrency and part sizes, internally handling socket usage efficiently. Reference the implementation in skills/core-skills/aws-sdk-js-v3-usage/references/s3.md:

import { Upload } from "@aws-sdk/lib-storage";

async function uploadLargeStream(bucket, key, readableStream) {
  const upload = new Upload({
    client: s3,                     // Reuse the shared client
    params: { Bucket: bucket, Key: key, Body: readableStream },
    queueSize: 4,                  // Parallel part uploads
    partSize: 5 * 1024 * 1024,     // 5 MiB minimum per part
  });

  upload.on("httpUploadProgress", (progress) => console.log(progress));
  await upload.done();             // Blocks until complete, properly managing sockets
}

Summary

• Always consume streaming bodies using transformToByteArray(), piping, or explicit destroy()/cancel() calls to release TCP sockets back to the pool
• Configure maxSockets in NodeHttpHandler to match your concurrency, preventing both serialization and EMFILE exhaustion
• Reuse S3 clients across requests to share socket pools and avoid creating new connections per operation
• Avoid awaiting stream sources before establishing pipes when socket limits are low to prevent deadlocks
• Use Upload from @aws-sdk/lib-storage for large or unknown-size streams to manage multipart uploads and socket allocation efficiently

Frequently Asked Questions

What happens if I don't read a streaming response in AWS SDK for JavaScript v3?

The TCP socket remains open and occupied in the connection pool indefinitely. Since the SDK uses keep-alive by default, these lingering sockets accumulate until the process hits the operating system's file descriptor limit (EMFILE), causing subsequent requests to hang or fail.

How do I configure maxSockets in AWS SDK for JavaScript v3?

Pass a custom NodeHttpHandler to the client's requestHandler option with an https.Agent configured with your desired maxSockets value. This controls the maximum number of concurrent connections to each AWS endpoint, balancing throughput against resource usage.

Can I reuse the same S3 client across multiple requests?

Yes, and you should. Create one S3Client instance per region and credential pair, then reuse it for all operations. This shares the underlying HTTP agent and socket pool across requests, significantly reducing connection overhead and preventing socket exhaustion.

Why does my Lambda function hang when processing large S3 downloads?

Lambda functions often hang when maxSockets is set to 1 and code awaits a GetObjectCommand before piping the body to another request. This creates a deadlock where the new request waits for the socket held by the unfinished stream. Always start both requests before awaiting their promises, or ensure streams are fully consumed before making subsequent SDK calls.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}