How Multi-Part Upload Works in S3-Like Object Storage: A Complete Technical Guide

Multi-part upload in S3-like object storage breaks large files into smaller, independently uploadable chunks that are processed in parallel using a unique upload ID, then reassembled into the final object upon completion.

According to the liquidslr/system-design-notes repository, specifically documented in 24. S3-like Object Storage/README.md, multi-part upload is the standard mechanism for handling large object storage efficiently. This design pattern allows clients to upload terabyte-scale objects while maintaining fault tolerance and maximizing throughput.

The Multi-Part Upload Workflow

The implementation described in the source code follows a strict four-phase protocol. Each phase requires specific API interactions between the client and the storage service.

Step 1: Initiating the Upload

The process begins when the client sends a request to initiate a multi-part upload session. As documented in lines 411-416 of 24. S3-like Object Storage/README.md, the data store responds with a unique upload ID that serves as the session identifier for all subsequent operations.

This upload ID must be included in every future request regarding this specific upload operation. The initiation step effectively reserves namespace and prepares the storage backend for incoming data chunks.

Step 2: Uploading Individual Parts

Once the session is established, the client splits the large file into multiple parts. Each part is uploaded independently using the upload ID from Step 1 alongside a sequential part number. According to the source documentation at lines 413-416, after successfully storing a part, the service returns an ETag (typically an MD5 checksum) that uniquely identifies that specific chunk.

This phase supports parallel uploads—the client can upload multiple parts simultaneously over separate HTTP connections, significantly reducing total transfer time for large objects.

Step 3: Completing the Upload

After all parts are uploaded and their ETags collected, the client sends a complete multi-part upload request. This request must contain:

• The upload ID
• A complete list of part numbers
• The corresponding ETags for verification

As noted in lines 415-417 of the design document, the data store then validates all parts and assembles the final object. This reassembly process may take several minutes for very large objects. Once successful, the service returns a confirmation response to the client.

Step 4: Cleanup and Garbage Collection

Following successful reassembly, the storage system removes any temporary part files that are no longer needed. The documentation at lines 418-421 describes a garbage-collection process that reclaims storage space from abandoned or temporary upload parts, ensuring efficient resource utilization.

Key Benefits of Multi-Part Upload Architecture

The design implemented in S3-like storage systems provides three critical advantages over single-request uploads:

• Parallelism: Network bandwidth is utilized more efficiently by uploading multiple parts concurrently rather than sequentially streaming a single large file.
• Resilience: If a specific part fails to upload due to network interruption, only that part requires retry—not the entire multi-gigabyte file.
• Scalability: Large objects are stored internally as a collection of smaller, manageable chunks that integrate naturally with distributed storage nodes and replication mechanisms.

Implementation Example

Below is a Python implementation demonstrating the three primary API calls required for multi-part upload. This code mirrors the protocol described in 24. S3-like Object Storage/README.md and implements the HTTP interactions required by S3-compatible APIs.

import requests
import hashlib

def initiate_multipart(bucket, object_name):
    """Step 1: Initiate upload session and retrieve upload_id."""
    resp = requests.post(
        f"https://s3.example.com/{bucket}/{object_name}?uploads",
        headers={"Authorization": "Bearer <token>"}
    )
    upload_id = resp.json()["UploadId"]
    return upload_id

def upload_part(bucket, object_name, upload_id, part_number, data):
    """Step 2: Upload individual part and capture ETag."""
    md5 = hashlib.md5(data).hexdigest()
    resp = requests.put(
        f"https://s3.example.com/{bucket}/{object_name}"
        f"?partNumber={part_number}&uploadId={upload_id}",
        data=data,
        headers={"Content-MD5": md5, "Authorization": "Bearer <token>"}
    )
    etag = resp.headers["ETag"]
    return {"PartNumber": part_number, "ETag": etag}

def complete_multipart(bucket, object_name, upload_id, parts):
    """Step 3: Finalize upload by providing all part numbers and ETags."""
    payload = {"Parts": parts}
    resp = requests.post(
        f"https://s3.example.com/{bucket}/{object_name}"
        f"?uploadId={upload_id}",
        json=payload,
        headers={"Authorization": "Bearer <token>"}
    )
    return resp.status_code == 200

Summary

• Multi-part upload requires four distinct phases: initiation, parallel part uploads, completion/assembly, and garbage collection.
• The upload ID returned during initiation serves as the session identifier for all part uploads.
• Each uploaded part returns an ETag (MD5 checksum) that must be provided during the completion phase for integrity verification.
• This architecture enables parallel uploads, supports fine-grained retry logic, and scales efficiently across distributed storage nodes as detailed in liquidslr/system-design-notes.

Frequently Asked Questions

What is the primary purpose of multi-part upload in object storage?

Multi-part upload exists to solve the challenges of transferring large files over unreliable networks. By dividing files into smaller chunks that upload independently, the system eliminates the need to restart massive transfers from scratch when network interruptions occur. Additionally, it enables parallel uploading, which saturates available bandwidth more effectively than sequential streaming.

What happens if one part fails during a multi-part upload?

If a specific part fails to upload, only that individual part needs to be retried—not the entire file. The client retains the upload ID and can reattempt uploading the failed part number with fresh data. This granular error recovery is one of the primary resilience benefits of the multi-part architecture described in the system-design notes.

Are there size limitations for individual parts in multi-part uploads?

While the specific repository documentation focuses on the protocol flow, S3-compatible implementations typically enforce minimum part sizes (usually 5 MB, except for the final part) and maximum part counts (typically 10,000 parts). Large files must be divided such that no individual part exceeds the service's maximum part size limit, often 5 GB per part.

How does multi-part upload differ from single-request upload?

Single-request uploads transmit the entire object in one HTTP PUT operation, which becomes impractical for files larger than a few gigabytes due to timeout risks and memory constraints. Multi-part upload, as implemented in 24. S3-like Object Storage/README.md, streams smaller chunks over multiple connections, supports pause/resume functionality, and allows the server to verify each chunk independently via ETag checksums before final assembly.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}