How to Configure S3 Checkpoint Upload with s5cmd in Nanotron

Nanotron automatically streams training checkpoints to Amazon S3 by running a background S3Mover process that executes s5cmd commands, configured entirely through the s3_upload section in your YAML configuration file.

The huggingface/nanotron framework provides native support for asynchronous checkpoint persistence to cloud storage without blocking training iterations. Configuring S3 checkpoint upload with s5cmd in Nanotron enables high-performance parallel copying from local scratch directories to S3 buckets while optionally freeing disk space automatically. This functionality is implemented through tight integration between the S3Mover state machine, the DistributedTrainer lifecycle hooks, and the S3UploadArgs configuration dataclass.

Architecture and Key Components

The S3 upload system consists of four core components that coordinate across the training loop:

• S3Mover – A background process manager defined in src/nanotron/s3_checkpoints/s3_mover.py that constructs and executes s5cmd commands, monitors upload progress, and handles local cleanup via a state machine (IDLE → UPLOADING → REMOVING_CHECKPOINT).

• check_path_is_local – A filesystem helper located in src/nanotron/s3_checkpoints/fsspec.py that distinguishes between local POSIX paths and S3 URIs to determine when activation is required.

• DistributedTrainer integration – The trainer class in src/nanotron/trainer.py instantiates S3Mover during post_init(), updates it via post_train_step() after every iteration, and finalizes uploads through post_training().

• S3UploadArgs – The configuration schema defined in src/nanotron/config/config.py that maps YAML parameters to the upload behavior.

Configuration Parameters

All S3-related settings reside under the top-level s3_upload key in your Nanotron YAML configuration. The following fields control the s5cmd integration:

Parameter	Type	Description
upload_s3_path	string	Destination S3 URI (e.g., s3://my-bucket/checkpoints).
remove_after_upload	boolean	If true, delete the local checkpoint directory after successful S3 transfer.
s5cmd_path	string	Absolute path to the s5cmd binary (defaults to s5cmd in $PATH).
s5cmd_numworkers	integer	Number of parallel workers passed to s5cmd via --numworkers.
s5cmd_concurrency	integer	Concurrency level passed via --concurrency.
s5cmd_credentials	string	Optional path to an AWS credentials file for --credentials-file.

How the Upload Process Works

The integration follows a specific lifecycle during distributed training:

1. Initialization – During DistributedTrainer.post_init(), the trainer checks for config.s3_upload. If present, it creates an S3Mover instance with local_path set to config.checkpoints.checkpoints_path and s3_path set to config.s3_upload.upload_s3_path.

2. Rank-based spawning – The trainer sets dummy=True on all ranks where LOCAL_RANK != 0. Only the rank-zero process executes the real s5cmd subprocess; other ranks use S3Mover.DummyPopen to prevent duplicate uploads.

3. Training loop – After every step, post_train_step() calls self.s3_mover.update(). If a checkpoint was just saved and the mover is in the IDLE state, it transitions to UPLOADING and spawns the s5cmd process.

4. Completion handling – Once s5cmd exits successfully, _post_uploading() triggers _start_removing() if remove_after_upload is enabled, deleting the local checkpoint directory to reclaim space.

5. Final synchronization – At the end of training, post_training() invokes self.s3_mover.distributed_wait_for_completion() to block all ranks until the final upload and optional removal finish.

YAML Configuration Example

general:
  project: nanotron-demo
  run: experiment-01

checkpoints:
  checkpoints_path: /scratch/checkpoints
  checkpoint_interval: 500
  save_initial_state: true

s3_upload:
  upload_s3_path: s3://my-bucket/nanotron-checkpoints
  remove_after_upload: true
  s5cmd_path: /opt/s5cmd/bin/s5cmd
  s5cmd_numworkers: 96
  s5cmd_concurrency: 10
  # s5cmd_credentials: /home/user/.aws/credentials

Practical Code Examples

Launching a Trainer with S3 Upload

from nanotron.trainer import DistributedTrainer
from nanotron.config import Config

# Load configuration containing the s3_upload section

config = Config.load_from_yaml("configs/training_s3.yaml")

# Initialize trainer (assumes distributed environment variables are set)

trainer = DistributedTrainer(config_or_config_file=config)

# Train; checkpoints automatically upload to S3 in the background

trainer.train(dataloader_dict)

Direct S3Mover Usage for Manual Uploads

from nanotron.s3_checkpoints import S3Mover
import time

mover = S3Mover(
    local_path="/scratch/checkpoints/step_1000",
    s3_path="s3://my-bucket/backup/step_1000",
    remove_after_upload=True,
    s5cmd_path="/opt/s5cmd/bin/s5cmd",
    s5cmd_numworkers=64,
    s5cmd_concurrency=8,
    dummy=False  # Set True on non-zero ranks only

)

mover.start_uploading()

while mover.state != mover.S3MoverState.IDLE:
    mover.update()
    time.sleep(0.5)

print("Checkpoint successfully persisted to S3")

Verifying Local Path Status

from nanotron.s3_checkpoints import check_path_is_local
from pathlib import Path

assert check_path_is_local(Path("/scratch/checkpoints")) is True
assert check_path_is_local(Path("s3://bucket/prefix")) is False

Distributed Training and Debugging Considerations

Multi-Process Safety – The S3Mover creates a .lock file alongside the first file in a checkpoint directory to prevent race conditions when multiple processes on the same node attempt simultaneous uploads. Only the rank holding the lock executes the s5cmd spawn.

Credentials Management – If your training nodes do not use the default AWS credential chain, specify s5cmd_credentials in the YAML to point to a custom credentials file. The mover automatically appends --credentials-file to the s5cmd invocation.

Monitoring Uploads – Set the environment variable NANOTRON_LOG_LEVEL=DEBUG to capture the full s5cmd command line and stdout/stderr in the Nanotron logs. The S3Mover stores the subprocess output in self.stdout for troubleshooting failed transfers.

Performance Tuning – For high-throughput NVMe-to-S3 transfers, set s5cmd_numworkers to match the number of available CPU cores and s5cmd_concurrency to the number of files written per checkpoint to maximize parallelization.

Summary

• Configure S3 checkpoint upload with s5cmd in Nanotron by adding the s3_upload section to your YAML file, specifying upload_s3_path and s5cmd options.
• The S3Mover class in src/nanotron/s3_checkpoints/s3_mover.py manages the background upload process and local cleanup.
• Only rank zero executes the actual s5cmd command; other ranks use a dummy process to avoid duplicate transfers.
• Set remove_after_upload: true to automatically reclaim local disk space after successful S3 persistence.
• Use s5cmd_numworkers and s5cmd_concurrency to optimize transfer throughput for your hardware configuration.

Frequently Asked Questions

What is s5cmd and why does Nanotron use it instead of the AWS CLI?

s5cmd is a high-performance command-line tool written in Go that executes S3 operations significantly faster than the standard AWS CLI, especially for many small files. Nanotron spawns s5cmd as a subprocess via the S3Mover class to achieve parallel, non-blocking uploads without adding Python GIL contention to the training process.

How does Nanotron prevent every GPU rank from uploading the same checkpoint?

During initialization in src/nanotron/trainer.py, the trainer inspects the LOCAL_RANK environment variable. It passes dummy=True to the S3Mover constructor on all ranks except rank zero. The dummy implementation simulates a successful subprocess exit immediately, ensuring only one physical upload occurs while maintaining API consistency across all processes.

Can I use a custom AWS credentials file or profile with the s5cmd integration?

Yes. Set the optional s5cmd_credentials field in your YAML configuration to the absolute path of your credentials file. The S3Mover automatically includes the --credentials-file argument when constructing the s5cmd command, allowing you to use non-default credential locations or profiles without modifying environment variables.

Why is my local checkpoint not being deleted after the upload completes?

The remove_after_upload flag only triggers deletion after the S3Mover confirms the s5cmd process exited successfully and the state machine transitions to the REMOVING_CHECKPOINT phase. If deletion fails, check that the process has write permissions on the local checkpoint directory and that no other processes hold open file handles on the checkpoint files. Enabling debug logging will reveal the specific error encountered during the removal phase.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}