How Nydus Optimizes I/O Operations for Container Workloads: 3 Core Techniques Explained

Nydus optimizes I/O operations for container workloads by merging image layers into a single RAFS filesystem tree, batching prefetch requests into merged backend operations, and amplifying small user reads into larger chunk-aligned requests.

The dragonflyoss/nydus repository implements a container image acceleration framework that eliminates traditional overlayfs bottlenecks through three complementary mechanisms. These techniques operate at different layers of the image stack to reduce startup latency, minimize storage backend round-trips, and improve runtime read performance.

Merged RAFS Filesystem Tree

Instead of mounting each image layer with overlayfs, Nydus builds a single RAFS superblock that contains the merged directory hierarchy of all layers. This eliminates the overlayfs "copy-up" indirection and reduces filesystem lookup overhead.

In builder/src/merge.rs, the Merger::merge implementation loads per-layer bootstraps, checks compatibility, and overlays directory entries according to overlayfs semantics. The result is a single bootstrap file representing the entire image:

// Conceptual flow from builder/src/merge.rs
// Merger::merge loads sources (per-layer bootstraps) and creates one merged tree

When a container runtime mounts this merged tree, open, stat, and path lookups traverse only one filesystem layer instead of multiple overlayfs layers. This reduces metadata operations significantly during container startup.

Prefetch and Merged Backend Requests

Nydus reduces network round-trip latency by merging many small chunk reads into single larger backend requests. The prefetch subsystem uses asynchronous workers that group consecutive BlobIoDesc objects into merged requests bounded by a configurable merging_size.

The implementation lives in storage/src/cache/filecache/mod.rs, specifically in generate_merged_requests_for_prefetch (lines 582-586). This function constructs merged requests where mr.blob_offset and mr.blob_size represent the aggregated read operation sent to the storage backend in a single I/O.

Configure prefetch behavior in the RAFS configuration:

{
  "fs_prefetch": {
    "enable": true,
    "threads_count": 4,
    "merging_size": 131072,
    "bandwidth_rate": 10485760
  }
}

• threads_count: Creates background prefetch workers
• merging_size: Maximum size of a merged read request (bytes)
• bandwidth_rate: Global token-bucket limit to prevent starving foreground I/O

Rate limiting is enforced in storage/src/cache/worker.rs, where AsyncPrefetchMessage::RateLimiter updates the token bucket before each merged request.

User I/O Amplification

For small reads that would otherwise trigger numerous tiny backend fetches, Nydus implements read-ahead amplification that expands the request window to the next chunk boundary.

In rafs/src/fs.rs (lines 45-66), the amplify_user_io path in the read routine calculates a window extending to the next chunk boundary, respecting the user_io_batch_size configuration. When required chunks are not cached, the daemon reads the entire amplified window from storage, caching extra data for subsequent reads.

This transforms many small random reads into fewer, larger sequential reads, improving both latency and throughput for typical container workloads that read many small files during startup.

Practical Configuration Examples

Building a Merged Nydus Image

Convert an OCI image into a Nydus image with embedded prefetch metadata:

nydusify \
  --source docker://busybox:latest \
  --target-dir ./nydus-image \
  --prefetch-policy fs \
  --prefetch-files ./preload.txt

The --prefetch-policy fs flag embeds a prefetch table in the bootstrap, while the builder creates the merged RAFS tree via Merger::merge in builder/src/merge.rs.

Running the Nydus Daemon with I/O Optimizations

Start nydusd with prefetch enabled and optimized batch sizes:

nydusd \
  --config ./nydus-config.json \
  --backend-type oss \
  --blobcache /var/cache/nydus \
  --fs-prefetch.enable=true \
  --fs-prefetch.threads_count=4 \
  --fs-prefetch.merging_size=131072 \
  --fs-prefetch.bandwidth_rate=10485760

These settings activate the background workers that call generate_merged_requests_for_prefetch and respect the bandwidth limits defined in storage/src/cache/worker.rs.

Container Runtime Integration

Deploy with containerd-nydus-snapshotter to leverage the optimizations:

nerdctl run --snapshotter=nydus \
  docker.io/library/busybox:latest ls /

The snapshotter mounts the single merged RAFS filesystem, automatically benefiting from the prefetch merging and user I/O amplification logic.

Summary

• Merged RAFS tree (builder/src/merge.rs): Eliminates overlayfs layer stitching by combining all layers into one bootstrap, reducing metadata lookups.
• Prefetch merging (storage/src/cache/filecache/mod.rs): Batches small chunk reads into larger backend requests up to merging_size, cutting network round-trips.
• User I/O amplification (rafs/src/fs.rs): Expands small reads to chunk boundaries, converting random I/O into sequential reads with better cache utilization.
• Rate limiting (storage/src/cache/worker.rs): Prevents background prefetch from starving foreground container I/O through token-bucket bandwidth control.

Frequently Asked Questions

How does Nydus differ from overlayfs for container I/O?

Nydus builds a single merged filesystem tree during image construction rather than stacking layers at runtime. According to the source code in builder/src/merge.rs, the Merger::merge function creates one RAFS superblock containing the entire directory hierarchy, eliminating the copy-up operations and multi-layer lookups that slow down overlayfs.

What is the optimal merging_size for prefetch operations?

The optimal merging_size depends on your storage backend latency and bandwidth. As implemented in storage/src/cache/filecache/mod.rs, the generate_merged_requests_for_prefetch function caps individual requests at this value (default typically 128KB). Higher values reduce backend request counts but increase memory pressure; 131072 bytes (128KB) provides a balanced starting point for most object storage backends.

How does user I/O amplification affect cache hit rates?

User I/O amplification improves cache hit rates by fetching additional data beyond the immediate request window. When the RAFS v5 implementation in rafs/src/fs.rs detects a small uncached read, the amplify_user_io logic extends the fetch to the next chunk boundary. This pre-populates the blob cache with adjacent data likely to be accessed next, reducing subsequent backend requests for sequential read patterns common in container startup.

Can Nydus optimize I/O for existing OCI images without rebuilding?

While the full optimization requires converting to the Nydus format using nydusify, the dragonflyoss/nydus project supports OCI-compatible conversion that preserves image content. The conversion process in builder/src/merge.rs generates the merged RAFS tree and can embed prefetch hints based on file access patterns, enabling I/O optimizations without modifying the original application code.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}