How Nydus Implements Lazy Loading for Container Images

Nydus implements lazy loading by splitting container images into metadata (the bootstrap) and data (blob files), fetching only the specific 1 MiB chunks required by running applications via on-demand FUSE or virtio-fs requests.

The dragonflyoss/nydus repository provides a production-ready solution for lazy loading container images, dramatically reducing startup time and network usage by avoiding full image pulls. Instead of downloading complete layers before container execution, Nydus mounts a lightweight metadata index and streams compressed data chunks on demand from remote registries.

The Two-Part Image Structure: Bootstrap and Blobs

Nydus achieves lazy loading through a strict separation of concerns between metadata and data. During the build process, nydus-image or nydusify generates two distinct artifacts:

• Bootstrap: A Merkle-tree structure containing all filesystem inodes, file attributes, and chunk metadata. Each file inode lists the SHA-256 digest and byte offset for every 1 MiB chunk stored in the accompanying blobs.
• Blob files: Compressed data chunks (using LZ4, gzip, or Zstd) that contain the actual file contents, stored separately from the metadata.

As defined in docs/nydus-design.md, this separation allows the Nydus daemon (nydusd) to mount a container filesystem instantly using only the bootstrap, deferring all blob downloads until specific file regions are accessed.

On-Demand Chunk Fetching Mechanism

When a process inside the container reads a file, the kernel forwards the request to the Nydus daemon via FUSE or virtio-fs. The daemon executes the following lookup sequence:

1. Inode resolution: The daemon traverses the Merkle-tree in the bootstrap to locate the target file inode.
2. Chunk mapping: The inode contains a table of chunk descriptors, each referencing a specific 1 MiB range within a blob file by SHA-256 digest and byte offset.
3. Cache check: The daemon verifies whether the required chunk exists in the local blob cache.
4. Remote fetch: If absent, the daemon issues an HTTP range request to pull exactly the compressed bytes needed, decompresses them on the fly, and returns the data to the kernel.

This mechanism is implemented in service/src/fs_service.rs and src/bin/nydusd/main.rs, where the daemon handles filesystem requests without requiring the entire image to be present locally.

ZRAN: Zero-Round-Trip Random Access

For compatibility with standard OCI images, Nydus implements ZRAN (Zero-Round-Trip Random-Access), a lightweight indexing strategy defined in docs/nydus-zran.md. When converting an existing OCI image using nydusify convert --oci-ref, the builder generates a ZRAN index (typically a few KB) that maps chunk offsets to their locations within compressed gzip streams.

The ZRAN implementation in builder/src/core/zran.rs and utils/src/compress/zlib_random.rs stores compression context information, enabling the daemon to request arbitrary chunks via a single HTTP range request without scanning the entire compressed stream. During runtime, storage/src/meta/zran.rs reads this index to facilitate rapid random access to legacy container images without full conversion.

Prefetch Hints for Startup Optimization

To mitigate cold-start latency for critical files, Nydus supports embedded prefetch hints created during image build. The nydusify tool accepts a list of paths via stdin or command-line arguments, which the builder records in the bootstrap's prefetch table at s_prefetch_table_offset.

When the container starts, a background thread in nydusd reads this table from builder/src/core/prefetch.rs and preemptively streams the specified chunks. This ensures that essential binaries and libraries are available locally before the application requests them, reducing the latency of first access while maintaining the bandwidth efficiency of lazy loading.

Lazy Loading Workflow in Practice

The complete lazy loading workflow in dragonflyoss/nydus operates across four distinct phases:

1. Image build: nydusify convert (or nydus-image create) processes the source image into a bootstrap and blob files. With the --oci-ref flag, it also emits a ZRAN index (*.nydus-oci-ref) for OCI compatibility. The builder stores chunk digests and offsets in the bootstrap structure defined in builder/src/core/bootstrap.rs.

2. Container startup: The container runtime (such as containerd with the nydus-snapshotter plugin) mounts the bootstrap through the nydusd daemon. The daemon registers the remote registry as a blob cache source and initializes the filesystem interface.

3. File access: Upon the first read request, the daemon looks up the chunk in the bootstrap, checks local cache, and fetches missing data using ZRAN-accelerated range requests or standard HTTP partial content requests.

4. Background prefetch: If prefetch hints exist, the daemon initiates background downloads of critical chunks immediately after mount, warming the cache for anticipated file accesses.

Implementation Examples

Converting an OCI Image for Lazy Loading

Create a ZRAN-enabled Nydus image that supports lazy pulling from standard registries:

# Install Nydus tools from the release artifacts

sudo install -D -m 755 nydusd nydus-image nydusify /usr/bin

# Convert to Nydus format with ZRAN index

sudo nydusify convert --oci-ref \
    --source docker.io/library/node:19.0 \
    --target docker.io/library/node:19.0-nydus-oci-ref

This command generates a bootstrap, blob files, and a ZRAN index, enabling the runtime to fetch only the Node.js binary and its dependencies when node -v executes.

Running a Container with On-Demand Fetching

Deploy the lazy-loaded image using containerd with the Nydus snapshotter:

# Run with nydus snapshotter and stargz support enabled

sudo nerdctl --snapshotter nydus run --rm -it \
    --enable-stargz \
    docker.io/library/node:19.0-nydus-oci-ref node -v

The snapshotter mounts the bootstrap via nydusd; the daemon pulls individual chunks on demand as the Node.js process accesses its filesystem.

Embedding Prefetch Hints During Build

Optimize startup performance by specifying critical files for background prefetching:

# Provide prefetch hints via stdin

cat <<EOF | nydusify convert \
    --source myapp:latest \
    --target myapp:nydus \
    --prefetch
/usr/bin/myapp
/usr/lib/libssl.so.1.1
EOF

These paths are serialized into the bootstrap's prefetch table by builder/src/core/prefetch.rs, causing the daemon to stream these chunks immediately after container start.

Summary

• Metadata-First Architecture: Nydus splits images into a lightweight bootstrap (Merkle-tree metadata) and separate blob files (compressed data), enabling instant container startup without downloading full images.
• Chunk-Level Granularity: The system fetches data in 1 MiB chunks on demand, using SHA-256-verified range requests to minimize network transfer and storage I/O.
• ZRAN Acceleration: A lightweight ZRAN index enables zero-round-trip random access to standard OCI images, allowing single HTTP range requests per chunk without full stream decompression.
• Prefetch Optimization: Build-time hints embedded in the bootstrap allow background warming of critical files, balancing lazy loading efficiency with startup performance.
• Runtime Integration: The nydusd daemon serves FUSE or virtio-fs requests, coordinating with containerd via the nydus-snapshotter to provide transparent lazy loading.

Frequently Asked Questions

How does Nydus determine which chunks to fetch when an application opens a file?

When the kernel requests a file read via FUSE or virtio-fs, the nydusd daemon consults the bootstrap's Merkle-tree inode structure to locate the file's chunk table. Each chunk descriptor in builder/src/core/bootstrap.rs contains the SHA-256 digest and byte offset within a specific blob. The daemon translates the requested file offset into the corresponding chunk index, checks the local blob cache, and issues an HTTP range request for exactly those compressed bytes if they are not present locally.

What is the difference between standard Nydus images and ZRAN-enabled OCI references?

Standard Nydus images use Nydus-specific blob formats optimized for random access, while ZRAN-enabled OCI references (created with nydusify convert --oci-ref) maintain compatibility with existing registry infrastructure by indexing standard gzip-compressed layers. As implemented in utils/src/compress/zlib_random.rs, the ZRAN index stores compression context information that allows the daemon to seek directly to arbitrary chunk offsets within the gzip stream, enabling lazy pulling without requiring registry support for Nydus-native formats.

Can prefetch hints eliminate cold-start latency entirely?

Prefetch hints reduce but do not eliminate cold-start latency for the specified files. When the daemon initializes in src/bin/nydusd/main.rs, it spawns a background thread that reads the s_prefetch_table_offset from the bootstrap and immediately begins downloading the listed chunks. While this parallelizes data transfer with application initialization, network latency for the first chunk remains. For complete elimination of startup delays, the deployment environment must ensure the blob cache is pre-populated or use sufficiently aggressive prefetching of the entire working set.

Which compression algorithms does Nydus support for lazy-loaded blobs?

Nydus supports LZ4, gzip, and Zstd compression for blob files. The choice of algorithm affects the trade-off between compression ratio and decompression speed during on-demand fetching. For ZRAN-enabled images targeting OCI compatibility, gzip is used for the source layer compression, while Nydus-native images typically use LZ4 or Zstd for optimal decompression performance in the hot path of service/src/fs_service.rs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}