How Nydus On-Demand Loading Works for Container Images: RAFS Architecture and Blob Caching Explained

Nydus on-demand loading fetches only the specific data chunks a container actually accesses by using RAFS metadata to map file offsets to remote blob chunks, eliminating full image pulls and reducing cold-start time from minutes to milliseconds.

Nydus, the high-performance container image service developed by the Dragonfly OSS project (dragonflyoss/nydus), eliminates the need to download complete container images before execution. Instead of pulling entire layers, Nydus stores images as a RAFS (Random Access File System) bootstrap metadata file alongside separate blob data files, enabling granular, chunk-level fetching from remote registries or object storage backends only when specific bytes are requested.

Architecture of Nydus On-Demand Loading

Nydus implements on-demand loading through two primary interfaces that converge on the same core logic: the FUSE-based daemon (nydusd) used by the Nydus snapshotter, and the kernel fscache driver with EROFS mounts in bind ondemand mode. Both paths utilize a RAFS device to resolve file offsets into chunk IDs and fetch missing data from configured backends.

Core Components

The on-demand pipeline involves specialized components working in sequence:

• nydus-snapshotter: A containerd plugin that mounts Nydus images and forwards read requests to the user-space daemon
• nydusd: The daemon implementing FUSE and fscache protocols, receiving kernel read requests and coordinating chunk retrieval from remote storage
• blobfs: Located in rafs/src/blobfs/mod.rs, this layer holds the blob_ondemand_cfg configuration and manages blob file operations, opening remote blobs and caching chunks locally
• Backend: Remote storage (registry, OSS, or local filesystem) supplying raw chunk data on first access
• blob-cache: Local directory managed by FsCacheHandler in service/src/fs_cache.rs, storing retrieved chunks for subsequent zero-latency reads

Request Flow Sequence

When a container process reads a file, the following sequence occurs:

1. Mount: The snapshotter mounts the Nydus image via FUSE, or the kernel mounts an EROFS image with fscache enabled
2. Read Intercept: The kernel sends a read request to nydusd through the chosen interface
3. Chunk Resolution: The RAFS device parses the bootstrap to map the file offset to a specific chunk ID and blob file
4. Cache Check: Blobfs checks its in-memory inode-map; on a cache miss, it opens the corresponding blob file
5. Backend Fetch: The daemon issues a range read to the remote backend and writes retrieved bytes to the blob-cache directory
6. Data Delivery: nydusd returns the data to the kernel, which delivers it to the container process
7. Subsequent Accesses: Future reads hit the local blob-cache, serving data with zero network latency

According to the source code in service/src/fs_cache.rs at line 311, the daemon initializes fscache on-demand mode by writing the command bind ondemand to the kernel fscache socket during startup.

Key Implementation Details

The on-demand mechanism relies on specific implementation points across the Nydus codebase in the dragonflyoss/nydus repository.

FSCache Bind Ondemand Integration

In bind ondemand mode, the kernel fscache driver issues open and read requests directly to nydusd, reducing context switches for cached reads. The initialization occurs in service/src/fs_cache.rs:

file.write_all(b"bind ondemand")

This command at line 311 registers the daemon as an on-demand data provider for the kernel's fscache subsystem, enabling the kernel to request specific data ranges on demand rather than prefetching entire files.

BlobFS Configuration Parsing

The blobfs layer stores on-demand configuration in the blob_ondemand_cfg field, defined in rafs/src/blobfs/mod.rs at line 71:

pub blob_ondemand_cfg: String

The BlobOndemandConfig::from_str method (lines 67-68 in the same file) parses the JSON configuration string, validates the RAFS configuration, and ensures the blob cache directory exists before serving requests.

RAFS Chunk Resolution and Backend Fetching

When processing reads, the RAFS device implementation in rafs/src/fs.rs provides the read_at method. This function uses the bootstrap's chunk table to translate file offsets into exact byte ranges within blob files, enabling precise range requests to remote storage.

For fscache-specific operations, service/src/fs_cache.rs implements handle_open_request and handle_open_data_blob. These functions look up blob keys from kernel requests, create FsCacheBlobCache objects (per-blob cache managers), and initiate backend reads when chunks are not present locally.

Configuring Nydus for On-Demand Loading

To enable on-demand loading, operators must configure the daemon with proper RAFS metadata paths and backend connectivity.

BlobFS JSON Configuration

The blobfs layer requires a JSON configuration passed via the blob_ondemand_cfg parameter. This specifies the bootstrap location, local cache directory, and remote backend settings:

{
  "bootstrap_path": "/var/lib/nydus/bootstrap.json",
  "blob_cache_dir": "/var/lib/nydus/blobcache",
  "rafs_conf": {
    "backend_type": "registry",
    "backend_config": { "scheme": "https" },
    "fs_driver": "fuse",
    "mode": "direct"
  }
}

The daemon passes this JSON string to the blob_ondemand_cfg field of rafs::blobfs::Config as defined in rafs/src/blobfs/mod.rs.

Starting nydusd with FSCache

For kernel-integrated on-demand loading using the fscache driver:

./target/release/nydusd \
   --fs-driver fscache \
   --bootstrap /path/to/bootstrap.json \
   --cache-dir /var/lib/nydus/blobcache

During initialization, this configuration writes the bind ondemand command to the fscache device, establishing the on-demand data path through the kernel.

Containerd Integration with Nydus Snapshotter

Using the Nydus snapshotter with containerd automatically enables on-demand loading via FUSE:

nerdctl pull --snapshotter=nydus docker.io/library/alpine:latest-nydus
nerdctl run --rm -ti --snapshotter=nydus docker.io/library/alpine:latest-nydus sh

The snapshotter mounts the image via FUSE, and all container reads trigger the lazy loading pipeline through nydusd.

Programmatic Initialization Example

When embedding Nydus or developing extensions, initialize blobfs programmatically as shown in the dragonflyoss/nydus source:

// Inside nydusd when processing `--fs-driver fscache`
let blob_cfg = r#"
{
  "bootstrap_path": "/opt/nydus/bootstrap.json",
  "blob_cache_dir": "/opt/nydus/cache",
  "rafs_conf": {
    "backend_type": "registry",
    "backend_config": { "scheme": "https" },
    "mode": "direct"
  }
}
"#;

// Build the BlobFs config and start the service
let cfg = rafs::blobfs::Config {
    ps_config: Default::default(),
    blob_ondemand_cfg: blob_cfg.to_string(),
};
let blobfs = rafs::blobfs::BlobFs::new(cfg).expect("init blobfs");
blobfs.import().expect("import blobfs");

This pattern appears in rafs/src/blobfs/mod.rs and demonstrates how the on-demand configuration bridges the RAFS metadata layer with the blob fetching backend.

Summary

• Nydus on-demand loading eliminates full image pulls by fetching only accessed chunks using RAFS metadata and separate blob storage, reducing startup time and network traffic.
• The architecture supports both FUSE-based (user-space) and fscache-based (kernel-integrated) access paths, converging on the same blobfs and RAFS core logic.
• Key implementation files include service/src/fs_cache.rs for fscache integration (including the bind ondemand command at line 311), rafs/src/blobfs/mod.rs for on-demand configuration parsing (lines 67-71), and rafs/src/fs.rs for chunk-to-offset resolution.
• The blob-cache directory stores retrieved chunks locally, ensuring subsequent reads serve from disk with zero network latency.
• The system supports multiple backends including OCI registries and object storage, configured through the blob_ondemand_cfg JSON parameter.

Frequently Asked Questions

How does Nydus map a file read request to a specific chunk in the blob?

Nydus uses the RAFS (Random Access File System) bootstrap metadata to maintain a chunk table that maps file offsets to specific chunk IDs and blob locations. When nydusd receives a read request, the RAFS device implementation in rafs/src/fs.rs executes the read_at method, which consults this chunk table to determine the exact byte range within the remote blob file that must be fetched from the backend.

What is the difference between FUSE and fscache on-demand modes?

The FUSE mode runs nydusd as a user-space filesystem daemon that intercepts all read requests through the kernel's FUSE interface, suitable for containerd snapshotter integration. The fscache (bind ondemand) mode utilizes the kernel's fscache driver with EROFS mounts, where the kernel itself issues open and read requests to nydusd through a dedicated socket. As implemented in service/src/fs_cache.rs at line 311, the daemon writes bind ondemand to register with the kernel, reducing context switches for cached reads and improving performance for frequently accessed data.

Where does Nydus store chunks retrieved through on-demand loading?

Retrieved chunks are stored in the blob-cache directory specified by the blob_cache_dir parameter in the blobfs configuration. The FsCacheHandler in service/src/fs_cache.rs manages this cache, writing fetched data to local storage and tracking cached chunks through an in-memory inode-map in blobfs. Subsequent reads of the same chunk hit this local cache, eliminating network latency and reducing backend load.

Can Nydus on-demand loading work with any container registry?

Yes, Nydus on-demand loading supports multiple backend types including standard OCI registries, Alibaba Cloud OSS, and local filesystems. The backend type is configured in the rafs_conf section of the blobfs JSON configuration through the backend_type and backend_config fields. As long as the backend supports HTTP range requests or equivalent byte-range access methods, Nydus can fetch individual chunks on demand without requiring full layer downloads, enabling efficient lazy loading from any compatible storage backend.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}