# How Nydus On-Demand Loading Works for Container Images: RAFS Architecture and Blob Caching Explained

> Discover how Nydus on-demand loading optimizes container images. Learn about RAFS architecture and blob caching to fetch only needed data chunks, slashing cold-start times.

- Repository: [dragonflyoss/nydus](https://github.com/dragonflyoss/nydus)
- Tags: internals
- Published: 2026-02-28

---

**Nydus on-demand loading fetches only the specific data chunks a container actually accesses by using RAFS metadata to map file offsets to remote blob chunks, eliminating full image pulls and reducing cold-start time from minutes to milliseconds.**

Nydus, the high-performance container image service developed by the Dragonfly OSS project (dragonflyoss/nydus), eliminates the need to download complete container images before execution. Instead of pulling entire layers, Nydus stores images as a RAFS (Random Access File System) bootstrap metadata file alongside separate blob data files, enabling granular, chunk-level fetching from remote registries or object storage backends only when specific bytes are requested.

## Architecture of Nydus On-Demand Loading

Nydus implements on-demand loading through two primary interfaces that converge on the same core logic: the **FUSE-based daemon** (`nydusd`) used by the Nydus snapshotter, and the **kernel fscache driver** with EROFS mounts in bind ondemand mode. Both paths utilize a RAFS device to resolve file offsets into chunk IDs and fetch missing data from configured backends.

### Core Components

The on-demand pipeline involves specialized components working in sequence:

- **nydus-snapshotter**: A containerd plugin that mounts Nydus images and forwards read requests to the user-space daemon
- **nydusd**: The daemon implementing FUSE and fscache protocols, receiving kernel read requests and coordinating chunk retrieval from remote storage
- **blobfs**: Located in [`rafs/src/blobfs/mod.rs`](https://github.com/dragonflyoss/nydus/blob/main/rafs/src/blobfs/mod.rs), this layer holds the `blob_ondemand_cfg` configuration and manages blob file operations, opening remote blobs and caching chunks locally
- **Backend**: Remote storage (registry, OSS, or local filesystem) supplying raw chunk data on first access
- **blob-cache**: Local directory managed by `FsCacheHandler` in [`service/src/fs_cache.rs`](https://github.com/dragonflyoss/nydus/blob/main/service/src/fs_cache.rs), storing retrieved chunks for subsequent zero-latency reads

### Request Flow Sequence

When a container process reads a file, the following sequence occurs:

1. **Mount**: The snapshotter mounts the Nydus image via FUSE, or the kernel mounts an EROFS image with fscache enabled
2. **Read Intercept**: The kernel sends a read request to `nydusd` through the chosen interface
3. **Chunk Resolution**: The RAFS device parses the bootstrap to map the file offset to a specific chunk ID and blob file
4. **Cache Check**: Blobfs checks its in-memory inode-map; on a cache miss, it opens the corresponding blob file
5. **Backend Fetch**: The daemon issues a range read to the remote backend and writes retrieved bytes to the blob-cache directory
6. **Data Delivery**: `nydusd` returns the data to the kernel, which delivers it to the container process
7. **Subsequent Accesses**: Future reads hit the local blob-cache, serving data with zero network latency

According to the source code in [`service/src/fs_cache.rs`](https://github.com/dragonflyoss/nydus/blob/main/service/src/fs_cache.rs) at line 311, the daemon initializes fscache on-demand mode by writing the command `bind ondemand` to the kernel fscache socket during startup.

## Key Implementation Details

The on-demand mechanism relies on specific implementation points across the Nydus codebase in the dragonflyoss/nydus repository.

### FSCache Bind Ondemand Integration

In bind ondemand mode, the kernel fscache driver issues open and read requests directly to `nydusd`, reducing context switches for cached reads. The initialization occurs in [`service/src/fs_cache.rs`](https://github.com/dragonflyoss/nydus/blob/main/service/src/fs_cache.rs):

```rust
file.write_all(b"bind ondemand")

```

This command at line 311 registers the daemon as an on-demand data provider for the kernel's fscache subsystem, enabling the kernel to request specific data ranges on demand rather than prefetching entire files.

### BlobFS Configuration Parsing

The blobfs layer stores on-demand configuration in the `blob_ondemand_cfg` field, defined in [`rafs/src/blobfs/mod.rs`](https://github.com/dragonflyoss/nydus/blob/main/rafs/src/blobfs/mod.rs) at line 71:

```rust
pub blob_ondemand_cfg: String

```

The `BlobOndemandConfig::from_str` method (lines 67-68 in the same file) parses the JSON configuration string, validates the RAFS configuration, and ensures the blob cache directory exists before serving requests.

### RAFS Chunk Resolution and Backend Fetching

When processing reads, the RAFS device implementation in [`rafs/src/fs.rs`](https://github.com/dragonflyoss/nydus/blob/main/rafs/src/fs.rs) provides the `read_at` method. This function uses the bootstrap's chunk table to translate file offsets into exact byte ranges within blob files, enabling precise range requests to remote storage.

For fscache-specific operations, [`service/src/fs_cache.rs`](https://github.com/dragonflyoss/nydus/blob/main/service/src/fs_cache.rs) implements `handle_open_request` and `handle_open_data_blob`. These functions look up blob keys from kernel requests, create `FsCacheBlobCache` objects (per-blob cache managers), and initiate backend reads when chunks are not present locally.

## Configuring Nydus for On-Demand Loading

To enable on-demand loading, operators must configure the daemon with proper RAFS metadata paths and backend connectivity.

### BlobFS JSON Configuration

The blobfs layer requires a JSON configuration passed via the `blob_ondemand_cfg` parameter. This specifies the bootstrap location, local cache directory, and remote backend settings:

```json
{
  "bootstrap_path": "/var/lib/nydus/bootstrap.json",
  "blob_cache_dir": "/var/lib/nydus/blobcache",
  "rafs_conf": {
    "backend_type": "registry",
    "backend_config": { "scheme": "https" },
    "fs_driver": "fuse",
    "mode": "direct"
  }
}

```

The daemon passes this JSON string to the `blob_ondemand_cfg` field of `rafs::blobfs::Config` as defined in [`rafs/src/blobfs/mod.rs`](https://github.com/dragonflyoss/nydus/blob/main/rafs/src/blobfs/mod.rs).

### Starting nydusd with FSCache

For kernel-integrated on-demand loading using the fscache driver:

```bash
./target/release/nydusd \
   --fs-driver fscache \
   --bootstrap /path/to/bootstrap.json \
   --cache-dir /var/lib/nydus/blobcache

```

During initialization, this configuration writes the bind ondemand command to the fscache device, establishing the on-demand data path through the kernel.

### Containerd Integration with Nydus Snapshotter

Using the Nydus snapshotter with containerd automatically enables on-demand loading via FUSE:

```bash
nerdctl pull --snapshotter=nydus docker.io/library/alpine:latest-nydus
nerdctl run --rm -ti --snapshotter=nydus docker.io/library/alpine:latest-nydus sh

```

The snapshotter mounts the image via FUSE, and all container reads trigger the lazy loading pipeline through `nydusd`.

### Programmatic Initialization Example

When embedding Nydus or developing extensions, initialize blobfs programmatically as shown in the dragonflyoss/nydus source:

```rust
// Inside nydusd when processing `--fs-driver fscache`
let blob_cfg = r#"
{
  "bootstrap_path": "/opt/nydus/bootstrap.json",
  "blob_cache_dir": "/opt/nydus/cache",
  "rafs_conf": {
    "backend_type": "registry",
    "backend_config": { "scheme": "https" },
    "mode": "direct"
  }
}
"#;

// Build the BlobFs config and start the service
let cfg = rafs::blobfs::Config {
    ps_config: Default::default(),
    blob_ondemand_cfg: blob_cfg.to_string(),
};
let blobfs = rafs::blobfs::BlobFs::new(cfg).expect("init blobfs");
blobfs.import().expect("import blobfs");

```

This pattern appears in [`rafs/src/blobfs/mod.rs`](https://github.com/dragonflyoss/nydus/blob/main/rafs/src/blobfs/mod.rs) and demonstrates how the on-demand configuration bridges the RAFS metadata layer with the blob fetching backend.

## Summary

- **Nydus on-demand loading** eliminates full image pulls by fetching only accessed chunks using RAFS metadata and separate blob storage, reducing startup time and network traffic.
- The architecture supports both **FUSE-based** (user-space) and **fscache-based** (kernel-integrated) access paths, converging on the same blobfs and RAFS core logic.
- Key implementation files include [`service/src/fs_cache.rs`](https://github.com/dragonflyoss/nydus/blob/main/service/src/fs_cache.rs) for fscache integration (including the `bind ondemand` command at line 311), [`rafs/src/blobfs/mod.rs`](https://github.com/dragonflyoss/nydus/blob/main/rafs/src/blobfs/mod.rs) for on-demand configuration parsing (lines 67-71), and [`rafs/src/fs.rs`](https://github.com/dragonflyoss/nydus/blob/main/rafs/src/fs.rs) for chunk-to-offset resolution.
- The **blob-cache** directory stores retrieved chunks locally, ensuring subsequent reads serve from disk with zero network latency.
- The system supports multiple backends including OCI registries and object storage, configured through the `blob_ondemand_cfg` JSON parameter.

## Frequently Asked Questions

### How does Nydus map a file read request to a specific chunk in the blob?

Nydus uses the RAFS (Random Access File System) bootstrap metadata to maintain a chunk table that maps file offsets to specific chunk IDs and blob locations. When `nydusd` receives a read request, the RAFS device implementation in [`rafs/src/fs.rs`](https://github.com/dragonflyoss/nydus/blob/main/rafs/src/fs.rs) executes the `read_at` method, which consults this chunk table to determine the exact byte range within the remote blob file that must be fetched from the backend.

### What is the difference between FUSE and fscache on-demand modes?

The **FUSE** mode runs `nydusd` as a user-space filesystem daemon that intercepts all read requests through the kernel's FUSE interface, suitable for containerd snapshotter integration. The **fscache** (bind ondemand) mode utilizes the kernel's fscache driver with EROFS mounts, where the kernel itself issues open and read requests to `nydusd` through a dedicated socket. As implemented in [`service/src/fs_cache.rs`](https://github.com/dragonflyoss/nydus/blob/main/service/src/fs_cache.rs) at line 311, the daemon writes `bind ondemand` to register with the kernel, reducing context switches for cached reads and improving performance for frequently accessed data.

### Where does Nydus store chunks retrieved through on-demand loading?

Retrieved chunks are stored in the **blob-cache** directory specified by the `blob_cache_dir` parameter in the blobfs configuration. The `FsCacheHandler` in [`service/src/fs_cache.rs`](https://github.com/dragonflyoss/nydus/blob/main/service/src/fs_cache.rs) manages this cache, writing fetched data to local storage and tracking cached chunks through an in-memory inode-map in blobfs. Subsequent reads of the same chunk hit this local cache, eliminating network latency and reducing backend load.

### Can Nydus on-demand loading work with any container registry?

Yes, Nydus on-demand loading supports multiple backend types including standard OCI registries, Alibaba Cloud OSS, and local filesystems. The backend type is configured in the `rafs_conf` section of the blobfs JSON configuration through the `backend_type` and `backend_config` fields. As long as the backend supports HTTP range requests or equivalent byte-range access methods, Nydus can fetch individual chunks on demand without requiring full layer downloads, enabling efficient lazy loading from any compatible storage backend.