Nydus Rafs v5 vs v6: Key Differences Between FUSE and EROFS Formats

Rafs v5 runs as a user-space FUSE daemon with optional caching on any modern Linux kernel, whereas Rafs v6 utilizes the in-kernel EROFS driver for direct kernel mounts, requiring Linux ≥5.16, mandatory blob caching, and offering significantly lower latency through zero-copy I/O.

The dragonflyoss/nydus repository provides two distinct generations of its container image format. Understanding the differences between Nydus Rafs v5 and v6 is essential for selecting the appropriate deployment strategy, as each version targets different performance requirements and kernel capabilities.

Architecture and Mount Mechanism

The fundamental architectural distinction lies in where the filesystem logic executes.

Rafs v5 (FUSE) implements the filesystem as a user-space daemon communicating with the kernel via /dev/fuse. This approach supports both traditional FUSE and virtio-fs transports, making it compatible with any recent Linux kernel and macOS systems. The daemon handles all filesystem operations, including metadata lookups and data chunk retrieval.

Rafs v6 (EROFS) abandons the user-space daemon model in favor of the native in-kernel EROFS driver. Instead of forwarding requests to a userspace process, the kernel mounts the image directly, parsing the on-disk layout internally. This eliminates context switches between kernel and user space but requires kernel version 5.16 or later with EROFS support enabled.

Metadata Format and On-Disk Layout

The two versions use completely incompatible super-block structures defined in separate source files.

In rafs/src/metadata/layout/v5.rs, the Rafs v5 super-block occupies 8 KiB with magic number 0x52414653 and version 0x500. It maintains a custom on-disk layout featuring a blob table that describes separate data blobs, supporting fixed-size 4 KiB (or user-configured) chunks.

The Rafs v6 super-block, defined in rafs/src/metadata/layout/v6.rs, follows the standard EROFS layout using a compact 128-byte header with magic 0xE0F5E1E2. It leverages the EROFS_INODE_CHUNK_BASED format for chunk storage, with chunk size encoded directly in the inode header using constants like EROFS_CHUNK_FORMAT_SIZE_MASK. The validation logic in RafsV6SuperBlock::validate (lines 57-100) checks alignment, meta-address ranges, and checksums, while RafsV5SuperBlock::validate (lines 146-180) verifies magic, version, block size, and flag bits.

Supported Modes and Cache Requirements

Operating mode flexibility differs significantly between versions.

Rafs v5 supports both direct and cached modes as defined in the RafsMode enum in rafs/src/metadata/mod.rs. The cached mode optionally stores chunk information in a local cache, but the system can operate with a dummy cache if desired.

Rafs v6 enforces stricter constraints. According to the source code in rafs/src/metadata/md_v6.rs (lines 68-69), attempting to use cached mode returns ENOSYS with the error message "Rafs v6 does not support cached mode". Furthermore, the Rafs::new constructor in rafs/src/fs.rs (lines 19-25) validates that the cache type must be either blobcache or filecache—a mandatory requirement that causes immediate abort if unconfigured.

Data Integrity and Validation Features

Feature parity is not complete between the two generations.

Rafs v5 supports integrity validation through the rafs_cfg.validate flag, allowing the daemon to verify data integrity during runtime operations.

Rafs v6 explicitly rejects validation requests. The initialization code in rafs/src/fs.rs (lines 27-31) deliberately aborts when the --validate flag is passed for v6 images, indicating this feature remains unimplemented for the EROFS-based format.

Performance Characteristics and Platform Support

The architectural choices create distinct performance profiles.

Rafs v5 incurs higher latency due to context switches between kernel and user-space for every filesystem operation. However, this trade-off provides maximum portability across Linux distributions and macOS environments without requiring special kernel modules.

Rafs v6 achieves zero-copy kernel I/O with significantly lower latency and higher throughput by executing entirely within the kernel. This performance advantage comes with the constraint of Linux-only support and the prerequisite of an EROFS-capable kernel (≥5.16).

Practical Mounting Examples

Mounting a Rafs v5 Image (FUSE)

# Start the Nydus daemon as a FUSE server

nydusd \
  --fs-type=rafs/v5 \
  --bootstrap=/var/lib/nydus/bootstrap.json \
  --blob-cache=/var/lib/nydus/blobcache \
  --fuse-mountpoint=/mnt/nydus-fuse

Key points:

• --fs-type=rafs/v5 selects the FUSE implementation
• --blob-cache can be a blobcache, filecache, or dummy
• No special kernel version is required

Mounting a Rafs v6 Image (EROFS)

# Start the daemon in kernel-mode (EROFS)

nydusd \
  --fs-type=rafs/v6 \
  --bootstrap=/var/lib/nydus/bootstrap.json \
  --blob-cache=/var/lib/nydus/blobcache \
  --mountpoint=/mnt/nydus-erofs

Key points:

• --fs-type=rafs/v6 configures kernel-level EROFS mounting
• The daemon must use a blobcache or filecache configuration
• Integrity validation (--validate) is currently rejected for v6

Summary

• Rafs v5 operates via user-space FUSE, supports both direct and cached modes, works on any modern Linux or macOS, and offers optional integrity validation.
• Rafs v6 utilizes the in-kernel EROFS driver, requires Linux ≥5.16, mandates blob caching, supports only direct mode, and rejects validation flags.
• The v5 format uses an 8 KiB super-block with magic 0x52414653, while v6 uses a 128-byte EROFS-compatible super-block with magic 0xE0F5E1E2.
• v6 provides superior performance through zero-copy kernel I/O but imposes stricter deployment requirements compared to the portable v5 implementation.

Frequently Asked Questions

What kernel version is required for Rafs v6?

Rafs v6 requires Linux kernel version 5.16 or later with EROFS support enabled. Unlike Rafs v5, which functions on any recent kernel through the FUSE interface, v6 depends on the native in-kernel EROFS driver that was stabilized in kernel 5.16.

Why does Rafs v6 reject cached mode?

According to the source code in rafs/src/metadata/md_v6.rs, the implementation explicitly returns ENOSYS when cached mode is requested because the EROFS-based design assumes direct access patterns with mandatory blob caching. The architecture relies on the blobcache or filecache layer for data persistence rather than maintaining a separate cached metadata layer.

Is integrity validation available for Rafs v6?

No. The daemon initialization code in rafs/src/fs.rs deliberately checks for the validation flag and aborts with an error when mounting Rafs v6 images. This feature is currently only implemented for Rafs v5, making v5 the preferred choice when runtime integrity verification is mandatory.

Can I convert existing Rafs v5 images to v6 format?

The article focuses on runtime differences rather than conversion utilities. While the builder implementation in builder/src/core/v6.rs supports creating v6-specific bootstraps using new_v6_inode and RafsV6SuperBlock, converting existing v5 images requires regenerating the bootstrap with the v6 layout, as the on-disk formats (8 KiB vs 128-byte super-blocks, different chunk layouts) are fundamentally incompatible.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}