How Nydus Implements the RAFS Filesystem Format: A Deep Dive into the EROFS-Based Layout

Nydus implements the RAFS filesystem format as a layered, EROFS-compatible on-disk layout comprising a dual superblock structure, device and blob tables, compact chunk addressing, and dual inode formats, enabling kernel-native mounting while supporting advanced container image features like prefetching and encryption.

The RAFS (Remote Access File System) format is the core storage engine behind the Nydus container image acceleration project. Designed to be compatible with the Linux EROFS driver, RAFS v6 enables images to mount directly in the kernel while maintaining a compact bootstrap that references external data blobs. This article examines the exact implementation details found in the dragonflyoss/nydus repository.

Dual Superblock Architecture

The RAFS v6 bootstrap begins with a dual superblock design that balances kernel compatibility with extended feature support.

Primary Superblock (RafsV6SuperBlock)

The first 4 KiB of every RAFS v6 bootstrap contains the RafsV6SuperBlock, defined in rafs/src/metadata/layout/v6.rs (lines 92-140). This structure mirrors the EROFS superblock layout exactly, allowing the Linux kernel's native EROFS driver to recognize and mount the image without modifications.

Key fields include the magic number, block size, and the root inode number (nid) that serves as the entry point for the filesystem tree.

Extended Superblock (RafsV6SuperBlockExt)

Immediately following the primary superblock, the RafsV6SuperBlockExt (lines 182-230) stores RAFS-specific metadata that the kernel ignores but the Nydus runtime requires. This includes:

• Chunk size and compression algorithm flags
• Blob table offset and size for locating data blob descriptors
• Prefetch table location for startup optimization hints
• Feature flags indicating compression, digest verification, and encryption capabilities

The extended superblock is validated during bootstrap loading to ensure feature compatibility between the image and the running Nydus daemon.

Device and Blob Metadata Tables

RAFS separates filesystem metadata from content data. The bootstrap contains tables describing where actual file data resides in external blobs.

Device Table (RafsV6Device)

Each additional data blob is described by a 128-byte RafsV6Device entry, also defined in rafs/src/metadata/layout/v6.rs (lines 73-84). These entries store:

• The blob's SHA-256 content identifier
• Block count and mapping address within the storage backend

The function rafsv6_load_blob_extra_info (lines 1554-1572) parses this table during bootstrap loading, building a HashMap<String, RafsBlobExtraInfo> keyed by blob ID for runtime lookup.

Blob Table (RafsV6Blob)

The blob table contains 256-byte RafsV6Blob structures (lines 93-124) describing each data blob's properties:

• Compression and digest algorithms (e.g., zstd, SHA-256)
• Chunk size and count within the blob
• Cipher information for encrypted blobs
• Offsets to the compression-information (CI) array and blob ToC (Table of Contents)

The method RafsV6Blob::to_blob_info (lines 1484-1529) converts these on-disk structures into runtime BlobInfo objects used by the storage layer.

Chunk Addressing and Data Layout

RAFS uses a compact addressing scheme to locate file data within blobs without loading full chunk maps into memory.

Chunk Table Entries

Each file data chunk is referenced by an RafsV6InodeChunkAddr (8 bytes), defined in rafs/src/metadata/layout/v6.rs (lines 141-170). This structure encodes:

• Blob index (7 bits) - identifies which blob contains the data
• Chunk-info index (24 bits) - position in the blob's compression info array
• Block address - offset within the decompressed blob

This deterministic addressing allows the runtime to compute data locations using minimal memory overhead, critical for container startup performance.

Inode Layout Variants

RAFS v6 supports two on-disk inode formats to optimize space for different file types.

Compact Inodes (RafsV6InodeCompact)

Files with simple metadata use the 32-byte RafsV6InodeCompact structure. This format fits within the standard EROFS compact inode layout and is used when data layout requirements are minimal.

Extended Inodes (RafsV6InodeExtended)

Larger files or those requiring extended attributes use the 64-byte RafsV6InodeExtended format. Both inode types implement the RafsV6OndiskInode trait (lines 334-424), providing uniform methods for setting size, UID/GID, mode bits, timestamps, and data layout information.

The factory function new_v6_inode in rafs/src/metadata/layout/mod.rs instantiates the appropriate inode type based on file characteristics.

Bootstrap Loading and Runtime Initialization

The high-level entry point for loading a RAFS v6 image is RafsSuper::try_load_v6 in rafs/src/metadata/md_v6.rs (lines 19-46). This function:

1. Reads and validates the primary superblock, extracting the root nid and blob table location
2. Loads and validates the extended superblock, checking compression/digest flags and table offsets
3. Creates a DirectSuperBlockV6 instance when operating in RafsMode::Direct, storing it in self.superblock for subsequent filesystem operations

The loader enforces that v6 images only operate in Direct mode, bypassing the FUSE userspace path for better performance.

RAFS Image Construction (Builder Side)

The builder crate assembles RAFS images through builder/src/core/v6.rs. The builder walks the source directory tree, creates inodes using new_v6_inode, and serializes:

• Inode tables (compact or extended based on file size)
• Chunk tables with compressed addressing
• Blob tables with encryption and compression metadata
• Prefetch tables containing arrays of inode numbers (u32) prioritized for early loading

The extended superblock is populated with feature flags via methods like set_compressor, set_digester, and set_cipher (starting at line 300 in layout/v6.rs).

Practical Code Examples

Loading a RAFS v6 Bootstrap

use nydus_storage::device::BlobDevice;
use nydus_rafs::metadata::RafsSuper;
use nydus_storage::RafsIoReader;

// Open the bootstrap file
let f = std::fs::File::open("/path/to/image.boot")?;
let mut reader: RafsIoReader = Box::new(f);

// Initialise a super-block in Direct mode
let mut superblock = RafsSuper {
    mode: nydus_rafs::metadata::RafsMode::Direct,
    ..Default::default()
};

// Try to load a v6 bootstrap
if superblock.try_load_v6(&mut reader)? {
    println!("RAFS v6 loaded, root nid = {}", superblock.meta.root_nid);
}

Creating an Inode During Image Build

use nydus_rafs::metadata::layout::new_v6_inode;
use nydus_rafs::metadata::inode::InodeWrapper;
use nydus_rafs::metadata::layout::RafsV6InodeCompact;

// `inode` is the in-memory representation collected by the builder
let on_disk_inode = new_v6_inode(
    &inode,                // InodeWrapper from the source tree
    /*datalayout=*/0,      // plain layout
    /*xattr_inline_count=*/0,
    /*compact=*/true,      // use 32-byte compact format
);
let bytes = on_disk_inode.store(&mut std::io::Cursor::new(Vec::new()))?;

Reading Blob Device Information

use nydus_rafs::metadata::layout::rafsv6_load_blob_extra_info;
use nydus_storage::RafsIoReader;

// Assuming `meta` is filled by the super-block loader
let mut reader = /* a RafsIoReader positioned at start */;
let blob_infos = rafsv6_load_blob_extra_info(&meta, &mut reader)?;
for (id, info) in blob_infos {
    println!("Blob {} mapped at blk {}", id, info.mapped_blkaddr);
}

Summary

• RAFS v6 uses a dual superblock structure (RafsV6SuperBlock and RafsV6SuperBlockExt) to maintain EROFS kernel compatibility while supporting extended features like compression and encryption.
• Device and blob tables in rafs/src/metadata/layout/v6.rs map abstract file data to specific storage blobs using SHA-256 identifiers and block addresses.
• Chunk addressing employs 8-byte RafsV6InodeChunkAddr structures to locate data via a 7-bit blob index and 24-bit chunk index, minimizing memory overhead.
• Dual inode formats (32-byte compact and 64-byte extended) optimize on-disk space while supporting files of varying complexity through the RafsV6OndiskInode trait.
• Bootstrap loading occurs via RafsSuper::try_load_v6 in rafs/src/metadata/md_v6.rs, which validates the layout and initializes direct-mount capabilities.
• Builder tooling in builder/src/core/v6.rs constructs valid RAFS images by serializing inode trees, chunk tables, and feature flags into the standardized layout.

Frequently Asked Questions

What is the relationship between RAFS and EROFS?

RAFS v6 is designed to be fully compatible with the Linux EROFS (Enhanced Read-Only File System) driver. The primary superblock (RafsV6SuperBlock) matches EROFS's on-disk structure exactly, allowing the kernel to mount RAFS images natively. The extended superblock adds container-specific metadata (blob locations, compression settings) that Nydus handles in userspace, creating a hybrid approach that leverages kernel performance for metadata operations while maintaining flexibility for image distribution.

How does RAFS handle large files split across multiple chunks?

Large files are divided into chunks described by RafsV6InodeChunkAddr entries. Each 8-byte entry contains a blob index (identifying which data blob holds the chunk), a chunk-info index (locating the chunk's compression metadata), and a block address (the offset within the blob). The runtime uses this information to fetch only the required chunks on demand, enabling lazy-loading of container images without downloading entire layers.

What determines whether an inode uses compact or extended format?

The builder automatically selects the compact format (RafsV6InodeCompact, 32 bytes) for files that fit EROFS's basic inode layout constraints, typically smaller files without extended attributes. Files requiring larger address spaces, additional timestamps, or extended attributes use the extended format (RafsV6InodeExtended, 64 bytes). The factory function new_v6_inode in rafs/src/metadata/layout/mod.rs handles this selection transparently during image construction.

How does the prefetch table improve container startup performance?

The prefetch table is an array of inode numbers embedded in the extended superblock that hints to the Nydus runtime which files should be loaded immediately upon mount. By prioritizing these inodes during the initial fetch phase, critical application files (such as dynamic libraries or configuration files) become available before the container process starts executing, reducing latency compared to strict on-demand loading. The table is validated during bootstrap loading in RafsV6SuperBlockExt::validate (lines 498-525).

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}