How the nydusd FUSE Driver Works: Architecture and Implementation

The nydusd FUSE driver implements a high-performance, read-only filesystem in Rust by spawning multiple worker threads that pull requests from /dev/fuse via a FuseSession, dispatching them to a VFS backend through the FuseServer request loop.

The nydusd FUSE driver serves as the primary filesystem interface for the Dragonfly Nydus project, transforming standard Linux mount points into content-addressable container image stores. Written in Rust and built atop the fuse_backend_rs library, this driver enables on-demand image data fetching through a multi-threaded architecture optimized for container startup performance.

Core Architecture Components

The nydusd FUSE driver consists of four tightly-coupled Rust components that together manage the kernel interface, request dispatch, and lifecycle operations.

FusedevFsService (Session and Backend Management)

The FusedevFsService struct, defined in service/src/fusedev.rs, serves as the central coordinator for the FUSE filesystem. It holds the VFS instance, the FuseSession, and the backend collection. This component creates the fuse session, mounts it on the target directory, and provides critical helpers for cache-invalidation and fail-over notifications through methods like drain_fuse_requests.

FuseServer (Request Dispatch Layer)

FuseServer acts as a thin wrapper around fuse_backend_rs::api::Server. According to the dragonflyoss/nydus source code, its svc_loop method (located at service/src/fusedev.rs lines 21-44) continuously pulls raw requests from /dev/fuse and forwards them to the VFS for processing. The server owns an Arc<Server<Vfs>> and a FuseChannel bound to the session.

FusedevDaemon (Lifecycle and Threading)

The FusedevDaemon object represents the daemon instance that the Nydus runtime interacts with. Its start method spawns a configurable number of worker threads (controlled by the threads_cnt parameter), with each thread running its own FuseServer instance. The daemon manages the complete lifecycle including mount, graceful shutdown via stop, and cleanup through wait.

create_fuse_daemon Factory Function

The create_fuse_daemon convenience factory, implemented in service/src/fusedev.rs (lines 106-122), constructs a fully initialized FusedevDaemon. This function mounts the session, stores the connection ID for fail-over tracking, and optionally prepares live-upgrade handles when API sockets are configured.

Request Processing Pipeline

The nydusd FUSE driver processes filesystem operations through a six-stage pipeline that bridges kernel requests to the RAFS backend.

1. Session Creation: When FusedevFsService::new instantiates the service, it creates a FuseSession on the target mount point using "rafs" as the registered filesystem name. The session abstracts the kernel-side /dev/fuse device and stores the connection ID used for fail-over operations.

2. Server Initialization: FusedevFsService::create_fuse_server builds a FuseServer that owns an Arc<Server<Vfs>> and binds it to a FuseChannel connected to the session.

3. Worker Thread Spawning: FusedevDaemon::start spawns N threads based on the threads_cnt configuration. Each thread executes kick_one_server, which creates a fresh FuseServer, registers a FuseOpWrapper metrics hook to record inflight operations, and enters the service loop.

4. Request Loop Execution: Inside FuseServer::svc_loop, each worker thread repeatedly calls self.ch.get_request() to retrieve the next request from the kernel. The thread hands the request to self.server.handle_message, passing the optional metrics hook. The loop handles EncodeMessage errors (indicating kernel session closure) and continues until receiving EBADF or an explicit umount call.

5. Cache Invalidation and Fail-over: When the daemon flushes stale data after a live-upgrade, FusedevFsService::drain_fuse_requests executes. Depending on the configured FailoverPolicy, it either writes to /proc/sys/fs/fuse/connections/*/resend via FuseSysfsNotifier or sends resend notifications directly through the /dev/fuse file descriptor using FusedevNotifier.

6. Lifecycle Termination: FusedevFsService::umount gracefully shuts down the FuseSession and wakes the channel. FusedevDaemon::stop forces the session's wake() method, causing all worker threads to exit their loops, while FusedevDaemon::wait joins all threads to ensure clean shutdown.

Starting a nydusd FUSE Daemon

The following example demonstrates how to instantiate the nydusd FUSE driver using the factory function:

use std::sync::Arc;
use fuse_backend_rs::api::Vfs;
use nydus_api::BuildTimeInfo;
use nydus_rafs::metadata::RafsSuperBlock;
use nydus_runtime::service::create_fuse_daemon;
use nydus_runtime::upgrade::FailoverPolicy;
use mio::Waker;

fn main() -> anyhow::Result<()> {
    // 1. Build a VFS (RAFS backend in this example)
    let vfs = create_vfs_backend(
        FsBackendType::Rafs,
        /* is_fuse = */ true,
        /* hybrid_mode = */ false,
    )?;

    // 2. Prepare runtime helpers
    let waker = Arc::new(Waker::new(poll::Poll::new()?.registry(), 0)?);
    let bti = BuildTimeInfo::default();

    // 3. Create and start the fuse daemon
    let daemon = create_fuse_daemon(
        "/tmp/nydus_mount",   // mount point
        vfs,
        None,                 // no supervisor socket
        Some("demo".into()), // daemon ID
        4,                    // number of worker threads
        waker,
        None,                 // no API socket (no live‑upgrade)
        false,                // upgrade disabled
        false,                // read‑write mode
        FailoverPolicy::None,
        None,                 // no explicit backend mount command
        bti,
    )?;

    // 4. The daemon is now serving the RAFS image via FUSE.
    //    It runs in background threads; block the main thread if needed.
    daemon.wait()?;
    Ok(())
}

This snippet mirrors the real factory call used throughout the dragonflyoss/nydus project, configuring four worker threads and read-only mode.

Key Source Files

File	Description
service/src/fusedev.rs	Core FUSE driver implementation containing FusedevFsService, FuseServer, FusedevDaemon, and create_fuse_daemon.
service/src/fs_service.rs	Definition of the FsService trait that FusedevFsService implements for mount, upgrade, and cache-invalidation operations.
service/src/daemon.rs	Generic daemon infrastructure including the state machine and channel plumbing that FusedevDaemon extends.
service/src/vfs.rs	VFS wrapper around fuse_backend_rs::api::Vfs that handles the actual request processing logic.
rafs/	RAFS image parsing and inode structures (e.g., metadata::RafsInode) used by the VFS to resolve filesystem operations.

Summary

• The nydusd FUSE driver architecture centers on FusedevFsService for session management, FuseServer for request handling, and FusedevDaemon for thread lifecycle control.
• Worker threads execute FuseServer::svc_loop to pull requests from /dev/fuse and dispatch them to the VFS backend.
• The driver registers itself with the kernel using the "rafs" filesystem type and maintains connection IDs for fail-over scenarios.
• Cache invalidation supports both sysfs-based (FuseSysfsNotifier) and direct descriptor (FusedevNotifier) notification methods depending on the FailoverPolicy.
• Graceful shutdown is coordinated through FusedevDaemon::stop and wait, ensuring all worker threads exit cleanly before the process terminates.

Frequently Asked Questions

What is the role of FusedevFsService in the nydusd FUSE driver?

FusedevFsService serves as the primary orchestrator located in service/src/fusedev.rs. It holds the VFS instance, manages the FuseSession connected to /dev/fuse, and provides the drain_fuse_requests method for cache invalidation during live upgrades.

How does nydusd handle concurrent FUSE requests?

The driver handles concurrency through FusedevDaemon, which spawns a configurable number of worker threads (set via threads_cnt) during startup. Each thread runs an independent FuseServer::svc_loop that pulls requests from the kernel, allowing parallel processing of filesystem operations against the RAFS backend.

What filesystem name does nydusd register with the kernel?

According to the source code in service/src/fusedev.rs, nydusd registers the filesystem name "rafs" when creating the FuseSession. This identifier appears in mount listings and kernel logs when the driver is active.

How does the driver handle fail-over during live upgrades?

During live upgrades, FusedevFsService::drain_fuse_requests executes based on the configured FailoverPolicy. The driver either writes to /proc/sys/fs/fuse/connections/*/resend using FuseSysfsNotifier or sends direct notifications through the /dev/fuse file descriptor using FusedevNotifier to force the kernel to resend pending requests to the new daemon instance.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}