How Nydus Manages Blob Files and Blob Caching: Architecture and Implementation

Nydus manages blob files by treating every container image layer as a blob backed by immutable metadata (BlobInfo) and runtime cache implementations (BlobCache), using a factory pattern (BlobCacheMgr) to handle lifecycle, encryption, compression, and prefetching across local disk or kernel FS-Cache backends.

Nydus is an open-source container image acceleration framework hosted at dragonflyoss/nydus. Understanding how Nydus manages blob files and blob caching is essential for optimizing container startup performance and storage efficiency. The architecture centers on three core components: immutable blob metadata, pluggable cache backends, and a global factory for resource lifecycle management.

Core Architecture: BlobInfo, BlobCache, and Factory

Nydus organizes blob management around three distinct layers that separate metadata from runtime I/O operations.

Component	Responsibility	Source Location
BlobInfo	Immutable metadata describing blob properties (size, compression, encryption, chunk layout)	storage/src/device.rs
BlobCache (trait)	Runtime interface for reading, decrypting, decompressing, and caching blob data	storage/src/cache/mod.rs
BlobCacheMgr	Factory and lifecycle manager that creates, owns, and garbage-collects BlobCache instances	storage/src/factory.rs

Blob Metadata Management with BlobInfo

When the Nydus daemon initializes, it parses the RAFS (Registry Acceleration File System) superblock to enumerate all blobs required by the image. For each layer, the system constructs a BlobInfo struct that serves as the single source of truth for that blob's characteristics.

In storage/src/device.rs, the BlobInfo::new constructor captures:

let blob = BlobInfo::new(
    blob_index,              // Position in RAFS blob table
    blob_id.clone(),         // SHA-256 hex identifier
    uncompressed_size,       // Logical size after decompression
    compressed_size,         // Raw size on backend storage
    chunk_size,              // Typically 4 MiB
    chunk_count,             // Number of addressable chunks
    blob_features,           // Flags: ZRAN, ENCRYPTED, BATCH, etc.
);

BlobInfo stores critical operational parameters:

• Compression algorithm (blob_compressor) – Determines decompression strategy (gzip, lz4, zstd).
• Encryption context (blob_cipher, cipher_ctx) – AES-GCM parameters for decrypting protected layers.
• Integrity verification (blob_digester) – Hash algorithm for chunk validation.
• Prefetch hints (prefetch_offset, prefetch_size) – Guides the prefetch worker to warm cache for sequential reads.

All I/O paths consult BlobInfo to determine how BlobCache::read_chunk_from_backend should process raw bytes.

The BlobCache Abstraction Layer

The BlobCache trait defined in storage/src/cache/mod.rs provides the runtime interface for blob access:

pub trait BlobCache: Send + Sync {
    fn blob_id(&self) -> &str;
    fn blob_uncompressed_size(&self) -> Result<u64>;
    fn blob_compressed_size(&self) -> Result<u64>;
    fn is_zran(&self) -> bool { false }
    fn is_batch(&self) -> bool { false }
    fn need_validation(&self) -> bool;
    fn reader(&self) -> &dyn BlobReader;
    fn read(&self, iovec: &mut BlobIoVec, buffers: &[FileVolatileSlice]) -> Result<usize>;
    fn prefetch(&self, ...);
}

Concrete implementations in storage/src/cache/ handle different deployment scenarios:

FileCache: Local Disk Caching

FileCacheMgr and FileCache (in storage/src/cache/filecache/) provide the most common production caching strategy. This implementation:

• Maintains a local file (or memory-mapped region) that mirrors the remote blob.
• Satisfies reads from the local copy after initial fetch.
• Supports background prefetch workers that populate the cache before data is requested.
• Implements garbage collection to reclaim disk space when blobs are no longer referenced.

FsCache: Kernel FS-Cache Integration

FsCacheMgr and FsCache (in storage/src/cache/fscache/) integrate with the Linux kernel's FS-Cache infrastructure. This approach:

• Delegates caching decisions to the operating system.
• Enables cache sharing across multiple processes.
• Leverages kernel LRU eviction policies rather than user-space logic.
• Requires Linux kernel support and appropriate configuration.

Factory Pattern and Lifecycle Management

The BlobCacheMgr in storage/src/factory.rs serves as the central factory and lifecycle manager for all blob caches. It operates through a global singleton BLOB_FACTORY that coordinates cache creation and destruction.

Key responsibilities include:

Creating caches via new_blob_cache:

BLOB_FACTORY
    .new_blob_cache(&config, &blob_info)
    .expect("failed to create blob cache")

This method either returns an existing cache or instantiates a new one based on the ConfigV2 specification (selecting between FileCache, FsCache, or DummyCache).

Reference counting through BlobCacheState::try_add. Each DataBlobConfig maintains an AtomicU32 reference count. Duplicate requests increment the counter; drops decrement it.

Periodic health checks via start_mgr_checker. A Tokio task runs every 5 seconds invoking check_cache_stat on each manager, allowing implementations to close idle file descriptors or flush pending writes.

Garbage collection through the gc method. When containers stop, service/src/fs_service.rs invokes BLOB_FACTORY.gc(Some((&config, blob_id))) to reclaim resources promptly.

Read and Prefetch Workflow

The typical data flow when a FUSE client accesses a file demonstrates how these components interact:

1. Chunk lookup: The RAFS driver identifies the BlobInfo and chunk index for the requested file range.

2. Cache acquisition: The system calls BLOB_FACTORY.new_blob_cache(&config, &blob_info) to obtain an Arc<dyn BlobCache>, stored within BlobDevice (which maintains an ArcSwap<Vec<Arc<dyn BlobCache>>> for hot-swapping).

3. I/O delegation: BlobDevice::read delegates to BlobCache::read, which:

   • Merges adjacent I/O descriptors via BlobIoMergeState to reduce round-trips.
   • Fetches compressed data from the backend through BlobReader::read.
   • Decrypts data if blob_info.is_encrypted().
   • Decompresses using the algorithm specified in blob_compressor.
   • Validates integrity using check_digest.

4. Prefetch optimization: For sequential read patterns, the RAFS driver generates BlobPrefetchRequest objects. BlobCache::prefetch spawns background workers (in storage/src/cache/worker.rs) that call read_chunks_from_backend with prefetch=true, populating the cache before the foreground request arrives.

Garbage Collection and Resource Management

Nydus implements a multi-tiered resource reclamation strategy to prevent unbounded cache growth:

Reference-counted lifecycle: Each blob cache maintains an AtomicU32 counter. When a container stops and drops its BlobCache reference, the count decrements. Upon reaching zero, the BlobCacheMgr removes the entry from its internal map.

Periodic health checks: The BlobFactory::start_mgr_checker task runs every 5 seconds, invoking check_cache_stat on each active manager. FileCacheMgr uses this opportunity to close idle file descriptors and flush pending writes to disk.

Explicit garbage collection: The service layer (service/src/fs_service.rs and service/src/blob_cache.rs) can trigger immediate cleanup via BLOB_FACTORY.gc(Some((&config, blob_id))), ensuring prompt resource reclamation during container shutdown.

Summary

• BlobInfo (storage/src/device.rs) provides immutable metadata for every container layer, capturing compression, encryption, and chunk layout information.
• BlobCache trait (storage/src/cache/mod.rs) abstracts runtime blob access, with concrete implementations including FileCache (local disk) and FsCache (kernel integration).
• BlobCacheMgr and the global BLOB_FACTORY (storage/src/factory.rs) handle cache creation, reference counting, and lifecycle management through periodic health checks and garbage collection.
• The read path merges I/O requests, decrypts, decompresses, and validates data chunks, while background prefetch workers populate caches for sequential access patterns.
• Garbage collection relies on atomic reference counting and explicit cleanup calls from the service layer to reclaim resources when containers stop.

Frequently Asked Questions

What is the difference between FileCache and FsCache in Nydus?

FileCache (storage/src/cache/filecache/) maintains a user-space local file or memory-mapped region that mirrors the remote blob, implementing its own prefetch and garbage collection logic. FsCache (storage/src/cache/fscache/) integrates with the Linux kernel's FS-Cache infrastructure, delegating caching decisions to the OS for better cross-process sharing and kernel-managed LRU eviction. FileCache works on any platform, while FsCache requires Linux kernel support.

How does Nydus handle blob encryption and compression?

Nydus stores encryption and compression parameters inside BlobInfo (storage/src/device.rs). The blob_compressor field indicates the algorithm (gzip, lz4, zstd), while blob_cipher and cipher_ctx hold AES-GCM encryption context. During the read path in BlobCache::read, the system checks these flags: if is_encrypted() returns true, it decrypts the chunk using the stored cipher context; if is_compressed() is true, it decompresses using the specified algorithm before validating the digest.

What triggers blob cache garbage collection?

Garbage collection triggers through three mechanisms: reference counting when BlobCache objects are dropped and their AtomicU32 count reaches zero; periodic health checks every 5 seconds via BlobFactory::start_mgr_checker calling check_cache_stat to close idle resources; and explicit GC calls from the service layer (service/src/fs_service.rs) invoking BLOB_FACTORY.gc() when containers stop, ensuring immediate reclamation of per-container caches.

How does prefetching improve container startup performance?

Prefetching accelerates startup by warming the cache before data is requested. When the RAFS driver detects sequential read patterns, it generates BlobPrefetchRequest objects. BlobCache::prefetch spawns background workers (implemented in storage/src/cache/worker.rs) that fetch chunks from the backend with prefetch=true, populating the local cache asynchronously. When the container actually requests these bytes, they are served from fast local storage rather than waiting for network round-trips, dramatically reducing I/O latency during initialization.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}