# How Nydus Manages Blob Files and Blob Caching: Architecture and Implementation

> Discover how Nydus manages blob files and blob caching with immutable metadata and runtime cache implementations for efficient lifecycle, encryption, compression, and prefetching.

- Repository: [dragonflyoss/nydus](https://github.com/dragonflyoss/nydus)
- Tags: architecture
- Published: 2026-02-28

---

**Nydus manages blob files by treating every container image layer as a blob backed by immutable metadata (`BlobInfo`) and runtime cache implementations (`BlobCache`), using a factory pattern (`BlobCacheMgr`) to handle lifecycle, encryption, compression, and prefetching across local disk or kernel FS-Cache backends.**

Nydus is an open-source container image acceleration framework hosted at `dragonflyoss/nydus`. Understanding how Nydus manages blob files and blob caching is essential for optimizing container startup performance and storage efficiency. The architecture centers on three core components: immutable blob metadata, pluggable cache backends, and a global factory for resource lifecycle management.

## Core Architecture: BlobInfo, BlobCache, and Factory

Nydus organizes blob management around three distinct layers that separate metadata from runtime I/O operations.

| Component | Responsibility | Source Location |
|-----------|---------------|-----------------|
| **`BlobInfo`** | Immutable metadata describing blob properties (size, compression, encryption, chunk layout) | [`storage/src/device.rs`](https://github.com/dragonflyoss/nydus/blob/main/storage/src/device.rs) |
| **`BlobCache` (trait)** | Runtime interface for reading, decrypting, decompressing, and caching blob data | [`storage/src/cache/mod.rs`](https://github.com/dragonflyoss/nydus/blob/main/storage/src/cache/mod.rs) |
| **`BlobCacheMgr`** | Factory and lifecycle manager that creates, owns, and garbage-collects `BlobCache` instances | [`storage/src/factory.rs`](https://github.com/dragonflyoss/nydus/blob/main/storage/src/factory.rs) |

## Blob Metadata Management with BlobInfo

When the Nydus daemon initializes, it parses the RAFS (Registry Acceleration File System) superblock to enumerate all blobs required by the image. For each layer, the system constructs a `BlobInfo` struct that serves as the single source of truth for that blob's characteristics.

In [`storage/src/device.rs`](https://github.com/dragonflyoss/nydus/blob/main/storage/src/device.rs), the `BlobInfo::new` constructor captures:

```rust
let blob = BlobInfo::new(
    blob_index,              // Position in RAFS blob table
    blob_id.clone(),         // SHA-256 hex identifier
    uncompressed_size,       // Logical size after decompression
    compressed_size,         // Raw size on backend storage
    chunk_size,              // Typically 4 MiB
    chunk_count,             // Number of addressable chunks
    blob_features,           // Flags: ZRAN, ENCRYPTED, BATCH, etc.
);

```

`BlobInfo` stores critical operational parameters:

- **Compression algorithm** (`blob_compressor`) – Determines decompression strategy (gzip, lz4, zstd).
- **Encryption context** (`blob_cipher`, `cipher_ctx`) – AES-GCM parameters for decrypting protected layers.
- **Integrity verification** (`blob_digester`) – Hash algorithm for chunk validation.
- **Prefetch hints** (`prefetch_offset`, `prefetch_size`) – Guides the prefetch worker to warm cache for sequential reads.

All I/O paths consult `BlobInfo` to determine how `BlobCache::read_chunk_from_backend` should process raw bytes.

## The BlobCache Abstraction Layer

The `BlobCache` trait defined in [`storage/src/cache/mod.rs`](https://github.com/dragonflyoss/nydus/blob/main/storage/src/cache/mod.rs) provides the runtime interface for blob access:

```rust
pub trait BlobCache: Send + Sync {
    fn blob_id(&self) -> &str;
    fn blob_uncompressed_size(&self) -> Result<u64>;
    fn blob_compressed_size(&self) -> Result<u64>;
    fn is_zran(&self) -> bool { false }
    fn is_batch(&self) -> bool { false }
    fn need_validation(&self) -> bool;
    fn reader(&self) -> &dyn BlobReader;
    fn read(&self, iovec: &mut BlobIoVec, buffers: &[FileVolatileSlice]) -> Result<usize>;
    fn prefetch(&self, ...);
}

```

Concrete implementations in `storage/src/cache/` handle different deployment scenarios:

### FileCache: Local Disk Caching

`FileCacheMgr` and `FileCache` (in `storage/src/cache/filecache/`) provide the most common production caching strategy. This implementation:

- Maintains a local file (or memory-mapped region) that mirrors the remote blob.
- Satisfies reads from the local copy after initial fetch.
- Supports **background prefetch** workers that populate the cache before data is requested.
- Implements **garbage collection** to reclaim disk space when blobs are no longer referenced.

### FsCache: Kernel FS-Cache Integration

`FsCacheMgr` and `FsCache` (in `storage/src/cache/fscache/`) integrate with the Linux kernel's FS-Cache infrastructure. This approach:

- Delegates caching decisions to the operating system.
- Enables cache sharing across multiple processes.
- Leverages kernel LRU eviction policies rather than user-space logic.
- Requires Linux kernel support and appropriate configuration.

## Factory Pattern and Lifecycle Management

The `BlobCacheMgr` in [`storage/src/factory.rs`](https://github.com/dragonflyoss/nydus/blob/main/storage/src/factory.rs) serves as the central factory and lifecycle manager for all blob caches. It operates through a global singleton `BLOB_FACTORY` that coordinates cache creation and destruction.

Key responsibilities include:

**Creating caches** via `new_blob_cache`:

```rust
BLOB_FACTORY
    .new_blob_cache(&config, &blob_info)
    .expect("failed to create blob cache")

```

This method either returns an existing cache or instantiates a new one based on the `ConfigV2` specification (selecting between `FileCache`, `FsCache`, or `DummyCache`).

**Reference counting** through `BlobCacheState::try_add`. Each `DataBlobConfig` maintains an `AtomicU32` reference count. Duplicate requests increment the counter; drops decrement it.

**Periodic health checks** via `start_mgr_checker`. A Tokio task runs every 5 seconds invoking `check_cache_stat` on each manager, allowing implementations to close idle file descriptors or flush pending writes.

**Garbage collection** through the `gc` method. When containers stop, [`service/src/fs_service.rs`](https://github.com/dragonflyoss/nydus/blob/main/service/src/fs_service.rs) invokes `BLOB_FACTORY.gc(Some((&config, blob_id)))` to reclaim resources promptly.

## Read and Prefetch Workflow

The typical data flow when a FUSE client accesses a file demonstrates how these components interact:

1. **Chunk lookup**: The RAFS driver identifies the `BlobInfo` and chunk index for the requested file range.

2. **Cache acquisition**: The system calls `BLOB_FACTORY.new_blob_cache(&config, &blob_info)` to obtain an `Arc<dyn BlobCache>`, stored within `BlobDevice` (which maintains an `ArcSwap<Vec<Arc<dyn BlobCache>>>` for hot-swapping).

3. **I/O delegation**: `BlobDevice::read` delegates to `BlobCache::read`, which:
   - Merges adjacent I/O descriptors via `BlobIoMergeState` to reduce round-trips.
   - Fetches compressed data from the backend through `BlobReader::read`.
   - Decrypts data if `blob_info.is_encrypted()`.
   - Decompresses using the algorithm specified in `blob_compressor`.
   - Validates integrity using `check_digest`.

4. **Prefetch optimization**: For sequential read patterns, the RAFS driver generates `BlobPrefetchRequest` objects. `BlobCache::prefetch` spawns background workers (in [`storage/src/cache/worker.rs`](https://github.com/dragonflyoss/nydus/blob/main/storage/src/cache/worker.rs)) that call `read_chunks_from_backend` with `prefetch=true`, populating the cache before the foreground request arrives.

## Garbage Collection and Resource Management

Nydus implements a multi-tiered resource reclamation strategy to prevent unbounded cache growth:

**Reference-counted lifecycle**: Each blob cache maintains an `AtomicU32` counter. When a container stops and drops its `BlobCache` reference, the count decrements. Upon reaching zero, the `BlobCacheMgr` removes the entry from its internal map.

**Periodic health checks**: The `BlobFactory::start_mgr_checker` task runs every 5 seconds, invoking `check_cache_stat` on each active manager. `FileCacheMgr` uses this opportunity to close idle file descriptors and flush pending writes to disk.

**Explicit garbage collection**: The service layer ([`service/src/fs_service.rs`](https://github.com/dragonflyoss/nydus/blob/main/service/src/fs_service.rs) and [`service/src/blob_cache.rs`](https://github.com/dragonflyoss/nydus/blob/main/service/src/blob_cache.rs)) can trigger immediate cleanup via `BLOB_FACTORY.gc(Some((&config, blob_id)))`, ensuring prompt resource reclamation during container shutdown.

## Summary

- **BlobInfo** ([`storage/src/device.rs`](https://github.com/dragonflyoss/nydus/blob/main/storage/src/device.rs)) provides immutable metadata for every container layer, capturing compression, encryption, and chunk layout information.
- **BlobCache** trait ([`storage/src/cache/mod.rs`](https://github.com/dragonflyoss/nydus/blob/main/storage/src/cache/mod.rs)) abstracts runtime blob access, with concrete implementations including **FileCache** (local disk) and **FsCache** (kernel integration).
- **BlobCacheMgr** and the global **BLOB_FACTORY** ([`storage/src/factory.rs`](https://github.com/dragonflyoss/nydus/blob/main/storage/src/factory.rs)) handle cache creation, reference counting, and lifecycle management through periodic health checks and garbage collection.
- The read path merges I/O requests, decrypts, decompresses, and validates data chunks, while background prefetch workers populate caches for sequential access patterns.
- Garbage collection relies on atomic reference counting and explicit cleanup calls from the service layer to reclaim resources when containers stop.

## Frequently Asked Questions

### What is the difference between FileCache and FsCache in Nydus?

**FileCache** (`storage/src/cache/filecache/`) maintains a user-space local file or memory-mapped region that mirrors the remote blob, implementing its own prefetch and garbage collection logic. **FsCache** (`storage/src/cache/fscache/`) integrates with the Linux kernel's FS-Cache infrastructure, delegating caching decisions to the OS for better cross-process sharing and kernel-managed LRU eviction. FileCache works on any platform, while FsCache requires Linux kernel support.

### How does Nydus handle blob encryption and compression?

Nydus stores encryption and compression parameters inside `BlobInfo` ([`storage/src/device.rs`](https://github.com/dragonflyoss/nydus/blob/main/storage/src/device.rs)). The `blob_compressor` field indicates the algorithm (gzip, lz4, zstd), while `blob_cipher` and `cipher_ctx` hold AES-GCM encryption context. During the read path in `BlobCache::read`, the system checks these flags: if `is_encrypted()` returns true, it decrypts the chunk using the stored cipher context; if `is_compressed()` is true, it decompresses using the specified algorithm before validating the digest.

### What triggers blob cache garbage collection?

Garbage collection triggers through three mechanisms: **reference counting** when `BlobCache` objects are dropped and their `AtomicU32` count reaches zero; **periodic health checks** every 5 seconds via `BlobFactory::start_mgr_checker` calling `check_cache_stat` to close idle resources; and **explicit GC calls** from the service layer ([`service/src/fs_service.rs`](https://github.com/dragonflyoss/nydus/blob/main/service/src/fs_service.rs)) invoking `BLOB_FACTORY.gc()` when containers stop, ensuring immediate reclamation of per-container caches.

### How does prefetching improve container startup performance?

Prefetching accelerates startup by warming the cache before data is requested. When the RAFS driver detects sequential read patterns, it generates `BlobPrefetchRequest` objects. `BlobCache::prefetch` spawns background workers (implemented in [`storage/src/cache/worker.rs`](https://github.com/dragonflyoss/nydus/blob/main/storage/src/cache/worker.rs)) that fetch chunks from the backend with `prefetch=true`, populating the local cache asynchronously. When the container actually requests these bytes, they are served from fast local storage rather than waiting for network round-trips, dramatically reducing I/O latency during initialization.