What Compression Strategies Are Used for Nydus Chunks?

Nydus supports four compression algorithms for chunks—None, LZ4-block, Gzip, and Zstd—with LZ4-block as the default for blob data and Zstd for reference-image metadata.

Nydus, the container image acceleration project hosted at dragonflyoss/nydus, stores file data as discrete chunks inside blob objects. Understanding what compression strategies are used for Nydus chunks is essential for optimizing storage costs and runtime performance when building or converting images.

Supported Compression Algorithms

The compression layer is defined in utils/src/compress/mod.rs through the Algorithm enum. Each variant maps to a specific identifier used in configuration and metadata.

None (Uncompressed)

The none algorithm stores chunks exactly as they are written, bypassing compression entirely.

• Identifier: none
• Implementation: Algorithm::None in utils/src/compress/mod.rs
• Typical use: Raw chunks or when compression overhead would not reduce storage size.

LZ4 Block

The lz4_block algorithm is the default compressor for Nydus blobs, offering fast compression and decompression speeds with reasonable space savings.

• Identifier: lz4_block
• Implementation: Algorithm::Lz4Block using lz4_compress and lz4_decompress functions.
• Typical use: General-purpose blob data where read performance is critical.

Gzip

The gzip algorithm provides compatibility with standard DEFLATE tooling and is supported for workloads requiring that specific format.

• Identifier: gzip
• Implementation: Algorithm::GZip wrapping flate2::write::GzEncoder and flate2::bufread::GzDecoder.
• Typical use: Environments requiring DEFLATE-compatible archives.

Zstandard (Zstd)

The zstd algorithm delivers higher compression ratios at moderate speed, making it ideal for compact metadata storage.

• Identifier: zstd
• Implementation: Algorithm::Zstd via zstd::bulk::compress and zstd::bulk::decompress_to_buffer.
• Typical use: Bootstrap metadata and chunk-info arrays when converting to reference images.

Build-Time Configuration

You select the compression strategy during image creation via the --compressor flag in the nydusify CLI, defined in contrib/nydusify/cmd/nydusify.go.

{
    Name:    "compressor",
    Usage:   "Algorithm to compress image data blob, possible values: none, lz4_block, zstd",
    // ...
}

This value populates the blob_compressor field inside BuildContext (builder/src/core/context.rs). During blob generation in builder/src/core/blob.rs, each chunk passes through the generic compress::compress helper:

let (compressed, is_compressed) = compress::compress(chunk_data, blob_ctx.blob_compressor)?;

Metadata vs. Data Compression

Nydus applies different compression strategies depending on whether it is processing file data or image metadata. The function get_compression_algorithm_for_meta in the builder logic determines this:

fn get_compression_algorithm_for_meta(ctx: &BuildContext) -> compress::Algorithm {
    if ctx.conversion_type.is_to_ref() {
        compress::Algorithm::Zstd
    } else {
        ctx.compressor
    }
}

When building a reference image (ctx.conversion_type.is_to_ref() returns true), the bootstrap and chunk-info array are compressed with Zstd regardless of the user-specified blob compressor. For standard builds, metadata follows the same algorithm as the blob data.

Automatic Size Regression Protection

The compress::compress implementation in utils/src/compress/mod.rs includes a safeguard against negative compression outcomes. If the compressed output is larger than the original data, or if the compression ratio does not meet the COMPRESSION_MINIMUM_RATIO threshold, the function automatically returns the uncompressed chunk and sets is_compressed to false.

Practical Examples

Compress Data with the Rust Library

use nydus::utils::compress::{compress, Algorithm};

let raw = b"The quick brown fox jumps over the lazy dog";
let (compressed, used) = compress(raw, Algorithm::Lz4Block).unwrap();

println!("Compressed? {}", used);   // true if compression saved space

Build an Image with a Specific Compressor

# LZ4 block (default)

nydusify -c source-dir -t myimage:latest --compressor lz4_block

# Gzip for DEFLATE compatibility

nydusify -c source-dir -t myimage:latest --compressor gzip

# Uncompressed for CPU-sensitive workloads

nydusify -c source-dir -t myimage:latest --compressor none

Inspect Compression in Blob Metadata

The compressor used for a blob is persisted in the SQLite catalog by src/bin/nydus-image/deduplicate.rs:

SELECT blob_id, blob_compressor FROM blob WHERE blob_id = '<blob-id>';

This column stores string values: "none", "lz4_block", "gzip", or "zstd".

Summary

• Nydus chunks support four algorithms: None, LZ4-block, Gzip, and Zstd.
• LZ4-block is the default for blob data, balancing speed and compression ratio.
• Zstd is reserved for metadata (bootstrap and chunk-info) when converting to reference images.
• The --compressor CLI flag in nydusify controls the algorithm selection.
• The system automatically falls back to uncompressed storage if compression does not reduce chunk size.

Frequently Asked Questions

What is the default compression algorithm for Nydus chunks?

The default is LZ4-block (lz4_block). According to the source in utils/src/compress/mod.rs, this algorithm is selected when no --compressor flag is provided, offering fast decompression speeds suitable for runtime container operations.

Why does Nydus use Zstd for metadata instead of LZ4?

When converting to a reference image, Nydus forces Zstd for metadata via get_compression_algorithm_for_meta in the builder code. Zstd achieves higher compression ratios on small metadata files (bootstrap and chunk dictionaries) compared to LZ4, reducing storage overhead for the image manifest structure.

Can I disable compression entirely when building a Nydus image?

Yes. Pass --compressor none to the nydusify CLI (defined in contrib/nydusify/cmd/nydusify.go). This sets Algorithm::None in the build context, causing all chunks to be written uncompressed to the blob.

How does Nydus handle chunks that don't compress well?

The compress function in utils/src/compress/mod.rs checks against COMPRESSION_MINIMUM_RATIO. If the compressed size exceeds the original, or the ratio threshold is not met, the function returns the raw data with is_compressed set to false, ensuring storage space is never wasted on negative compression.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}