Persistent Memory (Pmem) vs. Regular Disk Configurations in CubeSandbox

CubeSandbox supports both standard ext4 disk storage and high-performance persistent memory (pmem) backends, with pmem providing Direct Access (DAX) capabilities for microsecond-level I/O latency through Intel Optane DC or NVDIMM hardware.

The TencentCloud CubeSandbox container runtime implements a flexible dual-storage architecture that allows operators to choose between traditional block-based storage and byte-addressable persistent memory. While regular disk configurations rely on standard ext4 filesystems backed by HDD or SSD devices, the persistent memory (pmem) backend eliminates the kernel page cache entirely by mapping files directly into memory address space. This technical distinction drives significant performance implications for container startup times and runtime I/O throughput.

Underlying Storage Architecture and Media

Regular Disk (ext4) Backend

Regular disk configurations in CubeSandbox utilize ordinary block devices formatted with the ext4 filesystem. Images and runtime data reside at paths such as /var/lib/cubesandbox/images/<instance>/<image>.ext4, managed by the generic ext4 image pipeline located in Cubelet/internal/cube/server/images/ext4image. This backend operates through the standard Linux VFS layer, incurring typical page cache overhead and block I/O latency dependent on the underlying hardware.

Persistent Memory (Pmem) Backend

The pmem backend targets Intel Optane DC Persistent Memory or equivalent NVDIMM hardware exposed through a DAX-enabled filesystem. All pmem artifacts live under a dedicated base path—typically /pmem—configured via CubeToolBaseDir. The Cubelet/pkg/container/pmem/pmem.go package centralizes path construction logic, ensuring consistent file organization across the persistent memory hierarchy.

Performance Characteristics and Workload Selection

I/O Latency and Throughput

Regular disk storage processes I/O through the standard block layer and page cache, resulting in latency measured in milliseconds for random access patterns. In contrast, pmem leverages Direct Access (DAX) to bypass the page cache entirely, achieving microsecond read/write latencies and dramatically faster container initialization times. This makes pmem ideal for high-performance workloads requiring rapid state persistence.

Activation via Pod Annotations

CubeSandbox distinguishes storage backends through the constants.AnnotationPmem annotation (defined in Cubelet/pkg/constants/const.go). When a pod specification includes the cube.pmem annotation, the container creation flow in Cubelet/services/cubebox/cube_container_create.go (lines 1026–1074) collects pmem configuration via pmem.CubePmem structs and stores the requirements in spec.Annotations[constants.AnnotationPmem]. Without this annotation, the system defaults to regular ext4 storage.

Lifecycle Management and Data Integrity

Synchronous vs. Asynchronous Destruction

Regular disk image deletion follows an asynchronous path that may leave temporary files and relies on the reflink-based storage pool implementation in Cubelet/storage/pool_withreflink.go. Pmem operations require stricter consistency guarantees because pmem files map directly into process memory address spaces.

As implemented in Cubelet/services/images/service.go at line 302, ext4 deletions route to a synchronous, idempotent pmem destroy path. The pmem.DestroyImage() function ensures that kernel and image files are removed atomically, preventing memory mapping conflicts during container teardown.

Initialization and Path Resolution

Before using pmem, the node must initialize the base directory:

// Executed once per node during Cubelet startup
pmem.Init(config.CubeToolBaseDir)  // Located in Cubelet/services/images/local.go line 125

The system then retrieves concrete file paths using the pmem package helpers:

// Get the root directory for a specific instance type
basePath := pmem.GetPmemBasePath(instanceType)

// Construct the full path to a raw image file
imgPath := pmem.GetRawImageFilePath(
    cubebox.InstanceType_cubebox.String(),
    containerReq.GetImage().GetImage(),
)

Kernel Handling and File Organization

Shared Kernel Optimization

Regular disk configurations store kernel files as ordinary files on the filesystem. The pmem backend implements a shared kernel file strategy to conserve persistent memory capacity. The pmem.GetSharedKernelFilePath() function returns a common kernel location, while pmem.GetRawKernelFilePath(instanceType, imageRef) handles instance-specific variants.

Before mounting, the system ensures kernel consistency:

// Ensure shared kernel exists before container startup
if err := pmem.EnsureKernelFilePresent(
    ctx,
    pmem.GetSharedKernelFilePath(),
    pmem.GetRawKernelFilePath(instanceType, imageRef),
); err != nil {
    return err
}

The RefreshKernelFile and EnsureKernelFilePresent utilities in Cubelet/internal/cube/server/images/ext4image/utils.go maintain kernel image consistency across pmem instances.

Complete Destruction Workflow

When removing pmem artifacts, use the idempotent destroy function rather than standard filesystem deletion:

// Synchronous, idempotent destruction of pmem artifacts
if err := pmem.DestroyImage(ctx, imgPath); err != nil {
    return err
}

The underlying implementation in Cubelet/internal/cube/server/images/ext4image/destroy.go handles the specific requirements of deallocating persistent memory-mapped files.

Key Source Files and Implementation Details

File Path	Responsibility
Cubelet/pkg/container/pmem/pmem.go	Core path generation and initialization API
Cubelet/services/images/service.go	Routes deletions to pmem destroy path (line 302)
Cubelet/services/cubebox/cube_container_create.go	Annotation parsing and pmem configuration injection (lines 1026–1074)
Cubelet/internal/cube/server/images/ext4image/utils.go	Kernel file presence verification
Cubelet/internal/cube/server/images/ext4image/destroy.go	Pmem-aware destruction logic
Cubelet/pkg/constants/const.go	Annotation constant definitions
Cubelet/services/images/local.go	Pmem initialization calls (line 125)

Summary

• Regular disk uses ext4 on block devices with asynchronous lifecycle management, suitable for general workloads and maximum compatibility.
• Persistent memory provides DAX-enabled direct access to Intel Optane or NVDIMM hardware, delivering microsecond I/O latency through the pmem package in Cubelet/pkg/container/pmem/pmem.go.
• Workload selection occurs via the cube.pmem annotation parsed during container creation in Cubelet/services/cubebox/cube_container_create.go.
• Data integrity requires synchronous, idempotent destruction for pmem files (implemented in Cubelet/services/images/service.go line 302) versus the asynchronous deletion used for regular ext4 images.
• Kernel optimization in pmem mode utilizes shared kernel files managed by EnsureKernelFilePresent to reduce memory footprint across containers.

Frequently Asked Questions

How does CubeSandbox determine whether to use pmem or regular disk for a container?

The controller checks for the cube.pmem annotation (defined as constants.AnnotationPmem in Cubelet/pkg/constants/const.go) during pod creation. If present in the pod specification, the container creation flow collects pmem configuration via pmem.CubePmem structs between lines 1026–1074 of Cubelet/services/cubebox/cube_container_create.go. Absent this annotation, the system defaults to the standard ext4 storage backend.

Why does pmem require synchronous destruction while regular disk uses asynchronous deletion?

Pmem files are directly memory-mapped into the process address space using DAX, meaning the hardware state reflects the file content immediately without page cache buffering. An asynchronous deletion could leave mapped memory in an inconsistent state if the file disappears while still referenced. The synchronous idempotent path in Cubelet/services/images/service.go (line 302) ensures atomic cleanup, whereas regular ext4 images tolerate eventual consistency and use the reflink-based pool in Cubelet/storage/pool_withreflink.go.

What hardware is required to use the pmem backend in CubeSandbox?

The pmem backend requires Intel Optane DC Persistent Memory or compatible NVDIMM hardware exposed through a DAX-enabled filesystem. The base directory (typically /pmem) must be initialized once per node via pmem.Init(config.CubeToolBaseDir), as implemented in Cubelet/services/images/local.go. Without this hardware and initialization, the pmem path generation functions in Cubelet/pkg/container/pmem/pmem.go cannot construct valid storage locations.

How does CubeSandbox optimize kernel storage when using persistent memory?

Rather than storing duplicate kernel images for each container, the pmem backend utilizes a shared kernel file pattern. The pmem.GetSharedKernelFilePath() function returns a common location, and pmem.EnsureKernelFilePresent() (defined in Cubelet/internal/cube/server/images/ext4image/utils.go) verifies or refreshes this file before container startup. This approach minimizes persistent memory consumption while ensuring all containers DAX-map the same kernel image for rapid boot.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}