# Persistent Memory (Pmem) vs. Regular Disk Configurations in CubeSandbox

> Understand Persistent Memory (Pmem) vs. regular disk in CubeSandbox. Discover Pmem's microsecond I/O latency with DAX capabilities for superior performance.

- Repository: [Tencent Cloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox)
- Tags: performance
- Published: 2026-07-15

---

**CubeSandbox supports both standard ext4 disk storage and high-performance persistent memory (pmem) backends, with pmem providing Direct Access (DAX) capabilities for microsecond-level I/O latency through Intel Optane DC or NVDIMM hardware.**

The TencentCloud CubeSandbox container runtime implements a flexible dual-storage architecture that allows operators to choose between traditional block-based storage and byte-addressable persistent memory. While regular disk configurations rely on standard ext4 filesystems backed by HDD or SSD devices, the **persistent memory (pmem)** backend eliminates the kernel page cache entirely by mapping files directly into memory address space. This technical distinction drives significant performance implications for container startup times and runtime I/O throughput.

## Underlying Storage Architecture and Media

### Regular Disk (ext4) Backend

Regular disk configurations in CubeSandbox utilize ordinary block devices formatted with the ext4 filesystem. Images and runtime data reside at paths such as `/var/lib/cubesandbox/images/<instance>/<image>.ext4`, managed by the generic ext4 image pipeline located in `Cubelet/internal/cube/server/images/ext4image`. This backend operates through the standard Linux VFS layer, incurring typical page cache overhead and block I/O latency dependent on the underlying hardware.

### Persistent Memory (Pmem) Backend

The pmem backend targets Intel Optane DC Persistent Memory or equivalent NVDIMM hardware exposed through a DAX-enabled filesystem. All pmem artifacts live under a dedicated base path—typically `/pmem`—configured via `CubeToolBaseDir`. The [`Cubelet/pkg/container/pmem/pmem.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/pkg/container/pmem/pmem.go) package centralizes path construction logic, ensuring consistent file organization across the persistent memory hierarchy.

## Performance Characteristics and Workload Selection

### I/O Latency and Throughput

Regular disk storage processes I/O through the standard block layer and page cache, resulting in latency measured in milliseconds for random access patterns. In contrast, **pmem leverages Direct Access (DAX)** to bypass the page cache entirely, achieving microsecond read/write latencies and dramatically faster container initialization times. This makes pmem ideal for high-performance workloads requiring rapid state persistence.

### Activation via Pod Annotations

CubeSandbox distinguishes storage backends through the `constants.AnnotationPmem` annotation (defined in [`Cubelet/pkg/constants/const.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/pkg/constants/const.go)). When a pod specification includes the `cube.pmem` annotation, the container creation flow in [`Cubelet/services/cubebox/cube_container_create.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/services/cubebox/cube_container_create.go) (lines 1026–1074) collects pmem configuration via `pmem.CubePmem` structs and stores the requirements in `spec.Annotations[constants.AnnotationPmem]`. Without this annotation, the system defaults to regular ext4 storage.

## Lifecycle Management and Data Integrity

### Synchronous vs. Asynchronous Destruction

Regular disk image deletion follows an asynchronous path that may leave temporary files and relies on the reflink-based storage pool implementation in [`Cubelet/storage/pool_withreflink.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/storage/pool_withreflink.go). Pmem operations require stricter consistency guarantees because pmem files map directly into process memory address spaces.

As implemented in [`Cubelet/services/images/service.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/services/images/service.go) at line 302, ext4 deletions route to a **synchronous, idempotent pmem destroy path**. The `pmem.DestroyImage()` function ensures that kernel and image files are removed atomically, preventing memory mapping conflicts during container teardown.

### Initialization and Path Resolution

Before using pmem, the node must initialize the base directory:

```go
// Executed once per node during Cubelet startup
pmem.Init(config.CubeToolBaseDir)  // Located in Cubelet/services/images/local.go line 125

```

The system then retrieves concrete file paths using the pmem package helpers:

```go
// Get the root directory for a specific instance type
basePath := pmem.GetPmemBasePath(instanceType)

// Construct the full path to a raw image file
imgPath := pmem.GetRawImageFilePath(
    cubebox.InstanceType_cubebox.String(),
    containerReq.GetImage().GetImage(),
)

```

## Kernel Handling and File Organization

### Shared Kernel Optimization

Regular disk configurations store kernel files as ordinary files on the filesystem. The pmem backend implements a **shared kernel file** strategy to conserve persistent memory capacity. The `pmem.GetSharedKernelFilePath()` function returns a common kernel location, while `pmem.GetRawKernelFilePath(instanceType, imageRef)` handles instance-specific variants.

Before mounting, the system ensures kernel consistency:

```go
// Ensure shared kernel exists before container startup
if err := pmem.EnsureKernelFilePresent(
    ctx,
    pmem.GetSharedKernelFilePath(),
    pmem.GetRawKernelFilePath(instanceType, imageRef),
); err != nil {
    return err
}

```

The `RefreshKernelFile` and `EnsureKernelFilePresent` utilities in [`Cubelet/internal/cube/server/images/ext4image/utils.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/internal/cube/server/images/ext4image/utils.go) maintain kernel image consistency across pmem instances.

### Complete Destruction Workflow

When removing pmem artifacts, use the idempotent destroy function rather than standard filesystem deletion:

```go
// Synchronous, idempotent destruction of pmem artifacts
if err := pmem.DestroyImage(ctx, imgPath); err != nil {
    return err
}

```

The underlying implementation in [`Cubelet/internal/cube/server/images/ext4image/destroy.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/internal/cube/server/images/ext4image/destroy.go) handles the specific requirements of deallocating persistent memory-mapped files.

## Key Source Files and Implementation Details

| File Path | Responsibility |
|-----------|---------------|
| [`Cubelet/pkg/container/pmem/pmem.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/pkg/container/pmem/pmem.go) | Core path generation and initialization API |
| [`Cubelet/services/images/service.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/services/images/service.go) | Routes deletions to pmem destroy path (line 302) |
| [`Cubelet/services/cubebox/cube_container_create.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/services/cubebox/cube_container_create.go) | Annotation parsing and pmem configuration injection (lines 1026–1074) |
| [`Cubelet/internal/cube/server/images/ext4image/utils.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/internal/cube/server/images/ext4image/utils.go) | Kernel file presence verification |
| [`Cubelet/internal/cube/server/images/ext4image/destroy.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/internal/cube/server/images/ext4image/destroy.go) | Pmem-aware destruction logic |
| [`Cubelet/pkg/constants/const.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/pkg/constants/const.go) | Annotation constant definitions |
| [`Cubelet/services/images/local.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/services/images/local.go) | Pmem initialization calls (line 125) |

## Summary

- **Regular disk** uses ext4 on block devices with asynchronous lifecycle management, suitable for general workloads and maximum compatibility.
- **Persistent memory** provides DAX-enabled direct access to Intel Optane or NVDIMM hardware, delivering microsecond I/O latency through the `pmem` package in [`Cubelet/pkg/container/pmem/pmem.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/pkg/container/pmem/pmem.go).
- **Workload selection** occurs via the `cube.pmem` annotation parsed during container creation in [`Cubelet/services/cubebox/cube_container_create.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/services/cubebox/cube_container_create.go).
- **Data integrity** requires synchronous, idempotent destruction for pmem files (implemented in [`Cubelet/services/images/service.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/services/images/service.go) line 302) versus the asynchronous deletion used for regular ext4 images.
- **Kernel optimization** in pmem mode utilizes shared kernel files managed by `EnsureKernelFilePresent` to reduce memory footprint across containers.

## Frequently Asked Questions

### How does CubeSandbox determine whether to use pmem or regular disk for a container?

The controller checks for the `cube.pmem` annotation (defined as `constants.AnnotationPmem` in [`Cubelet/pkg/constants/const.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/pkg/constants/const.go)) during pod creation. If present in the pod specification, the container creation flow collects pmem configuration via `pmem.CubePmem` structs between lines 1026–1074 of [`Cubelet/services/cubebox/cube_container_create.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/services/cubebox/cube_container_create.go). Absent this annotation, the system defaults to the standard ext4 storage backend.

### Why does pmem require synchronous destruction while regular disk uses asynchronous deletion?

Pmem files are directly memory-mapped into the process address space using DAX, meaning the hardware state reflects the file content immediately without page cache buffering. An asynchronous deletion could leave mapped memory in an inconsistent state if the file disappears while still referenced. The synchronous idempotent path in [`Cubelet/services/images/service.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/services/images/service.go) (line 302) ensures atomic cleanup, whereas regular ext4 images tolerate eventual consistency and use the reflink-based pool in [`Cubelet/storage/pool_withreflink.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/storage/pool_withreflink.go).

### What hardware is required to use the pmem backend in CubeSandbox?

The pmem backend requires Intel Optane DC Persistent Memory or compatible NVDIMM hardware exposed through a DAX-enabled filesystem. The base directory (typically `/pmem`) must be initialized once per node via `pmem.Init(config.CubeToolBaseDir)`, as implemented in [`Cubelet/services/images/local.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/services/images/local.go). Without this hardware and initialization, the pmem path generation functions in [`Cubelet/pkg/container/pmem/pmem.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/pkg/container/pmem/pmem.go) cannot construct valid storage locations.

### How does CubeSandbox optimize kernel storage when using persistent memory?

Rather than storing duplicate kernel images for each container, the pmem backend utilizes a shared kernel file pattern. The `pmem.GetSharedKernelFilePath()` function returns a common location, and `pmem.EnsureKernelFilePresent()` (defined in [`Cubelet/internal/cube/server/images/ext4image/utils.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/internal/cube/server/images/ext4image/utils.go)) verifies or refreshes this file before container startup. This approach minimizes persistent memory consumption while ensuring all containers DAX-map the same kernel image for rapid boot.