How CubeCoW Enables O(1) Snapshots Using XFS Reflink in CubeSandbox

CubeCoW achieves constant-time snapshot creation by leveraging the XFS reflink (FICLONE) ioctl, which clones file metadata without copying data blocks until write time.

The TencentCloud/CubeSandbox project uses CubeCoW (cubecow) as its storage engine to provide high-performance container snapshots. By exploiting the reflink capability of XFS and Btrfs filesystems, CubeCoW eliminates the linear copy penalty traditionally associated with snapshot operations. This article examines the source code implementation to explain how CubeCoW O(1) snapshots using XFS reflink achieve constant complexity through kernel-level copy-on-write semantics.

The Architecture of CubeCoW Reflink Snapshots

CubeCoW implements a three-layer architecture that delegates heavy lifting to the kernel's reflink mechanism. This design ensures that snapshot creation time remains constant regardless of volume size.

Reflink-Capable Backing Filesystem

All volume files reside as regular files on a reflink-capable filesystem such as XFS or Btrfs. As documented in Cubelet/pkg/cubecow/doc.go, the engine requires "regular files on a reflink-capable filesystem (XFS or Btrfs)" to function. The kernel provides the FICLONE ioctl on these filesystems, enabling block-level sharing between files without duplicating data.

The Rust-Based Copy-on-Write Engine

The core cloning logic resides in a Rust crate wrapped by Go bindings in Cubelet/pkg/cubecow/cubecow.go. When creating a snapshot, the engine issues ioctl(FICLONE) to perform a reflink clone. This operation creates a new inode that shares all data blocks with the source file while consuming only metadata space. The kernel handles copy-on-write automatically when either file is modified, copying only the specific blocks that change rather than the entire volume.

Implementation in CubeSandbox Storage Layer

The Cubelet storage layer orchestrates reflink operations through specific backend selection and pool management strategies.

Backend Selection and Configuration

In Cubelet/storage/plugin.go, the constant cowBackendReflink = "reflink" identifies the reflink backend at lines 31-35. Administrators enable this backend by setting pool_type: copy_reflink in the Cubelet configuration (e.g., configs/single-node/cubelet.yaml). This configuration directs the storage layer to use the cubecow reflink-only backend for all copy-on-write operations.

Pool Management with Reflink

The file Cubelet/storage/pool_withreflink.go implements the poolWithReflink structure that manages pre-created ext4 image files on the XFS backing store. The init method (lines 44-55) maps cp_reflink_type to a pool instance. When allocating new volumes, the pool invokes InitBaseFile and GetSync to generate file paths, then delegates the actual cloning operation to the cubecow manager. This architecture minimizes overhead because the pool logic remains minimal while the kernel handles block management.

Snapshot API Operations

High-level snapshot functions in Cubelet/storage/cubecow_snapshot_artifacts.go provide thin wrappers around the reflink engine. The CommitTemplateRootfs function (lines 84-96) and CommitTemplateMemoryFromBase (lines 23-30) validate arguments before calling manager.CommitTemplateRootfs or manager.CommitTemplateMemory. These manager methods internally execute the reflink Clone operation, triggering the O(1) snapshot creation.

Zero-Copy Snapshot Creation Process

When CubeSandbox pauses a sandbox or commits a template, CubeCoW creates snapshots without copying data. The process works as follows:

1. The storage layer identifies the source volume file on XFS.
2. The cubecow manager invokes the Rust-based Clone operation via ioctl(FICLONE).
3. The kernel creates a new file inode that references the same data blocks as the source.
4. The snapshot completes in constant time because only metadata changes, not data blocks.

Later writes to either the source or snapshot trigger the kernel's copy-on-write mechanism, duplicating only the modified blocks while preserving the original data for the snapshot. This ensures that snapshot creation cost depends solely on metadata operations, not volume size.

Practical Usage Examples

The following examples demonstrate reflink snapshot creation using the Go API and command-line interface.

Creating Snapshots via Go API

ctx := context.TODO()
srcSnap, _ := storage.GetSandboxRootfsForSnapshot(ctx, sandboxID, "")
newSnap, err := storage.CommitTemplateRootfs(ctx, srcSnap, templateID)
if err != nil {
    log.Fatalf("snapshot failed: %v", err)
}
fmt.Printf("New snapshot %s at %s (size %d)\n",
    newSnap.Name, newSnap.DevPath, newSnap.SizeBytes)

The CommitTemplateRootfs call triggers the reflink clone operation described in cubecow_snapshot_artifacts.go, executing the O(1) snapshot logic.

Command-Line Usage

The example program in Cubelet/pkg/cubecow/examples/go-test/main.go demonstrates backend selection:

./cubecow-test -backend reflink -reflink-root-dir /var/lib/cubecow/reflink

This invocation uses cp --reflink=always internally (as shown in lines 140-149) to create reflink copies and report the resulting device paths.

Summary

• CubeCoW leverages XFS/Btrfs reflink: The storage engine requires a reflink-capable filesystem to provide kernel-level FICLONE support, as specified in Cubelet/pkg/cubecow/doc.go.
• O(1) complexity via metadata cloning: Snapshots complete in constant time because the ioctl(FICLONE) operation duplicates only inode metadata, not data blocks.
• Rust engine with Go bindings: The actual reflink operation is implemented in the Rust cubecow crate and exposed through Cubelet/pkg/cubecow/cubecow.go.
• Pool-based allocation: The poolWithReflink implementation in Cubelet/storage/pool_withreflink.go manages volume files on XFS backing stores.
• Lazy copy-on-write: Data blocks are copied only when modified after snapshot creation, ensuring efficient space utilization.

Frequently Asked Questions

What filesystems support CubeCoW reflink snapshots?

CubeCoW requires XFS or Btrfs filesystems that support the FICLONE ioctl. According to Cubelet/pkg/cubecow/doc.go, these filesystems provide the necessary reflink capability for the copy-on-write engine to function correctly.

Why are CubeCoW snapshots O(1) instead of O(n)?

Traditional snapshots copy all data blocks, resulting in O(n) complexity where n is the volume size. CubeCoW uses the kernel's reflink feature to share data blocks between source and snapshot files, modifying only metadata. As implemented in the Rust cubecow crate, this approach makes snapshot creation time independent of volume size.

How does copy-on-write work after a reflink snapshot?

After the initial reflink clone, both files share identical data blocks. When either file is modified, the kernel's copy-on-write mechanism duplicates only the specific block being written, leaving the original block unchanged for the snapshot. This lazy copying ensures that only changed blocks consume additional storage space.

Where is the reflink clone operation implemented?

The clone operation is implemented in the Rust cubecow crate and wrapped by the Go API in Cubelet/pkg/cubecow/cubecow.go. The high-level storage APIs in Cubelet/storage/cubecow_snapshot_artifacts.go invoke this functionality through methods like CommitTemplateRootfs and CommitTemplateMemoryFromBase.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}