How CubeCoW Implements Copy-on-Write Snapshots with Hundred-Millisecond Granularity

CubeCoW achieves sub-second snapshot latency by combining filesystem reflinks for instant metadata-only cloning with a lightweight copy-on-write overlay that tracks only anonymous page writes.

CubeCoW is the storage engine powering TencentCloud's CubeSandbox platform. It delivers copy-on-write snapshots with hundred-millisecond granularity by leveraging Linux reflink capabilities and a purpose-built overlay system. This architecture allows developers to create consistent point-in-time snapshots of running sandboxes in approximately 50–150 milliseconds, regardless of volume size.

The Reflink Foundation: Zero-Copy Snapshot Creation

Traditional snapshot methods copy data blocks serially, creating latency proportional to volume size. CubeCoW eliminates this bottleneck by using filesystem reflinks to create block-device clones that share underlying data blocks with the parent volume.

The implementation resides in Cubelet/storage/pool.go, where the snapshot path sets cp_type = copy_reflink and invokes the CopyReflink helper. This helper, defined in Cubelet/storage/shell.go, executes the kernel's ioctl(FICLONE) system call. Because reflink operations modify only filesystem metadata without duplicating actual data blocks, the clone operation completes in a few milliseconds even for multi-gigabyte volumes.

The CoW Overlay for Anonymous Pages

After establishing the reflink clone, CubeCoW attaches a metadata-only snapshot object (CowSnapshotObject) that manages write tracking. This overlay lives in Cubelet/storage/cubecow_snapshot_artifacts.go, specifically implementing the logic described in lines 124–130: "memory snapshot … only writes CoW anonymous pages".

The overlay operates on a simple principle: reads are served from the immutable parent snapshot, while the first write to any block triggers a real copy-on-write at the block level. The RollbackDeriveNewGen function (line 198) initializes this overlay, creating a thin layer that intercepts writes without requiring upfront data duplication.

Achieving Sub-Second Granularity

The hundred-millisecond guarantee stems from the fact that reflink cloning is purely a metadata operation. No data movement occurs during snapshot creation, so latency depends only on overlay initialization overhead.

On supported filesystems (ext4 and XFS with reflink enabled), typical completion times range from 50–150 milliseconds. The test suite in Cubelet/storage/shell_test.go validates this performance, confirming that reflink copies of 1 GB volumes finish well under one second regardless of actual data occupancy.

Instant Rollback Without Data Movement

Restoring from a snapshot reattaches the CoW overlay to a new sandbox instance and reuses the existing reflink clone. Because the snapshot already shares data blocks with the parent, rollback requires zero data movement and completes within a few hundred milliseconds.

This flow is implemented in Cubelet/storage/cubecow_engine.go through the CreateSnapshot and DeleteSnapshot APIs. The Python demonstration in examples/snapshot-rollback-clone/rollback_demo.py exercises this path, printing timestamps that verify sub-second rollback latency.

Code Example: Snapshot and Rollback in Milliseconds

The following Go snippet demonstrates the complete snapshot creation and rollback flow, with both operations completing in approximately 100 ms:

// Assume `engine` is a *cubecow.Engine that has been initialised.
srcVol   := "vol-12345"          // existing volume name
snapName := "snap-quick"         // desired snapshot name

// 1️⃣ Create a reflink‑based snapshot (fast)
snapPath, err := engine.CreateSnapshot(srcVol, snapName)
if err != nil { log.Fatalf("snapshot failed: %v", err) }
// snapPath now points to a new block device that shares data with srcVol

// 2️⃣ Rollback a sandbox to the newly‑created snapshot
sandboxID := "sandbox-abc"
_, err = engine.RollbackDeriveNewGen(context.Background(),
    sandboxID, snapPath, 0, 0) // new generation‑ID = 0, size unchanged
if err != nil { log.Fatalf("rollback failed: %v", err) }

// The two calls above each finish in ~100 ms on a typical VM.

This pattern is replicated in the official Python demo examples/snapshot-rollback-clone/rollback_demo.py, which instruments each operation to display the actual latency achieved on running systems.

Summary

• CubeCoW uses ioctl(FICLONE) via CopyReflink in Cubelet/storage/shell.go to create metadata-only reflink clones in milliseconds.
• The CowSnapshotObject in Cubelet/storage/cubecow_snapshot_artifacts.go provides a lightweight overlay that tracks anonymous page writes without copying data upfront.
• Snapshot creation completes in 50–150 ms because no data blocks are duplicated during the initial clone operation.
• Rollback reattaches existing overlays and reflink clones, requiring no data movement and finishing in sub-second timeframes.
• The architecture is validated by Cubelet/storage/shell_test.go and demonstrated in examples/snapshot-rollback-clone/rollback_demo.py.

Frequently Asked Questions

What filesystems support CubeCoW's hundred-millisecond snapshots?

CubeCoW requires ext4 or XFS with reflink support enabled. The reflink capability depends on the FICLONE ioctl, which is available on modern Linux kernels when the filesystem is mounted with the appropriate options. Without reflink support, the system falls back to block-level copying, which breaks the sub-second granularity guarantee.

How does CubeCoW handle writes after a snapshot is created?

After snapshot creation, the CowSnapshotObject intercepts write operations to the cloned volume. The first write to any block triggers a true copy-on-write event: the engine allocates a new block, copies the original data from the parent snapshot, and redirects the write to this new block. Subsequent reads to modified blocks are served from this new location, while unmodified blocks continue to be served from the shared parent via the reflink.

What is the difference between CubeCoW's reflink cloning and traditional LVM snapshots?

Traditional LVM snapshots use a copy-on-write table that redirects reads and writes through a snapshot layer, often introducing performance overhead and requiring pre-allocated snapshot space. CubeCoW's reflink approach creates a true independent block device that shares data blocks at the filesystem level, eliminating the need for a copy-on-write table for existing data. This allows instant creation and deletion without performance degradation or capacity planning constraints.

Can CubeCoW snapshots be used across different storage pools?

No, CubeCoW snapshots are pool-local because reflink clones require the source and destination inodes to reside on the same filesystem. The snapshot catalog in Cubelet/storage/snapshot_catalog.go maintains mappings between snapshot IDs and their local paths. Moving a snapshot to a different pool requires a full data copy, which is handled by separate migration utilities rather than the CoW snapshot mechanism.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}