Sub-60ms VM Cold Start Optimization in CubeSandbox's RustVMM

CubeSandbox's Rust VMM achieves sub-60 millisecond cold start times by writing only anonymous pages to sparse snapshot files, leaving zero-filled memory regions as holes to minimize I/O during VM creation.

CubeSandbox, TencentCloud's open-source sandbox platform, implements aggressive cold start optimization in its Rust-based VMM to launch virtual machines in under 60 milliseconds. By leveraging the Linux pagemap interface and sparse file techniques, the memory_manager module avoids expensive full-memory copies that traditionally delay VM instantiation.

Anonymous Page Filtering with pagemap_anon

The core optimization relies on the observation that guest memory is largely zero-filled at boot. Instead of writing the entire memory footprint to disk, the VMM creates a sparse snapshot file containing only anonymous (Copy-on-Write) pages while leaving non-anonymous regions as holes.

According to the source code in hypervisor/vmm/src/memory_manager.rs, the cold start path is explicitly documented at lines 2378-2380, where the implementation notes that only anonymous pages are written during initial snapshot creation. The actual filtering logic resides at lines 2384-2405, where filter_memory_ranges_by_pagemap_anon processes the memory ranges to identify pages that have been privately mapped via MAP_PRIVATE.

The VMM obtains the set of anonymous pages through the Linux kernel's pagemap interface, which reveals exactly which pages differ from a pristine zero-filled image. This approach yields a self-contained snapshot that can be restored later without additional handling, as implemented at lines 2389-2392.

Sparse File Implementation Details

When no base snapshot exists, the VMM executes the cold start path by creating a new snapshot file and writing only the filtered anonymous pages. The remaining memory regions are left as sparse holes—zeros that consume no disk space and require no write bandwidth.

Logging at lines 2406-2412 reports the size of the anonymous portion and the achieved I/O savings, confirming that the resulting file contains only the essential data needed for a fast VM boot. This optimization cuts the VM creation latency to single-digit milliseconds on typical hardware by reducing the snapshot size from potentially gigabytes to a few megabytes of actual data.

Incremental Snapshots and Soft-Dirty Fallback

While the initial cold start uses pagemap_anon, subsequent snapshots leverage soft-dirty tracking for efficiency. If a base snapshot already exists, the VMM uses send_soft_dirty_memory to overwrite only the anonymous pages that changed, leaving the rest of the file untouched.

To ensure compatibility across kernel configurations, the VMM implements lazy soft-dirty tracking. Soft-dirty tracking arms only after the first successful snapshot; until then, the system falls back to the pagemap_anon path. This guarantees the sub-60ms first-snapshot performance even on kernels without CONFIG_MEM_SOFT_DIRTY.

Code Implementation Examples

The cold start logic is implemented in hypervisor/vmm/src/memory_manager.rs, with detection logic residing in CubeShim/shim/src/container/mod.rs (lines 425-463) where the is_cold_start flag determines which path to execute.

// Cold start: Creating the initial sparse snapshot
let vm = /* obtain VMM instance */;
let dst_url = "file:///var/lib/cubesandbox/snapshots/vm1.snap";
let mem_vol = None; // No base snapshot for cold start

// Triggers pagemap_anon filtering and sparse file creation
vm.memory_manager
   .send_pagemap_anon_memory(dst_url, &mem_vol)
   .expect("failed to create cold-start snapshot");

For incremental snapshots after the initial cold start:

// Incremental update: Overwrite only changed anonymous pages
let dst_url = "file:///var/lib/cubesandbox/snapshots/vm1.snap";
let mem_vol = None;

// Uses soft-dirty tracking for minimal I/O
vm.memory_manager
   .send_soft_dirty_memory(dst_url, &mem_vol)
   .expect("failed to create incremental snapshot");

Unit tests validating these cold start paths are located in hypervisor/vmm/src/lib.rs at lines 2312-2645, covering various scenarios including test_vmm_vm_cold_* test cases.

Summary

• Sparse-file snapshotting with pagemap_anon writes only anonymous pages during cold start, leaving zero-filled regions as holes to minimize I/O.
• In-place overwrite for incremental runs updates only changed pages when a base snapshot exists, avoiding full file rewrites.
• Lazy soft-dirty fallback ensures the sub-60ms cold start path works regardless of kernel configuration, using pagemap_anon until soft-dirty tracking is available.
• The implementation centers on hypervisor/vmm/src/memory_manager.rs with cold start detection in CubeShim/shim/src/container/mod.rs.

Frequently Asked Questions

How does CubeSandbox achieve sub-60ms cold start times?

CubeSandbox achieves sub-60ms cold starts by avoiding full memory dumps during initial VM creation. The VMM uses the Linux pagemap interface to identify only anonymous pages that contain actual data, writing exclusively those pages to a sparse snapshot file while leaving zero-filled memory as unallocated holes. This reduces the snapshot size from hundreds of megabytes to a few megabytes, cutting disk I/O and serialization time dramatically.

What is the difference between pagemap_anon and soft-dirty tracking?

pagemap_anon filters memory to identify only anonymous pages that have been privately mapped, which is ideal for cold starts where no previous snapshot exists. Soft-dirty tracking, used via send_soft_dirty_memory, monitors which pages have been modified since the last snapshot, making it efficient for incremental updates. The VMM uses pagemap_anon for the first snapshot and switches to soft-dirty tracking for subsequent iterations to minimize write amplification.

Where is the cold start detection logic located?

Cold start detection resides in CubeShim/shim/src/container/mod.rs at lines 425-463, where the is_cold_start boolean determines whether the VMM should execute the sparse-file creation path or the incremental update path. This flag propagates to the memory manager in hypervisor/vmm/src/memory_manager.rs, which selects between send_pagemap_anon_memory and send_soft_dirty_memory accordingly.

Why are sparse files critical for this optimization?

Sparse files are critical because they allow the VMM to create a complete memory image structure without writing zero-filled pages to disk. By leaving holes for non-anonymous memory regions, the system avoids wasting disk space and write bandwidth on empty data, enabling the sub-60ms creation time. The Linux kernel automatically handles these holes as zeros when the file is memory-mapped, ensuring the guest VM sees the expected zero-initialized memory without the I/O overhead.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}