# Sub-60ms VM Cold Start Optimization in CubeSandbox's RustVMM

> Discover how CubeSandbox's RustVMM achieves sub-60ms VM cold starts by optimizing sparse snapshot files and minimizing I/O for faster VM creation.

- Repository: [Tencent Cloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox)
- Tags: performance
- Published: 2026-07-16

---

**CubeSandbox's Rust VMM achieves sub-60 millisecond cold start times by writing only anonymous pages to sparse snapshot files, leaving zero-filled memory regions as holes to minimize I/O during VM creation.**

CubeSandbox, TencentCloud's open-source sandbox platform, implements aggressive cold start optimization in its Rust-based VMM to launch virtual machines in under 60 milliseconds. By leveraging the Linux pagemap interface and sparse file techniques, the `memory_manager` module avoids expensive full-memory copies that traditionally delay VM instantiation.

## Anonymous Page Filtering with pagemap_anon

The core optimization relies on the observation that guest memory is largely zero-filled at boot. Instead of writing the entire memory footprint to disk, the VMM creates a sparse snapshot file containing only **anonymous (Copy-on-Write) pages** while leaving non-anonymous regions as holes.

According to the source code in [`hypervisor/vmm/src/memory_manager.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/vmm/src/memory_manager.rs), the cold start path is explicitly documented at lines 2378-2380, where the implementation notes that only anonymous pages are written during initial snapshot creation. The actual filtering logic resides at lines 2384-2405, where `filter_memory_ranges_by_pagemap_anon` processes the memory ranges to identify pages that have been privately mapped via `MAP_PRIVATE`.

The VMM obtains the set of anonymous pages through the Linux kernel's **pagemap** interface, which reveals exactly which pages differ from a pristine zero-filled image. This approach yields a **self-contained snapshot** that can be restored later without additional handling, as implemented at lines 2389-2392.

## Sparse File Implementation Details

When no base snapshot exists, the VMM executes the cold start path by creating a new snapshot file and writing only the filtered anonymous pages. The remaining memory regions are left as sparse holes—zeros that consume no disk space and require no write bandwidth.

Logging at lines 2406-2412 reports the size of the anonymous portion and the achieved I/O savings, confirming that the resulting file contains only the essential data needed for a fast VM boot. This optimization cuts the VM creation latency to single-digit milliseconds on typical hardware by reducing the snapshot size from potentially gigabytes to a few megabytes of actual data.

## Incremental Snapshots and Soft-Dirty Fallback

While the initial cold start uses `pagemap_anon`, subsequent snapshots leverage **soft-dirty tracking** for efficiency. If a base snapshot already exists, the VMM uses `send_soft_dirty_memory` to overwrite only the anonymous pages that changed, leaving the rest of the file untouched.

To ensure compatibility across kernel configurations, the VMM implements **lazy soft-dirty tracking**. Soft-dirty tracking arms only after the first successful snapshot; until then, the system falls back to the `pagemap_anon` path. This guarantees the sub-60ms first-snapshot performance even on kernels without `CONFIG_MEM_SOFT_DIRTY`.

## Code Implementation Examples

The cold start logic is implemented in [`hypervisor/vmm/src/memory_manager.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/vmm/src/memory_manager.rs), with detection logic residing in [`CubeShim/shim/src/container/mod.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeShim/shim/src/container/mod.rs) (lines 425-463) where the `is_cold_start` flag determines which path to execute.

```rust
// Cold start: Creating the initial sparse snapshot
let vm = /* obtain VMM instance */;
let dst_url = "file:///var/lib/cubesandbox/snapshots/vm1.snap";
let mem_vol = None; // No base snapshot for cold start

// Triggers pagemap_anon filtering and sparse file creation
vm.memory_manager
   .send_pagemap_anon_memory(dst_url, &mem_vol)
   .expect("failed to create cold-start snapshot");

```

For incremental snapshots after the initial cold start:

```rust
// Incremental update: Overwrite only changed anonymous pages
let dst_url = "file:///var/lib/cubesandbox/snapshots/vm1.snap";
let mem_vol = None;

// Uses soft-dirty tracking for minimal I/O
vm.memory_manager
   .send_soft_dirty_memory(dst_url, &mem_vol)
   .expect("failed to create incremental snapshot");

```

Unit tests validating these cold start paths are located in [`hypervisor/vmm/src/lib.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/vmm/src/lib.rs) at lines 2312-2645, covering various scenarios including `test_vmm_vm_cold_*` test cases.

## Summary

- **Sparse-file snapshotting** with `pagemap_anon` writes only anonymous pages during cold start, leaving zero-filled regions as holes to minimize I/O.
- **In-place overwrite** for incremental runs updates only changed pages when a base snapshot exists, avoiding full file rewrites.
- **Lazy soft-dirty fallback** ensures the sub-60ms cold start path works regardless of kernel configuration, using `pagemap_anon` until soft-dirty tracking is available.
- The implementation centers on [`hypervisor/vmm/src/memory_manager.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/vmm/src/memory_manager.rs) with cold start detection in [`CubeShim/shim/src/container/mod.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeShim/shim/src/container/mod.rs).

## Frequently Asked Questions

### How does CubeSandbox achieve sub-60ms cold start times?

CubeSandbox achieves sub-60ms cold starts by avoiding full memory dumps during initial VM creation. The VMM uses the Linux pagemap interface to identify only anonymous pages that contain actual data, writing exclusively those pages to a sparse snapshot file while leaving zero-filled memory as unallocated holes. This reduces the snapshot size from hundreds of megabytes to a few megabytes, cutting disk I/O and serialization time dramatically.

### What is the difference between pagemap_anon and soft-dirty tracking?

**`pagemap_anon`** filters memory to identify only anonymous pages that have been privately mapped, which is ideal for cold starts where no previous snapshot exists. **Soft-dirty tracking**, used via `send_soft_dirty_memory`, monitors which pages have been modified since the last snapshot, making it efficient for incremental updates. The VMM uses `pagemap_anon` for the first snapshot and switches to soft-dirty tracking for subsequent iterations to minimize write amplification.

### Where is the cold start detection logic located?

Cold start detection resides in [`CubeShim/shim/src/container/mod.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeShim/shim/src/container/mod.rs) at lines 425-463, where the `is_cold_start` boolean determines whether the VMM should execute the sparse-file creation path or the incremental update path. This flag propagates to the memory manager in [`hypervisor/vmm/src/memory_manager.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/vmm/src/memory_manager.rs), which selects between `send_pagemap_anon_memory` and `send_soft_dirty_memory` accordingly.

### Why are sparse files critical for this optimization?

Sparse files are critical because they allow the VMM to create a complete memory image structure without writing zero-filled pages to disk. By leaving holes for non-anonymous memory regions, the system avoids wasting disk space and write bandwidth on empty data, enabling the sub-60ms creation time. The Linux kernel automatically handles these holes as zeros when the file is memory-mapped, ensuring the guest VM sees the expected zero-initialized memory without the I/O overhead.