How CubeSandbox Achieves Sub-60ms Boot Times with KVM MicroVMs

CubeSandbox achieves sub-60ms cold starts by combining copy-on-write snapshots, pre-allocated memory pools, batch vCPU creation, and a direct kernel boot path that bypasses traditional bootloaders entirely.

TencentCloud's CubeSandbox delivers production-grade sandbox isolation with the startup latency of containers by aggressively optimizing every phase of the KVM MicroVM boot path. By eliminating heavyweight initialization steps and leveraging pre-built resource pools, the project consistently achieves <60ms cold starts and ≈10ms warm starts while maintaining <5MB memory overhead per instance.

Pre-Built Images and Copy-on-Write Snapshots

CubeSandbox eliminates the costly image preparation stage that typically delays VM startup by using frozen templates and efficient storage layering.

Minimalist Kernel and Root Filesystem

Rather than booting a general-purpose distribution, CubeSandbox uses a pre-built minimalist Linux kernel (≈5MB) compiled with CONFIG_INIT_RAMDISK disabled. The root filesystem is stripped to bare essentials—no package manager, no traditional init system, and only the envd agent required for sandbox control. This template is stored as a read-only base image that requires no unpacking or installation at boot time.

CubeCoW Snapshot Engine

New sandboxes start from a Copy-on-Write (CoW) snapshot managed by the CubeCoW engine. Instead of copying the full root filesystem, the system creates a thin writable layer over the frozen template. According to the codebase, this allows the VM to begin execution from a ready-to-run state without waiting for file extraction or disk I/O, effectively reducing the storage initialization phase to near-zero latency.

Fixed Memory Allocation and Batch vCPU Setup

Resource allocation—typically a source of millisecond-scale delays—is streamlined through pre-determination and batching.

Pre-Allocated Guest Memory

In hypervisor/vmm/src/memory_manager.rs, the system allocates a fixed <5MB guest RAM region via a single mmap-ed backing file called boot_guest_memory. This memory region is reserved at VMM initialization and reused for every sandbox instance. By eliminating dynamic heap allocation and page-fault-heavy bootloader work during the hot path, CubeSandbox removes unpredictable memory setup latency from the boot sequence.

Batch vCPU Creation

The vCPU configuration is baked into the static VM definition in hypervisor/vmm/src/vm_config.rs via the boot_vcpus parameter. The VCPU manager creates all virtual CPUs in a single batch invocation (cpu_manager.boot_vcpus()), reducing the number of expensive KVM_CREATE_VCPU syscalls. The implementation in hypervisor/vmm/src/vm.rs further optimizes this by disabling unnecessary MSRs and using the minimal KVM exit path, ensuring the hypervisor touches only essential processor state during initialization.

Streamlined Device and Kernel Initialization

CubeSandbox minimizes the number of devices and eliminates legacy boot stages that typically consume tens of milliseconds.

Essential VirtIO Devices Only

Only three virtio devices are instantiated before the VM launches: a block device for the read-only template rootfs, a vsock device for host-guest control, and a tap device for networking (referenced in network-agent/internal/service/tap_fd_provider.go). By creating these devices entirely before VM execution begins, the guest kernel discovers them instantly without triggering I/O-related VM exits during early boot.

Direct Kernel Loading Without Bootloader

The VMM loads the kernel image directly into guest memory in hypervisor/vmm/src/vm.rs::boot, bypassing GRUB, UEFI, or any traditional bootloader. The kernel boots with init=/sbin/init as the only userspace process, skipping init-random-disk decompression and legacy hardware probing. This direct injection reduces the boot path to the absolute minimum number of execution steps.

The Millisecond-Scale Boot Sequence

The actual VM launch executes through a fast-path routine that keeps all operations in user space.

The Vm::boot Hot Path

The core boot logic resides in Vm::boot() within hypervisor/vmm/src/vm.rs. This function performs exactly three steps: loading the kernel image and command line, setting up the boot vCPU entry point, and starting the vCPUs via start_boot_vcpus(). The entire sequence is instrumented with tracing events (event!("vm", "booted")), confirming that guest code begins executing within approximately 20 microseconds after the final vCPU start instruction.

A minimal implementation of this flow appears as follows:

// Pseudo-code extracted from hypervisor/vmm/src/vm.rs
fn fast_boot(vm_cfg: Arc<Mutex<VmConfig>>, mem: GuestMemoryMmap) -> Result<()> {
    // 1️⃣ Load the kernel image (already built for the sandbox)
    let kernel = std::fs::read("kernel/Image")?;
    vm_cfg.lock().unwrap().kernel_image = kernel;

    // 2️⃣ Prepare boot vCPUs (single-core for most sandboxes)
    let boot_vcpus = 1;
    vm_cfg.lock().unwrap().cpus.boot_vcpus = boot_vcpus;

    // 3️⃣ Create the VM struct and attach the pre-allocated memory
    let mut vm = Vm::new(vm_cfg.clone(), mem.clone())?;

    // 4️⃣ Boot the VM – this is the hot path
    vm.boot()?;               // Calls `event!("vm", "booted")` internally

    Ok(())
}

Warm Pool Optimization

For hot-path scenarios, the Cubelet service maintains a pool of pre-allocated VM structs. When a new sandbox is requested, the VMM clones an entry from this pool rather than allocating fresh resources. According to comments in the networking layer regarding concurrent boot latency, this approach eliminates allocation contention and enables parallel sandbox creation with negligible overhead, resulting in ≈10ms warm starts when the VM structure is already resident in memory.

Seccomp Isolation for Predictable Latency

CubeSandbox runs the VMM under a hardened seccomp profile that whitelists only the syscalls required for boot: mmap, ioctl(KVM_*), and basic read/write operations. By blocking unused syscalls, the system prevents kernel side-channel overhead and context-switch penalties that could otherwise introduce jitter into the sub-60ms boot timeline. This security hardening ensures that the fast path remains deterministic without sacrificing isolation guarantees.

Summary

• Copy-on-Write snapshots eliminate image unpacking and filesystem initialization delays
• Fixed memory pools (boot_guest_memory) remove dynamic allocation from the boot path
• Batch vCPU creation (boot_vcpus) minimizes expensive KVM hypercalls
• Direct kernel loading bypasses bootloaders and initramdisk decompression entirely
• Warm pools in Cubelet enable sub-10ms starts for pre-allocated VM structs
• Seccomp whitelisting ensures syscall overhead does not perturb latency guarantees

Frequently Asked Questions

What makes KVM MicroVMs faster than traditional VMs for sandboxing?

KVM MicroVMs presented by CubeSandbox strip away legacy hardware emulation, BIOS phases, and full device models that characterize traditional VMs. By using a minimalist kernel, direct memory-mapped loading, and only three essential virtio devices, the system avoids the hundreds of milliseconds typically spent in bootloader menus, hardware probing, and driver initialization that plague conventional virtual machines.

How does CubeSandbox's Copy-on-Write (CoW) snapshot engine reduce boot latency?

The CubeCoW engine allows new sandboxes to mount a writable layer atop a frozen, read-only root filesystem template rather than copying or extracting image data. Because the guest sees a complete filesystem immediately without requiring block-level writes during initialization, the storage setup phase completes in microseconds rather than the hundreds of milliseconds required to unpack a traditional disk image.

What is the difference between cold start and warm pool boot times in CubeSandbox?

Cold starts (≈60ms) occur when the VMM must initialize a new VM struct, allocate fixed memory, and execute the full Vm::boot() sequence from scratch. Warm starts (≈10ms) occur when the Cubelet service reclaims a VM from its pre-allocated pool, requiring only a lightweight clone operation and the hot-path boot sequence without resource allocation overhead.

How does CubeSandbox maintain security while achieving millisecond-scale boot times?

Security is enforced through seccomp profiles that whitelist only the six to eight syscalls required for KVM operation, preventing malicious or errant system calls from adding latency. Additionally, the minimal attack surface—achieved by removing package managers, shells, and unnecessary kernel modules—ensures that hardening checks do not add overhead to the critical boot path, unlike traditional VMs that must initialize complex security frameworks during startup.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}