How CubeHypervisor Manages KVM MicroVM Lifecycle: A Complete Technical Guide

CubeHypervisor manages KVM MicroVM lifecycle through a six-phase process: hypervisor detection via hypervisor::new(), VM instantiation with create_vm(), memory and VCPU configuration, event-driven execution via KVM_RUN, stateful pause/resume operations, and automatic resource cleanup through Rust's Drop trait.

CubeHypervisor abstracts the KVM kernel-mode virtual machine behind a uniform interface, enabling lightweight KVM MicroVM (PVM) orchestration in the TencentCloud CubeSandbox repository. Understanding this lifecycle is essential for developers building secure, high-performance virtualization layers.

Detecting and Instantiating the KVM Hypervisor

The lifecycle begins with hypervisor discovery in hypervisor/hypervisor/src/kvm/mod.rs. The new() function probes the host kernel for KVM capabilities before returning a concrete KvmHypervisor instance.

// hypervisor/hypervisor/src/kvm/mod.rs
pub fn new() -> Result<Arc<dyn Hypervisor>> {
    if kvm::KvmHypervisor::is_available()? {
        Ok(Arc::new(KvmHypervisor::new()?))
    } else {
        Err(HypervisorError::Unsupported)
    }
}

This function validates the kernel interface by checking KVM_GET_API_VERSION and required extensions via check_required_kvm_extensions. When the KVM PVM (micro-VM) variant is selected, the hypervisor type is set to HypervisorType::KvmPvm, distinguishing it from legacy full-VM mode.

Creating the KVM MicroVM Instance

Once the hypervisor exists, the VMM creates a VM object through the create_vm() or create_vm_with_type() methods. In hypervisor/hypervisor/src/kvm/mod.rs, these methods wrap the low-level KVM VM file descriptor in a KvmVm struct.

// hypervisor/hypervisor/src/kvm/mod.rs
fn create_vm(&self) -> hypervisor::Result<Arc<dyn vm::Vm>> {
    self.create_vm_with_type(KvmVmType::LegacyVm)
}

fn create_vm_with_type(&self, vm_type: u64) -> hypervisor::Result<Arc<dyn vm::Vm>> {
    let vm = self.kvm.create_vm_with_type(vm_type)?;
    Ok(Arc::new(KvmVm { inner: vm, ... }))
}

The resulting KvmVm holds the VM FD, memory-slot bookkeeping structures, and sec-comp filters. According to the source in hypervisor/vmm/src/vm.rs (line 466), the VMM selects the hypervisor type before invoking these factory methods.

Configuring Memory and Virtual Devices

Configuration happens in three parallel tracks: memory regions, VCPU structures, and virtio devices.

Memory Registration

The MemoryManager registers guest physical memory with the KVM kernel via KvmVm::create_user_memory_region:

// hypervisor/vmm/src/memory_manager.rs
pub fn create_user_memory_region(&self, region: MemoryRegion) -> Result<()> {
    self.vm.create_user_memory_region(region)?;
    // registers the region in the KVM memory-slot table
}

VCPU Creation

Virtual CPUs are instantiated through the VM object:

// hypervisor/vmm/src/vcpu.rs
pub fn create_vcpu(&self, id: u64) -> Result<Arc<dyn cpu::Vcpu>> {
    let vcpu = self.vm.create_vcpu(id)?;
    Ok(Arc::new(KvmVcpu { inner: vcpu, ... }))
}

Device Attachment

Virtio devices attach to the VM through the transport layer:

// hypervisor/virtio-devices/src/device.rs
impl Device {
    pub fn attach(&mut self, vm: &Arc<dyn vm::Vm>) -> Result<()> {
        // queries hypervisor for configuration
    }
}

All configuration flows converge in hypervisor/vmm/src/vm.rs during the initialization phase (lines 770-800).

Executing the VCPU Event Loop

Each VCPU runs in a dedicated thread that enters the kernel via KvmVcpu::run():

// hypervisor/vmm/src/vcpu.rs
pub fn run(&self) -> Result<VmExit> {
    self.inner.run()  // → KVM_RUN ioctl
}

The KVM_RUN ioctl returns a VmExit enum that drives the event loop. The VmExit type (defined in hypervisor/vmm/src/vm_exit.rs) handles I/O port accesses, MMIO operations, and hypervisor-specific events:

match vm_exit {
    VmExit::IoIn { port, .. } => handle_io_in(port),
    VmExit::MmioRead { addr, .. } => handle_mmio(addr),
    VmExit::Hlt => break,  // graceful shutdown
    ...
}

This loop continues until the guest executes a halt instruction or triggers a shutdown event.

Pausing and Resuming MicroVM Execution

CubeHypervisor supports live pause/resume by saving and restoring VCPU state.

Pausing captures the current register state and stops execution:

// hypervisor/vmm/src/vcpu.rs
pub fn pause(&self) -> Result<()> {
    let state = self.get_state()?;
    self.inner.pause()?;  // stops KVM_RUN
    self.saved_state = Some(state);
    Ok(())
}

Resuming restores the saved state and re-enters the run loop:

pub fn resume(&self) -> Result<()> {
    if let Some(state) = self.saved_state.take() {
        self.set_state(state)?;
        self.run()?;
    }
    Ok(())
}

The get_state() and set_state() methods serialize the VCPU's internal registers and special registers for migration or snapshotting.

Shutdown and Resource Cleanup

When the Arc<KvmVm> is dropped, Rust's Drop trait automatically releases kernel resources:

// hypervisor/hypervisor/src/kvm/kvm_vm.rs
impl Drop for KvmVm {
    fn drop(&mut self) {
        // implicit close of the underlying VM fd
        // memory slots automatically released by the kernel
    }
}

This ensures that the VM FD closes and all memory slots deregister without manual intervention, preventing file descriptor leaks in long-running VMM processes.

Complete Lifecycle Example

The following Rust code demonstrates the full KVM MicroVM lifecycle from detection to shutdown:

use hypervisor::{new, Hypervisor, HypervisorType};
use hypervisor::vmm::{VmConfig, VmExit};

fn start_micro_vm() -> anyhow::Result<()> {
    // 1. Detect KVM hypervisor
    let hv = hypervisor::new()?;
    assert_eq!(hv.hypervisor_type(), HypervisorType::KvmPvm);

    // 2. Create VM instance
    let vm = hv.create_vm()?;

    // 3. Configure 1GiB memory and one VCPU
    vm.create_user_memory_region(VmConfig::default_memory(1 << 30))?;
    let vcpu = vm.create_vcpu(0)?;

    // 4. Run event loop
    loop {
        match vcpu.run()? {
            VmExit::Hlt => break,
            VmExit::Shutdown => break,
            _ => {}  // handle I/O, MMIO, interrupts
        }
    }

    // 5. Automatic cleanup on scope exit
    Ok(())
}

Summary

• CubeHypervisor abstracts KVM through a type-safe Rust interface that supports both legacy VMs and lightweight KVM MicroVM (PVM) instances.
• The lifecycle follows six distinct phases: detection via hypervisor::new(), creation via create_vm_with_type(), configuration of memory and devices, execution via KvmVcpu::run(), pause/resume through state save/restore, and cleanup via the Drop trait.
• Key source files include hypervisor/hypervisor/src/kvm/mod.rs for hypervisor management, hypervisor/vmm/src/vcpu.rs for CPU lifecycle, and hypervisor/hypervisor/src/kvm/kvm_vm.rs for resource cleanup.
• The VmExit enum in hypervisor/vmm/src/vm_exit.rs defines the contract between the KVM kernel and userspace event loop.

Frequently Asked Questions

What is the difference between KVM PVM and Legacy VM in CubeHypervisor?

KVM PVM (Protected Virtual Machine) represents the micro-VM variant optimized for lightweight isolation, while Legacy VM refers to traditional full-virtualization mode. In hypervisor/hypervisor/src/kvm/mod.rs, the create_vm_with_type() method accepts a vm_type parameter that selects between these modes. The PVM type sets up reduced emulation overhead and tighter sec-comp filters compared to the legacy path.

How does pause/resume work at the KVM kernel level?

Pause operations invoke KvmVcpu::get_state() to serialize the VCPU's register file (including general-purpose and special registers) into a VcpuState structure, then signal the KVM kernel to exit the KVM_RUN ioctl. Resume operations call set_state() to restore registers before re-entering run(). This state management enables live migration and snapshotting without guest awareness.

Where does memory cleanup happen when a MicroVM shuts down?

Memory cleanup occurs automatically in the Drop implementation of KvmVm located in hypervisor/hypervisor/src/kvm/kvm_vm.rs. When the last Arc<KvmVm> reference is dropped, the destructor closes the VM file descriptor, which causes the kernel to release all associated memory slots and virtual CPUs. This Rust-specific pattern eliminates resource leaks without explicit cleanup code.

What triggers different VmExit conditions during VCPU execution?

The VmExit enum in hypervisor/vmm/src/vm_exit.rs covers hardware-assisted exits including IoIn/IoOut for port-mapped I/O, MmioRead/MmioWrite for memory-mapped device access, and Hlt for guest-initiated shutdown. Additionally, Shutdown indicates a triple-fault or architectural exit, while interrupt-related exits handle virtio device events. The VMM's event loop processes these exits to emulate hardware behavior.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}