CubeHypervisor KVM MicroVM Configuration and Management Internals

CubeSandbox implements a hypervisor-agnostic architecture that isolates KVM-specific logic behind Rust traits, using a three-tier system of generic hypervisor abstractions, strongly-typed configuration parsers, and orchestrated VM lifecycle management to enable secure, lightweight micro-VMs.

The TencentCloud CubeSandbox project provides a secure container runtime built on KVM-based micro-virtualization. At its core, the CubeHypervisor component abstracts KVM internals through a clean trait-based interface, allowing the same configuration and management logic to target multiple hypervisor backends including standard KVM and Intel TDX. This design decouples the user-facing configuration API from the underlying ioctl-heavy KVM implementation.

Hypervisor-Agnostic Core Architecture

CubeHypervisor defines a generic hypervisor interface in hypervisor/hypervisor/src/hypervisor.rs that masks platform-specific details behind a set of Rust traits. The Hypervisor trait declares the fundamental operations required for any backend implementation:

pub trait Hypervisor: Send + Sync {
    fn hypervisor_type(&self) -> HypervisorType;
    fn create_vm(&self) -> Result<Arc<dyn Vm>>;
    fn create_vm_with_type(&self, _vm_type: u64) -> Result<Arc<dyn Vm>> { unreachable!() }
    #[cfg(target_arch = "x86_64")]
    fn get_cpuid(&self) -> Result<Vec<CpuIdEntry>>;
    // … other optional helpers …
}

The concrete KVM implementation resides in hypervisor/kvm/src/lib.rs, where KvmHypervisor implements this trait. When the orchestrator calls create_vm, the KVM backend returns a KvmVm instance that handles the low-level ioctl plumbing, including identity map setup (set_identity_map_address), TSS address configuration (set_tss_address), and split IRQ enablement (enable_split_irq). This abstraction allows the rest of the codebase to operate on generic Vm objects without direct KVM dependencies.

VM Configuration Pipeline

All micro-VM parameters flow through a strongly-typed configuration system centered in hypervisor/vmm/src/config.rs. The VmParams::from_arg_matches function parses command-line arguments using the option_parser crate, converting strings like --cpus boot=2,max=4 into structured Rust types:

• CPU configuration – CpusConfig parses boot and maximum vCPU counts, topology, and CPU features, validating against errors like ValidationError::CpuTopologyCount.
• Memory configuration – MemoryConfig handles base size, huge-pages, hot-plug methods (Acpi vs VirtioMem), and per-NUMA zone definitions.
• Device configuration – DiskConfig, NetConfig, and FsConfig provide parse methods that transform comma-separated strings into typed structs with validation.

The resulting VmConfig is stored in a thread-safe Arc<Mutex<_>> and consumed by the orchestrator during VM instantiation.

MicroVM Lifecycle Orchestration

The Vm::new function in hypervisor/vmm/src/vm.rs serves as the entry point for micro-VM creation, executing a six-stage initialization sequence:

1. Hypervisor VM creation – Calls Self::create_hypervisor_vm to instantiate the underlying KVM VM via hypervisor.create_vm() or create_vm_with_type for TDX-specific configurations.
2. Physical address space calculation – Computes guest-visible physical address width via physical_bits, clipping to host limits and handling special cases like KVM-PVM (max 43 bits).
3. Memory manager initialization – MemoryManager::new allocates guest RAM, shared memory regions, and optional SGX EPC pages based on VmConfig.memory and the calculated physical bits.
4. Device manager setup – DeviceManager::new constructs virtual PCI, MMIO, and I/O buses, registers virtio devices (network, block, filesystem, vsock, balloon, RNG), and initializes console/PTTY infrastructure.
5. Kernel loading – Vm::load_payload handles ELF/PE loading via linux_loader, supporting kernel-only, firmware-only, or combined boot paths, storing the entry point in an EntryPoint structure.
6. System configuration – Vm::configure_system injects ACPI tables, initramfs, kernel command-line, and metadata (serial number, UUID, OEM strings) through architecture-specific arch::configure_system calls.

The orchestrator exposes runtime operations including resize, add_device, fs_device_update, and shutdown, maintaining consistent VmConfig state throughout the micro-VM lifecycle.

Dynamic Resource Hot-Plug

CubeHypervisor supports live resizing of CPU and memory resources without micro-VM restart. The Vm::resize method coordinates these changes through the device and memory managers:

CPU hot-plug forwards requests to cpu::CpuManager::resize. When the vCPU count exceeds the boot configuration, the device manager triggers ACPI notifications via AcpiNotificationFlags::CPU_DEVICES_CHANGED.

Memory hot-plug behavior depends on MemoryConfig.hotplug_method. The ACPI path signals changes through AcpiNotificationFlags::MEMORY_DEVICES_CHANGED, while the virtio-mem path bypasses ACPI entirely. MemoryManager::resize returns an optional MemoryRegion that the device manager integrates into the MMIO bus.

All hot-plug events update the persisted VmConfig, ensuring subsequent reboots recreate the micro-VM with current resource allocations.

Snapshot and Migration Support

The micro-VM implements the Migratable trait through vm_migration::protocol, enabling live migration and snapshotting. Vm::new_from_snapshot and Vm::create_hypervisor_vm handle snapshot restoration using hypervisor-specific snapshot IDs (MEMORY_MANAGER_SNAPSHOT_ID, DEVICE_MANAGER_SNAPSHOT_ID). This mechanism rebuilds exact memory layouts, device states, and CPU registers, allowing micro-VMs to migrate across physical hosts while maintaining runtime consistency.

Practical Implementation Examples

Below are minimal Rust snippets demonstrating common CubeHypervisor interactions:

// 1️⃣ Create a KVM hypervisor instance (factory is hidden behind the trait)
let hypervisor = hypervisor::new_hypervisor()?; // hypervisor::new_hypervisor() selects KVM

// 2️⃣ Build a VM configuration from a CLI string
let args = clap::Command::new("cube")
    .arg(clap::Arg::new("cpus").long("cpus").default_value("boot=2,max=4"))
    .arg(clap::Arg::new("memory").long("memory").default_value("size=512M"))
    .arg(clap::Arg::new("net").long("net").default_value("tap=tap0,mac=52:54:00:12:34:56"))
    .get_matches();
let vm_params = hypervisor::vmm::VmParams::from_arg_matches(&args);

// 3️⃣ Instantiate the full VM (memory and devices are allocated)
let vm = hypervisor::vmm::Vm::new(
    Arc::new(Mutex::new(vm_params.into())),
    EventFd::new().unwrap(),
    EventFd::new().unwrap(),
    /* debug event */ EventFd::new().unwrap(),
    &SeccompAction::Allow,
    hypervisor.clone(),
    EventFd::new().unwrap(),
    None,
    None,
    None,
    /* sandbox ID */ "sandbox1".to_string(),
    Arc::new(AtomicBool::new(false)),
)?;

// 4️⃣ Adjust resources at runtime
vm.resize(Some(6), Some(1024 << 20), None)?;   // add CPUs and memory

// 5️⃣ Add a virtio‑blk device after boot
let disk_cfg = hypervisor::vmm::DiskConfig::parse("path=/var/lib/cube/disk.img,readonly=off")?;
let pci_info = vm.add_device(hypervisor::vmm::DeviceConfig::Disk(disk_cfg))?;
println!("Added disk on PCI {}", pci_info.bdf);

// 6️⃣ Graceful shutdown
vm.shutdown()?;

Summary

• CubeHypervisor abstracts KVM behind the Hypervisor and Vm traits in hypervisor/hypervisor/src/hypervisor.rs, enabling backend-agnostic micro-VM management.
• Configuration flows through hypervisor/vmm/src/config.rs, where VmParams validates and converts CLI arguments into type-safe VmConfig structures.
• Orchestration in hypervisor/vmm/src/vm.rs coordinates hypervisor creation, memory layout (MemoryManager), device attachment (DeviceManager), and kernel loading.
• Hot-plug supports dynamic CPU and memory resizing via ACPI or virtio-mem mechanisms, with state persisted to VmConfig.
• Migration implements the Migratable trait using snapshot IDs to reconstruct micro-VM state across hosts.

Frequently Asked Questions

How does CubeHypervisor isolate KVM-specific code from the main VMM logic?

CubeHypervisor defines a generic Hypervisor trait in hypervisor/hypervisor/src/hypervisor.rs that declares standard operations like create_vm and get_cpuid. The KVM-specific implementation in hypervisor/kvm/src/lib.rs provides concrete KvmHypervisor and KvmVm types that handle low-level ioctls, while the rest of the codebase interacts with trait objects (Arc<dyn Vm>). This architecture allows the same VMM logic to run on TDX or other hypervisor backends without modification.

What configuration options does CubeHypervisor support for micro-VMs?

The configuration system in hypervisor/vmm/src/config.rs supports comprehensive micro-VM tuning including CPU topology and features (CpusConfig), memory sizes and hot-plug methods (MemoryConfig), block devices (DiskConfig), network interfaces (NetConfig), and filesystem sharing (FsConfig). Each component provides a parse method that converts comma-separated CLI strings into validated Rust structs, with full support for NUMA topology and huge-page backing.

How does CubeHypervisor handle memory and CPU hot-plug?

The Vm::resize method in hypervisor/vmm/src/vm.rs coordinates hot-plug operations by forwarding CPU requests to CpuManager::resize and memory requests to MemoryManager::resize. CPU additions trigger AcpiNotificationFlags::CPU_DEVICES_CHANGED events, while memory expansions use either ACPI notifications or virtio-mem protocols depending on the MemoryConfig.hotplug_method setting. All changes update the thread-safe VmConfig to ensure persistence across reboots.

Can CubeHypervisor micro-VMs be migrated between hosts?

Yes, CubeHypervisor implements the Migratable trait through vm_migration::protocol. The Vm::new_from_snapshot function reconstructs micro-VMs using hypervisor snapshot IDs (MEMORY_MANAGER_SNAPSHOT_ID, DEVICE_MANAGER_SNAPSHOT_ID) to restore exact memory layouts, device states, and CPU register contents. This enables live migration across physical hosts while maintaining guest runtime consistency, with the KVM backend handling the low-level VM state transfer.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}