# CubeHypervisor KVM MicroVM Configuration and Management Internals

> Explore CubeHypervisor KVM microVM configuration and management internals. Learn how CubeSandbox uses a hypervisor-agnostic architecture for secure, lightweight micro virtual machines.

- Repository: [Tencent Cloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox)
- Tags: internals
- Published: 2026-07-03

---

**CubeSandbox implements a hypervisor-agnostic architecture that isolates KVM-specific logic behind Rust traits, using a three-tier system of generic hypervisor abstractions, strongly-typed configuration parsers, and orchestrated VM lifecycle management to enable secure, lightweight micro-VMs.**

The TencentCloud CubeSandbox project provides a secure container runtime built on KVM-based micro-virtualization. At its core, the CubeHypervisor component abstracts KVM internals through a clean trait-based interface, allowing the same configuration and management logic to target multiple hypervisor backends including standard KVM and Intel TDX. This design decouples the user-facing configuration API from the underlying ioctl-heavy KVM implementation.

## Hypervisor-Agnostic Core Architecture

CubeHypervisor defines a generic hypervisor interface in [`hypervisor/hypervisor/src/hypervisor.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/hypervisor/src/hypervisor.rs) that masks platform-specific details behind a set of Rust traits. The `Hypervisor` trait declares the fundamental operations required for any backend implementation:

```rust
pub trait Hypervisor: Send + Sync {
    fn hypervisor_type(&self) -> HypervisorType;
    fn create_vm(&self) -> Result<Arc<dyn Vm>>;
    fn create_vm_with_type(&self, _vm_type: u64) -> Result<Arc<dyn Vm>> { unreachable!() }
    #[cfg(target_arch = "x86_64")]
    fn get_cpuid(&self) -> Result<Vec<CpuIdEntry>>;
    // … other optional helpers …
}

```

The concrete KVM implementation resides in [`hypervisor/kvm/src/lib.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/kvm/src/lib.rs), where `KvmHypervisor` implements this trait. When the orchestrator calls `create_vm`, the KVM backend returns a `KvmVm` instance that handles the low-level ioctl plumbing, including identity map setup (`set_identity_map_address`), TSS address configuration (`set_tss_address`), and split IRQ enablement (`enable_split_irq`). This abstraction allows the rest of the codebase to operate on generic `Vm` objects without direct KVM dependencies.

## VM Configuration Pipeline

All micro-VM parameters flow through a strongly-typed configuration system centered in [`hypervisor/vmm/src/config.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/vmm/src/config.rs). The `VmParams::from_arg_matches` function parses command-line arguments using the `option_parser` crate, converting strings like `--cpus boot=2,max=4` into structured Rust types:

- **CPU configuration** – `CpusConfig` parses boot and maximum vCPU counts, topology, and CPU features, validating against errors like `ValidationError::CpuTopologyCount`.
- **Memory configuration** – `MemoryConfig` handles base size, huge-pages, hot-plug methods (`Acpi` vs `VirtioMem`), and per-NUMA zone definitions.
- **Device configuration** – `DiskConfig`, `NetConfig`, and `FsConfig` provide `parse` methods that transform comma-separated strings into typed structs with validation.

The resulting `VmConfig` is stored in a thread-safe `Arc<Mutex<_>>` and consumed by the orchestrator during VM instantiation.

## MicroVM Lifecycle Orchestration

The `Vm::new` function in [`hypervisor/vmm/src/vm.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/vmm/src/vm.rs) serves as the entry point for micro-VM creation, executing a six-stage initialization sequence:

1. **Hypervisor VM creation** – Calls `Self::create_hypervisor_vm` to instantiate the underlying KVM VM via `hypervisor.create_vm()` or `create_vm_with_type` for TDX-specific configurations.
2. **Physical address space calculation** – Computes guest-visible physical address width via `physical_bits`, clipping to host limits and handling special cases like KVM-PVM (max 43 bits).
3. **Memory manager initialization** – `MemoryManager::new` allocates guest RAM, shared memory regions, and optional SGX EPC pages based on `VmConfig.memory` and the calculated physical bits.
4. **Device manager setup** – `DeviceManager::new` constructs virtual PCI, MMIO, and I/O buses, registers virtio devices (network, block, filesystem, vsock, balloon, RNG), and initializes console/PTTY infrastructure.
5. **Kernel loading** – `Vm::load_payload` handles ELF/PE loading via `linux_loader`, supporting kernel-only, firmware-only, or combined boot paths, storing the entry point in an `EntryPoint` structure.
6. **System configuration** – `Vm::configure_system` injects ACPI tables, initramfs, kernel command-line, and metadata (serial number, UUID, OEM strings) through architecture-specific `arch::configure_system` calls.

The orchestrator exposes runtime operations including `resize`, `add_device`, `fs_device_update`, and `shutdown`, maintaining consistent `VmConfig` state throughout the micro-VM lifecycle.

## Dynamic Resource Hot-Plug

CubeHypervisor supports live resizing of CPU and memory resources without micro-VM restart. The `Vm::resize` method coordinates these changes through the device and memory managers:

**CPU hot-plug** forwards requests to `cpu::CpuManager::resize`. When the vCPU count exceeds the boot configuration, the device manager triggers ACPI notifications via `AcpiNotificationFlags::CPU_DEVICES_CHANGED`.

**Memory hot-plug** behavior depends on `MemoryConfig.hotplug_method`. The ACPI path signals changes through `AcpiNotificationFlags::MEMORY_DEVICES_CHANGED`, while the virtio-mem path bypasses ACPI entirely. `MemoryManager::resize` returns an optional `MemoryRegion` that the device manager integrates into the MMIO bus.

All hot-plug events update the persisted `VmConfig`, ensuring subsequent reboots recreate the micro-VM with current resource allocations.

## Snapshot and Migration Support

The micro-VM implements the `Migratable` trait through `vm_migration::protocol`, enabling live migration and snapshotting. `Vm::new_from_snapshot` and `Vm::create_hypervisor_vm` handle snapshot restoration using hypervisor-specific snapshot IDs (`MEMORY_MANAGER_SNAPSHOT_ID`, `DEVICE_MANAGER_SNAPSHOT_ID`). This mechanism rebuilds exact memory layouts, device states, and CPU registers, allowing micro-VMs to migrate across physical hosts while maintaining runtime consistency.

## Practical Implementation Examples

Below are minimal Rust snippets demonstrating common CubeHypervisor interactions:

```rust
// 1️⃣ Create a KVM hypervisor instance (factory is hidden behind the trait)
let hypervisor = hypervisor::new_hypervisor()?; // hypervisor::new_hypervisor() selects KVM

// 2️⃣ Build a VM configuration from a CLI string
let args = clap::Command::new("cube")
    .arg(clap::Arg::new("cpus").long("cpus").default_value("boot=2,max=4"))
    .arg(clap::Arg::new("memory").long("memory").default_value("size=512M"))
    .arg(clap::Arg::new("net").long("net").default_value("tap=tap0,mac=52:54:00:12:34:56"))
    .get_matches();
let vm_params = hypervisor::vmm::VmParams::from_arg_matches(&args);

// 3️⃣ Instantiate the full VM (memory and devices are allocated)
let vm = hypervisor::vmm::Vm::new(
    Arc::new(Mutex::new(vm_params.into())),
    EventFd::new().unwrap(),
    EventFd::new().unwrap(),
    /* debug event */ EventFd::new().unwrap(),
    &SeccompAction::Allow,
    hypervisor.clone(),
    EventFd::new().unwrap(),
    None,
    None,
    None,
    /* sandbox ID */ "sandbox1".to_string(),
    Arc::new(AtomicBool::new(false)),
)?;

// 4️⃣ Adjust resources at runtime
vm.resize(Some(6), Some(1024 << 20), None)?;   // add CPUs and memory

// 5️⃣ Add a virtio‑blk device after boot
let disk_cfg = hypervisor::vmm::DiskConfig::parse("path=/var/lib/cube/disk.img,readonly=off")?;
let pci_info = vm.add_device(hypervisor::vmm::DeviceConfig::Disk(disk_cfg))?;
println!("Added disk on PCI {}", pci_info.bdf);

// 6️⃣ Graceful shutdown
vm.shutdown()?;

```

## Summary

- **CubeHypervisor** abstracts KVM behind the `Hypervisor` and `Vm` traits in [`hypervisor/hypervisor/src/hypervisor.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/hypervisor/src/hypervisor.rs), enabling backend-agnostic micro-VM management.
- **Configuration** flows through [`hypervisor/vmm/src/config.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/vmm/src/config.rs), where `VmParams` validates and converts CLI arguments into type-safe `VmConfig` structures.
- **Orchestration** in [`hypervisor/vmm/src/vm.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/vmm/src/vm.rs) coordinates hypervisor creation, memory layout (`MemoryManager`), device attachment (`DeviceManager`), and kernel loading.
- **Hot-plug** supports dynamic CPU and memory resizing via ACPI or virtio-mem mechanisms, with state persisted to `VmConfig`.
- **Migration** implements the `Migratable` trait using snapshot IDs to reconstruct micro-VM state across hosts.

## Frequently Asked Questions

### How does CubeHypervisor isolate KVM-specific code from the main VMM logic?

CubeHypervisor defines a generic `Hypervisor` trait in [`hypervisor/hypervisor/src/hypervisor.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/hypervisor/src/hypervisor.rs) that declares standard operations like `create_vm` and `get_cpuid`. The KVM-specific implementation in [`hypervisor/kvm/src/lib.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/kvm/src/lib.rs) provides concrete `KvmHypervisor` and `KvmVm` types that handle low-level ioctls, while the rest of the codebase interacts with trait objects (`Arc<dyn Vm>`). This architecture allows the same VMM logic to run on TDX or other hypervisor backends without modification.

### What configuration options does CubeHypervisor support for micro-VMs?

The configuration system in [`hypervisor/vmm/src/config.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/vmm/src/config.rs) supports comprehensive micro-VM tuning including CPU topology and features (`CpusConfig`), memory sizes and hot-plug methods (`MemoryConfig`), block devices (`DiskConfig`), network interfaces (`NetConfig`), and filesystem sharing (`FsConfig`). Each component provides a `parse` method that converts comma-separated CLI strings into validated Rust structs, with full support for NUMA topology and huge-page backing.

### How does CubeHypervisor handle memory and CPU hot-plug?

The `Vm::resize` method in [`hypervisor/vmm/src/vm.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/vmm/src/vm.rs) coordinates hot-plug operations by forwarding CPU requests to `CpuManager::resize` and memory requests to `MemoryManager::resize`. CPU additions trigger `AcpiNotificationFlags::CPU_DEVICES_CHANGED` events, while memory expansions use either ACPI notifications or virtio-mem protocols depending on the `MemoryConfig.hotplug_method` setting. All changes update the thread-safe `VmConfig` to ensure persistence across reboots.

### Can CubeHypervisor micro-VMs be migrated between hosts?

Yes, CubeHypervisor implements the `Migratable` trait through `vm_migration::protocol`. The `Vm::new_from_snapshot` function reconstructs micro-VMs using hypervisor snapshot IDs (`MEMORY_MANAGER_SNAPSHOT_ID`, `DEVICE_MANAGER_SNAPSHOT_ID`) to restore exact memory layouts, device states, and CPU register contents. This enables live migration across physical hosts while maintaining guest runtime consistency, with the KVM backend handling the low-level VM state transfer.