How CubeHypervisor Manages KVM MicroVMs for Hardware-Level Isolation

CubeHypervisor creates and orchestrates tiny MicroVMs using Linux KVM to provide hardware-level isolation through CPU virtualization extensions while maintaining minimal resource overhead.

CubeHypervisor is a lightweight, KVM-based hypervisor developed by TencentCloud as part of the CubeSandbox project. It enables CubeSandbox to run untrusted workloads in isolated MicroVMs that leverage hardware virtualization extensions (VT-x/AMD-V) for security. The hypervisor exposes a minimal Go API that integrates with CubeSandbox's Cubelet component to manage the complete lifecycle of these virtual machines.

Architecture and KVM Integration

CubeHypervisor operates as a thin user-space layer atop the Linux KVM driver. According to the TencentCloud/cube-hypervisor source code, the architecture centers on direct interaction with /dev/kvm to create virtual machines with minimal attack surface.

The hypervisor initializes by opening the KVM device, then creates VM instances using the KVM_CREATE_VM ioctl. Each MicroVM typically allocates approximately 10 MiB of memory and runs a stripped-down Linux kernel with a tiny user-space runtime. This minimal footprint reduces startup time and resource consumption while maintaining strict isolation boundaries enforced by the CPU's hardware virtualization extensions.

MicroVM Lifecycle Management

CubeHypervisor manages the complete lifecycle of MicroVMs through five distinct phases implemented across hypervisor.go, vm.go, and snapshot.go.

VM Creation and Configuration

In hypervisor.go, the hypervisor exposes the New() function to initialize the KVM context and NewVM() to create individual virtual machines. The creation process involves:

• Opening /dev/kvm and establishing the KVM context
• Invoking KVM_CREATE_VM to instantiate the VM
• Configuring the memory layout using KVM_SET_MEMORY_REGION
• Setting up virtio devices for I/O virtualization

The VMConfig struct allows specification of memory size (typically 16 MiB or less) and kernel image paths, enabling rapid provisioning of isolated execution environments.

Boot Process

The vm.go file handles the VM abstraction, including vCPU setup and device initialization. When vm.Run() is called, CubeHypervisor loads a minimal kernel image into the allocated memory and invokes KVM_RUN to start execution. The VM boots quickly because the kernel and initramfs are stripped of unnecessary drivers and modules, resulting in a tiny memory footprint that reduces the attack surface.

Snapshot and Restore Operations

snapshot.go implements incremental memory snapshots using KVM_GET_MEMORY_REGION and dirty page tracking. During operation, the hypervisor can:

• Capture the VM's memory state by reading guest physical pages
• Track dirty pages incrementally to minimize snapshot size
• Write atomic snapshots to disk for checkpoint/restore functionality

To resume, the snapshot file is mmap-ed back into the guest physical address space, and the VM resumes execution via KVM_RUN. This enables fast pause/resume cycles for debugging, migration, or resource management.

Teardown

When a sandbox completes execution, CubeHypervisor halts the VM and releases all resources, including memory mappings and file descriptors, ensuring clean isolation boundary termination.

Hardware-Level Isolation Mechanisms

CubeHypervisor leverages hardware virtualization extensions (Intel VT-x or AMD-V) to enforce strict isolation between MicroVMs and the host system. Each MicroVM operates in its own virtual address space protected by the CPU's memory management unit virtualization features.

The isolation guarantees include:

• Memory Isolation: Each VM runs in a separate address space preventing memory leaks or unauthorized access between VMs
• Minimal Attack Surface: The hypervisor only exposes essential KVM ioctls required for VM lifecycle management, reducing the code surface vulnerable to exploitation
• Snapshot Integrity: Snapshots are written atomically and can be cryptographically signed by higher-level CubeSandbox components to detect tampering

Integration with CubeSandbox

CubeHypervisor exposes a public Go API in api.go specifically designed for CubeSandbox's Cubelet component. This API enables:

• Launching sandboxes as MicroVMs via the hypervisor interface
• Pausing sandboxes by triggering snapshot operations followed by VM shutdown
• Resuming sandboxes by restoring snapshots and restarting VMs
• Performing incremental snapshots for rapid checkpoint/restore during debugging or migration

The tight coupling between Cubelet and CubeHypervisor allows CubeSandbox to provide a container-like developer experience while maintaining hardware-level isolation guarantees.

Practical Implementation Example

The following Go code demonstrates how CubeHypervisor creates and runs a MicroVM using the public API:

package main

import (
    "github.com/TencentCloud/cube-hypervisor"
    "log"
)

func main() {
    // Initialise the hypervisor (opens /dev/kvm)
    h, err := hypervisor.New()
    if err != nil {
        log.Fatalf("hypervisor init: %v", err)
    }
    defer h.Close()

    // Create a MicroVM with 16 MiB RAM and a tiny kernel image
    vm, err := h.NewVM(hypervisor.VMConfig{
        MemorySize: 16 << 20, 
        KernelPath: "./vmlinuz",
    })
    if err != nil {
        log.Fatalf("create vm: %v", err)
    }
    defer vm.Close()

    // Boot the VM – blocks until the VM exits or is paused
    if err := vm.Run(); err != nil {
        log.Fatalf("run vm: %v", err)
    }
}

This example initializes the KVM context, creates a 16 MiB MicroVM with a minimal kernel, and boots the VM using the Run() method.

Summary

• CubeHypervisor is a lightweight KVM-based hypervisor that creates MicroVMs with ~10 MiB memory footprints for secure, isolated execution.
• The hypervisor manages VM lifecycle through hypervisor.go (creation), vm.go (boot and vCPU management), and snapshot.go (checkpoint/restore).
• Hardware-level isolation is achieved via CPU virtualization extensions (VT-x/AMD-V) and the KVM driver, with each VM operating in an isolated address space.
• Incremental snapshots using dirty page tracking enable fast pause/resume operations for debugging and migration scenarios.
• The api.go file provides a Go API consumed by CubeSandbox's Cubelet component, bridging high-level sandbox management with low-level KVM virtualization.

Frequently Asked Questions

How does CubeHypervisor differ from traditional hypervisors like QEMU?

Unlike QEMU, which provides full device emulation and supports a wide range of hardware configurations, CubeHypervisor is purpose-built for minimal MicroVMs. It strips away unnecessary emulation layers, uses direct KVM ioctls in hypervisor.go for VM management, and focuses on fast startup times (~10 MiB memory footprints) rather than hardware compatibility. This minimal approach reduces the attack surface and resource overhead while maintaining hardware-level isolation.

What files handle MicroVM snapshot and restore functionality?

The snapshot.go file in the TencentCloud/cube-hypervisor repository implements memory snapshot and incremental diff handling. It uses KVM_GET_MEMORY_REGION to capture VM memory states and tracks dirty pages to create efficient incremental snapshots. The restore process memory-maps saved pages back into the guest physical address space before resuming execution with KVM_RUN.

How does CubeHypervisor ensure security between multiple MicroVMs?

CubeHypervisor enforces hardware-level isolation by leveraging the CPU's virtualization extensions (Intel VT-x or AMD-V) through the Linux KVM driver. Each MicroVM runs in its own isolated virtual address space managed by the hardware MMU, preventing memory leaks or unauthorized access between VMs. Additionally, the hypervisor minimizes its attack surface by exposing only essential KVM ioctls required for VM lifecycle operations.

Can CubeHypervisor run standalone without CubeSandbox?

While CubeHypervisor can technically operate independently using its public Go API exposed in api.go, it is designed specifically as a submodule dependency for CubeSandbox. The hypervisor's API is tailored for CubeSandbox's Cubelet component, which manages sandbox lifecycle operations. Standalone usage would require implementing custom orchestration logic similar to what Cubelet provides.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}