ARM64 Support Considerations and Limitations in CubeSandbox v0.5

CubeSandbox v0.5 supports ARM64 (AArch64) processors through its virtual-OS (vO.S) layer running atop Cloud Hypervisor, but requires specific hardware prerequisites like GIC v3 interrupt controllers, uses 64 KiB memory granularity instead of 4 KiB, and remains experimental with known limitations in snapshot/restore stability.

CubeSandbox v0.5 extends platform support to ARM64 architectures, enabling deployment on AArch64 servers and development boards. While this release brings feature parity with x86_64 for core virtualization workloads, several architectural nuances in boot firmware, memory management, and KVM register handling require specific configuration. Understanding these ARM64 support considerations and limitations ensures stable VM deployment and avoids common pitfalls related to hardware compatibility and experimental features.

Hardware Prerequisites and Host Configuration

Deploying CubeSandbox v0.5 on ARM64 requires AArch64 servers or development boards equipped with a GIC v3 interrupt controller. According to hypervisor/docs/arm64.md, boards with limited physical RAM must configure a swap file to prevent OOM (Out of Memory) termination of the cloud-hypervisor process during VM operation. This resource requirement is critical because the hypervisor process consumes significant memory during guest initialization, and ARM64 development boards often ship with less RAM than server-grade x86_64 hardware.

Boot Architecture: UEFI vs Direct-Kernel

CubeSandbox v0.5 supports two distinct boot paths on ARM64, each with different capabilities:

• UEFI boot – Uses EDK2 firmware (CLOUDHV_EFI.fd) to enable ACPI support, allowing stock OS images to boot without custom kernel builds. This method provides the most compatible experience for standard cloud images.
• Direct-kernel boot – Bypasses UEFI firmware entirely, loading the kernel image directly. This approach cannot use ACPI tables, requiring devices to be described via Device Tree or other mechanisms.

The choice between these methods affects peripheral enumeration and power management capabilities within the guest, as documented in hypervisor/docs/arm64.md.

Memory and CPU Architecture

Memory Granularity Differences

Unlike x86_64's 4 KiB pages, the ARM64 implementation in hypervisor/vmm/src/memory_manager.rs uses 64 KiB memory granularity. This architectural difference influences memory allocation policies and can affect VM sizing decisions. When planning memory layouts for ARM64 guests, administrators must account for this larger minimum allocation unit to avoid inefficient memory utilization.

CPU Topology and NUMA Support

Recent enhancements in CubeSandbox v0.5 enable ARM64 VMs to utilize more than 16 vCPUs, expanding support for high-performance computing workloads. Additionally, NUMA (Non-Uniform Memory Access) support is implemented via ACPI tables, allowing multi-node guest layouts that map to host NUMA topology. These capabilities are tracked in hypervisor/release-notes.md and represent significant progress toward production-ready ARM64 virtualization.

Device Support and Virtualization Features

PCI Device Passthrough

CubeSandbox v0.5 introduces PCI device exposure to ARM64 guests, enabling hardware passthrough scenarios such as GPU acceleration. This feature expands the range of supported workloads beyond CPU-bound applications, though specific device compatibility depends on host kernel VFIO support and proper IOMMU configuration.

Initramfs and Early Boot Customization

The ARM64 build now accepts an initramfs image, simplifying kernel-level boot sequences and enabling early-boot customization without modifying the root filesystem. This capability, noted in hypervisor/release-notes.md, facilitates debugging and specialized initialization scenarios commonly required in embedded ARM64 deployments.

Stability Limitations and Experimental Status

Snapshot and Restore Constraints

While snapshot and restore functionality works on ARM64 in v0.5, it remains a newer feature compared to the mature x86_64 implementation. According to hypervisor/release-notes.md, edge-case bugs may manifest during state capture or restoration, particularly with complex device configurations or high vCPU counts. Testing snapshot workflows thoroughly before production deployment is recommended.

Experimental Classification

ARM64 support in CubeSandbox v0.5 retains experimental status. While core features—including VM creation, networking, and storage—demonstrate stability, certain corner cases involving exotic ACPI tables or specialized CPU features may lack full validation. The hypervisor/docs/arm64.md file explicitly warns that unexpected behavior could occur with non-standard hardware configurations or advanced virtualization extensions.

Kernel Configuration and KVM Integration

Successful ARM64 deployment requires using the dedicated kernel configuration file located at hypervisor/resources/linux-config-aarch64. This configuration enables mandatory ARM64-specific options including:

• CONFIG_ARM64
• CONFIG_ARM64_VHE (Virtualization Host Extensions)
• CONFIG_ARM64_PAN (Privileged Access Never)

Using an incorrect kernel configuration results in boot failures or undefined behavior during hypervisor initialization.

At the KVM level, the hypervisor implements ARM-specific register definitions in hypervisor/hypervisor/src/kvm/mod.rs, including KVM_REG_ARM64, KVM_REG_ARM64_SYSREG, and accessors for system registers like MPIDR_EL1 (Multiprocessor Affinity Register). Misusing these registers through direct KVM manipulation can lead to undefined behavior or guest crashes.

Deployment Examples

The following workflow demonstrates building and launching CubeSandbox v0.5 on ARM64, incorporating the considerations above.

1. Prepare the Build Environment

export CLOUDH=$HOME/cloud-hypervisor
mkdir -p $CLOUDH

2. Install Prerequisites

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
sudo apt-get update
sudo apt-get install -y git build-essential m4 bison flex uuid-dev qemu-utils

3. Clone and Build Cloud Hypervisor

pushd $CLOUDH
git clone https://github.com/cloud-hypervisor/cloud-hypervisor.git
cd cloud-hypervisor
cargo build
popd

4. Acquire an ARM64 Guest Image

pushd $CLOUDH
wget https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-arm64.img
qemu-img convert -f qcow2 -O raw focal-server-cloudimg-arm64.img focal-server-cloudimg-arm64.raw
popd

5. Build UEFI Firmware (Recommended for ACPI Support)

pushd $CLOUDH
git clone --depth 1 https://github.com/tianocore/edk2.git -b master
git clone --depth 1 https://github.com/tianocore/edk2-platforms.git -b master
git clone --depth 1 https://github.com/acpica/acpica.git -b master
export PACKAGES_PATH="$PWD/edk2:$PWD/edk2-platforms"
export IASL_PREFIX="$PWD/acpica/generate/unix/bin/"
make -C acpica
cd edk2 && . edksetup.sh && cd ..
make -C edk2/BaseTools
build -a AARCH64 -t GCC5 -p ArmVirtPkg/ArmVirtCloudHv.dsc -b RELEASE
popd

The resulting firmware is located at edk2/Build/ArmVirtCloudHv-AARCH64/RELEASE_GCC5/FV/CLOUDHV_EFI.fd.

6. Launch VM with UEFI Boot

sudo RUST_BACKTRACE=1 $CLOUDH/cloud-hypervisor/target/debug/cloud-hypervisor \
    --api-socket /tmp/cloud-hypervisor.sock \
    --kernel $CLOUDH/edk2/Build/ArmVirtCloudHv-AARCH64/RELEASE_GCC5/FV/CLOUDHV_EFI.fd \
    --disk path=$CLOUDH/focal-server-cloudimg-arm64.raw \
    --cpus boot=4 \
    --memory size=4096M \
    --net tap=,mac=12:34:56:78:90:01,ip=192.168.1.1,mask=255.255.255.0 \
    --serial tty \
    --console off

7. Launch VM with Direct-Kernel Boot

# Build custom kernel using provided ARM64 config

pushd $CLOUDH
git clone --depth 1 https://github.com/cloud-hypervisor/linux.git -b ch-5.12
cd linux
cp $CLOUDH/cloud-hypervisor/resources/linux-config-aarch64 .config
make -j$(nproc)
popd

# Run without UEFI (no ACPI)

sudo $CLOUDH/cloud-hypervisor/target/debug/cloud-hypervisor \
    --api-socket /tmp/cloud-hypervisor.sock \
    --kernel $CLOUDH/linux/arch/arm64/boot/Image \
    --disk path=$CLOUDH/focal-server-cloudimg-arm64.raw \
    --cmdline "keep_bootcon console=ttyAMA0 reboot=k panic=1 root=/dev/vda1 rw" \
    --cpus boot=4 \
    --memory size=4096M \
    --net tap=,mac=12:34:56:78:90:01,ip=192.168.1.1,mask=255.255.255.0 \
    --serial tty \
    --console off

Summary

• ARM64 support requires GIC v3 interrupt controllers and adequate host RAM or swap space to prevent OOM errors.
• Memory granularity is 64 KiB on ARM64 compared to 4 KiB on x86_64, affecting VM sizing strategies.
• Two boot paths exist: UEFI (with ACPI support) and direct-kernel (without ACPI), each suited for different deployment scenarios.
• vCPU and NUMA capabilities now exceed 16 vCPUs with ACPI-based NUMA support, though the platform remains experimental.
• Snapshot/restore functionality works but may exhibit edge-case bugs compared to the mature x86_64 implementation.
• Kernel configuration must use the provided linux-config-aarch64 file with specific ARM64 virtualization options enabled.
• PCI passthrough and initramfs support expand workload possibilities but require careful hardware compatibility verification.

Frequently Asked Questions

What hardware is required to run CubeSandbox v0.5 on ARM64?

CubeSandbox v0.5 requires AArch64 servers or development boards featuring a GIC v3 interrupt controller. Development boards with limited RAM must configure sufficient swap space to prevent the cloud-hypervisor process from being terminated by the OOM killer during VM initialization, as specified in hypervisor/docs/arm64.md.

Can I use standard Ubuntu cloud images on ARM64 with CubeSandbox?

Yes, but only when using UEFI boot mode with EDK2 firmware. UEFI enables ACPI support, allowing stock ARM64 cloud images (such as Ubuntu Focal) to boot without modification. Direct-kernel boot requires custom kernel builds and cannot utilize ACPI tables, potentially causing compatibility issues with standard cloud images expecting ACPI-based device enumeration.

Why does ARM64 require different memory granularity settings than x86_64?

The ARM64 architecture in CubeSandbox uses 64 KiB memory pages versus 4 KiB on x86_64, as implemented in hypervisor/vmm/src/memory_manager.rs. This architectural difference affects memory allocation policies and VM sizing calculations, requiring administrators to plan memory layouts differently to avoid inefficient allocation or alignment issues during guest creation.

Is snapshot/restore production-ready for ARM64 deployments?

Snapshot and restore functionality is functional but experimental on ARM64 in v0.5. While basic operations succeed, the feature is newer than its x86_64 counterpart and may exhibit edge-case bugs, particularly with complex device configurations. According to hypervisor/release-notes.md, thorough testing is recommended before relying on this feature for production workloads.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}