CubeSandbox ARM64 Native Support: architectural Details and Build Requirements

CubeSandbox 0.5.0 delivers full-stack native support for ARM64 (aarch64) hosts by migrating from x86-specific PIO to MMIO for device signaling, updating seccomp filters for ARM64 syscall numbers, and providing architecture-aware build scripts that enable native compilation on ARM64 machines.

TencentCloud/CubeSandbox is a lightweight secure container runtime that uses hardware virtualization to provide strong isolation. Version 0.5.0 introduces comprehensive ARM64 native support, requiring architectural changes across the hypervisor layer, guest agent, and build toolchain to accommodate differences in CPU instruction sets, device access patterns, and system calls.

Hypervisor Architecture Changes for ARM64

The hypervisor layer in hypervisor/vmm/src/ contains architecture-specific code gated by #[cfg(target_arch = "aarch64")] conditionals. These changes ensure proper hardware virtualization on ARM64 hosts.

MMIO-Based SysCtrl Device

On x86_64, the SysCtrl device uses PIO port 0x680 for guest-to-host signaling. For ARM64, this is replaced by a memory-mapped MMIO region starting at LEGACY_SYS_CTRL_MAPPED_IO_START (physical address 0x0903_0000). The device registration moves from the I/O bus to the MMIO bus in hypervisor/vmm/src/device_manager.rs (lines 872-877).

KVM Register Access and CPU Initialization

The hypervisor updates KVM register access to use get_one_reg and set_one_reg APIs compatible with ARM64's system register model. CPU initialization in hypervisor/vmm/src/vm.rs (lines 50-55) handles ARM64-specific features including the Generic Interrupt Controller (GIC) and Performance Monitoring Unit (PMU). If the host lacks PMUv3, the system gracefully falls back to PMU-less initialization (lines 1229-1241).

Seccomp Filter Adaptations

ARM64 uses different syscall numbers than x86_64. The seccomp filters in hypervisor/vmm/src/seccomp_filters.rs (lines 399-404) replace SYS_lstat with SYS_fstatat/SYS_newfstatat and add SYS_faccessat2 to match ARM64's syscall table.

Serial Console and PL011 UART

ARM64 guests use the PL011 UART rather than the virtio console used on x86. The serial manager in hypervisor/vmm/src/serial_manager.rs (lines 8-12) configures console=ttyAMA0,115200 for ARM64 targets, while the kernel configuration file hypervisor/resources/linux-config-aarch64 enables CONFIG_SERIAL_AMBA_PL011 and other ARM64-specific drivers.

Guest Agent and Shim Modifications

The guest agent and CubeShim components require architecture-specific adjustments for device communication and kernel parameters.

MMIO Signaling in Guest Agent

The guest agent writes readiness signals to the MMIO region at 0x0903_0000 via /dev/mem instead of using ioperm() and PIO. This logic is implemented in CubeAPI/src/services/sandboxes.rs.

Architecture-Aware Command Line Generation

CubeShim in Cubelet/src/main.rs generates kernel command lines conditionally: ARM64 builds use console=ttyAMA0,115200 while x86_64 uses console=hvc0. Architecture-specific flags like no_timer_check are gated with #[cfg(target_arch = "x86_64")] to prevent inclusion on ARM64.

Build Requirements and Toolchain

Building CubeSandbox for ARM64 requires specific toolchain configurations and supports both native compilation on ARM64 hosts and cross-compilation from x86_64.

Native vs Cross-Compilation

The builder image defined in docker/Dockerfile.builder installs the ARM64 cross-compiler (aarch64-linux-gnu-gcc) and Clang ≥14. When the host architecture matches the target (aarch64 on an ARM64 machine), the build system automatically performs native compilation without cross-compilation overhead.

Kernel Build Configuration

The scripts/build-kernel.sh script determines the target architecture from KERNEL_TARGET_ARCH (defaulting to the host). For ARM64 builds, it sets ARCH=arm64 and uses CROSS_COMPILE=aarch64-linux-gnu- only when cross-compiling. The script builds the Image target and renames it to vmlinux for downstream consumers.

Build the guest kernel natively on ARM64:

export KERNEL_SRC_DIR=$HOME/linux-5.15
export KERNEL_CONFIG=$HOME/linux-5.15/arch/arm64/configs/defconfig
export KERNEL_OUTPUT_DIR=$HOME/kernel-build
export KERNEL_TARGET_ARCH=aarch64

./scripts/build-kernel.sh

Build the complete CubeSandbox stack:

make builder
make build

The Makefile detects the host architecture and executes cargo builds with --target aarch64-unknown-linux-gnu when appropriate.

Deployment on ARM64 Hosts

ARM64 deployment requires bare-metal hardware with native KVM support.

Bare-Metal Requirements

Unlike x86_64, ARM64 hosts cannot use the nested-KVM PVM layer and must run on bare-metal servers with /dev/kvm accessible. The docs/guide/bare-metal-deploy.md documentation specifies that ARM64 hosts require physical servers with native KVM support.

Self-Build Deployment

For native builds on ARM64, set TARGET_ARCH=aarch64 or let the build scripts auto-detect the host architecture. The resulting tarball (e.g., cube-sandbox-one-click-<version>-arm64.tar.gz) installs via the standard install.sh script.

Deploy a self-built bundle:

scp cube-sandbox-one-click-*-arm64.tar.gz root@arm64-host:/tmp
ssh root@arm64-host "cd /opt && tar -xzf /tmp/cube-sandbox-one-click-*-arm64.tar.gz && cd cube-sandbox-oneclick-* && sudo ./install.sh"

Known Limitations

Several features remain x86_64-only:

• PVM Support: The nested-KVM PVM layer runs only on x86_64 kernels.
• Live Migration: Currently only available for x86_64 guests.
• Snapshot Granularity: On ARM64 hosts with 64KB pages, dirty-bitmaps respect the host page size (fixed in v0.5.1).

Summary

• CubeSandbox 0.5.0 provides full native ARM64 support by replacing x86 PIO with MMIO at 0x0903_0000 for the SysCtrl device.
• The hypervisor uses #[cfg(target_arch = "aarch64")] blocks in hypervisor/vmm/src/vm.rs to handle ARM64-specific initialization, PL011 UART, and seccomp filters.
• Build scripts in scripts/build-kernel.sh automatically detect ARM64 hosts and perform native compilation when KERNEL_TARGET_ARCH=aarch64 matches the host.
• ARM64 deployments require bare-metal hardware with native KVM; PVM and live migration are not supported on ARM64.

Frequently Asked Questions

Does CubeSandbox support running ARM64 guests on x86_64 hosts?

No, CubeSandbox requires the host architecture to match the target architecture. ARM64 guests must run on ARM64 hosts with native KVM support; the PVM nested virtualization layer is x86_64 only according to the source code in docs/guide/bare-metal-deploy.md.

What kernel configuration is required for ARM64 builds?

Use the provided hypervisor/resources/linux-config-aarch64 which enables CONFIG_KVM, CONFIG_SERIAL_AMBA_PL011 for the PL011 UART, and ARM64-specific CPU features like LSE and SVE. This configuration ensures the guest kernel can communicate via the MMIO-based SysCtrl device and PL011 serial console.

Can I cross-compile CubeSandbox for ARM64 on an x86_64 machine?

Yes, the build system supports cross-compilation. Set KERNEL_TARGET_ARCH=aarch64 and ensure the aarch64-linux-gnu-gcc toolchain is installed. The docker/Dockerfile.builder includes the necessary cross-compiler, and scripts/build-kernel.sh automatically sets CROSS_COMPILE=aarch64-linux-gnu- when the host architecture differs from the target.

Why does ARM64 use ttyAMA0 instead of hvc0 for the console?

ARM64 guests utilize the PL011 UART device mapped to ttyAMA0 rather than the virtio console (hvc0) used on x86_64. This is configured in the kernel command line generated by CubeShim in Cubelet/src/main.rs and the device manager in hypervisor/vmm/src/serial_manager.rs (lines 8-12), reflecting the different serial hardware available on ARM64 platforms.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}