How CubeSandbox Supports Live Migration of VM States Between Hosts

CubeSandbox enables live migration of VM states through a coordinated architecture spanning the CubeMaster control plane, the hypervisor's migration state machine in hypervisor/vmm/src/migration.rs, and incremental memory transfer using soft-dirty page tracking to minimize downtime.

TencentCloud/CubeSandbox implements live migration of VM states by orchestrating snapshot transfer, memory synchronization, and lifecycle management across distributed hosts. The system combines a migration coordinator in the CubeMaster service with a hypervisor-level state machine that handles low-level VM pausing, memory copying, and device state synchronization. This architecture ensures minimal service interruption when moving running sandboxes between physical hosts.

Migration Control Plane Architecture

The CubeMaster service acts as the central orchestrator for live migration of VM states. When a migration is initiated, CubeMaster receives the request via its HTTP/GRPC API at the /sandboxes/:id/migrate endpoint, as defined in CubeMaster/api/services/sandbox/v1/sandbox.proto. The control plane records the migration intent in the central database using the job DTO structures found in CubeMaster/pkg/templatecenter/job_dto.go, and updates the sandbox's lifecycle state to Migrating. Throughout the process, the cube-lifecycle-manager (referenced in cube-lifecycle-manager/internal/discovery/discovery.go) monitors the operation and updates the sandbox's endpoint to point to the new host upon completion.

Hypervisor State Machine and Memory Transfer

At the hypervisor layer, the VMM (Virtual Machine Monitor) component executes the migration protocol implemented in hypervisor/vmm/src/migration.rs. This module defines a MigrationState enum that transitions through Init, PreCopy, StopAndCopy, Resume, and Done states. The core run_migration function drives the state machine, coordinating with the destination host via a TCP-based migration channel to stream the VM snapshot and memory pages.

State Capture and Snapshotting

During the initial phase, the hypervisor pauses the guest VM only long enough to quiesce device I/O, then collects a complete VM snapshot comprising CPU registers, device configuration, and block-device state. This snapshot is immediately streamed to the destination host to establish the base state for the new VM instance.

Incremental Memory Synchronization

Following the initial snapshot, the system enters the pre-copy phase. The hypervisor leverages soft-dirty bitmap tracking implemented in hypervisor/vmm/src/soft_dirty.rs to identify modified memory pages since the last transfer. The state machine iteratively sends these dirty pages to the destination while the source VM continues running, using the following logic:

1. Collect dirty pages using the soft-dirty bitmap.
2. Transmit the pages over the migration channel.
3. Clear the dirty flags and repeat until the working set stabilizes.

When the host supports it, CubeSandbox utilizes virtio-mem for zero-copy ballooning optimizations, as documented in the live migration enhancements section of hypervisor/release-notes.md. This incremental approach significantly reduces the amount of data that must transfer during the final stop-and-copy phase.

Destination Coordination and Final Switchover

The destination host prepares by creating a new VM instance with an identical configuration and pre-allocating the required memory. As the source streams memory pages, the destination applies incremental updates in real-time. Once the final memory page transfers and the destination confirms receipt, the source VM stops, and the target VM resumes execution from the exact point of suspension. CubeMaster then finalizes the migration by updating the sandbox's endpoint and marking the state as Running again.

Platform Constraints and Limitations

Live migration of VM states in CubeSandbox currently supports x86_64 architectures only; ARM64 hosts are restricted to cold migration, as noted in hypervisor/README.md. Additionally, migration across different major versions is explicitly disallowed to prevent schema and ABI mismatches between hypervisor instances. The implementation is designed to be idempotent—if a migration fails at any point, the source VM remains running and the operation can be safely retried without state corruption.

Code Implementation Examples

Triggering Migration via the Go SDK

The CubeSandbox SDK abstracts the underlying API calls. The following example demonstrates initiating a migration using the Go SDK:

import "github.com/TencentCloud/CubeSandbox/sdk/go"

func migrateSandbox(sandboxID, destHost string) error {
    client := go.NewClient("https://api.cubesandbox.example")
    // The SDK exposes a MigrationRequest struct that maps to the
    // CubeMaster /sandboxes/:id/migrate endpoint.
    req := &go.MigrationRequest{
        DestinationHost: destHost,
        // optional: timeout, keep‑alive settings, etc.
    }
    _, err := client.Sandboxes.Migrate(sandboxID, req)
    return err
}

Core Migration Logic in Rust

The internal state machine handles the complex coordination between source and destination. The following excerpt from hypervisor/vmm/src/migration.rs illustrates the migration flow:

// hypervisor/vmm/src/migration.rs (excerpt)
enum MigrationState {
    Init,
    PreCopy,
    StopAndCopy,
    Resume,
    Done,
}

fn run_migration(vm: &mut Vm, dst: &mut MigrationSink) -> Result<()> {
    // 1. Pause VM and snapshot registers
    vm.pause()?;
    let snap = vm.take_snapshot()?;
    dst.send_snapshot(snap)?;

    // 2. Pre‑copy dirty pages
    while vm.has_dirty_pages() {
        let pages = vm.collect_dirty_pages();
        dst.send_memory(pages)?;
        vm.clear_dirty();
    }

    // 3. Final stop‑and‑copy
    vm.stop()?;
    let final_pages = vm.collect_all_pages();
    dst.send_memory(final_pages)?;

    // 4. Resume on destination
    dst.finalize()?;
    Ok(())
}

Summary

• CubeSandbox coordinates live migration of VM states through CubeMaster's control plane API and the hypervisor's state machine in hypervisor/vmm/src/migration.rs.
• The system uses soft-dirty page tracking in hypervisor/vmm/src/soft_dirty.rs to enable efficient incremental memory transfer during the pre-copy phase, reducing final switchover time.
• Migration is currently x86_64-only and explicitly disallowed across different major versions to ensure ABI compatibility.
• The architecture is idempotent, ensuring source VMs remain running if migration fails, allowing for safe retry operations without service interruption.

Frequently Asked Questions

Does CubeSandbox support live migration on ARM64 architectures?

No, live migration of VM states is currently restricted to x86_64 hosts. According to hypervisor/README.md, ARM64 hosts can only perform cold migration, which requires stopping the VM before transferring state.

What happens if a live migration fails mid-transfer?

The migration implementation is idempotent. If the process fails at any stage, the source VM remains running and the target state is discarded. Operators can safely retry the migration without risking state corruption or service interruption on the source host.

Can I migrate VMs between different CubeSandbox versions?

No, migration across different major versions is explicitly disallowed. The hypervisor/README.md documentation notes that live migration is not supported across different versions to avoid schema and ABI mismatches between the source and destination hypervisors.

How does the hypervisor minimize downtime during the final switchover?

The hypervisor minimizes downtime by using an iterative pre-copy algorithm. It transfers the majority of memory pages while the VM runs, then only pauses the VM briefly during the StopAndCopy phase to transfer the final dirty pages and CPU state. This final pause typically lasts only milliseconds, as implemented in hypervisor/vmm/src/migration.rs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}