# How CubeSandbox Supports Live Migration of VM States Between Hosts

> CubeSandbox enables live VM migration with a coordinated control plane, hypervisor state machine, and incremental memory transfer using soft-dirty page tracking for minimal downtime.

- Repository: [Tencent Cloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox)
- Tags: how-to-guide
- Published: 2026-07-16

---

**CubeSandbox enables live migration of VM states through a coordinated architecture spanning the CubeMaster control plane, the hypervisor's migration state machine in [`hypervisor/vmm/src/migration.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/vmm/src/migration.rs), and incremental memory transfer using soft-dirty page tracking to minimize downtime.**

TencentCloud/CubeSandbox implements live migration of VM states by orchestrating snapshot transfer, memory synchronization, and lifecycle management across distributed hosts. The system combines a **migration coordinator** in the CubeMaster service with a **hypervisor-level state machine** that handles low-level VM pausing, memory copying, and device state synchronization. This architecture ensures minimal service interruption when moving running sandboxes between physical hosts.

## Migration Control Plane Architecture

The **CubeMaster** service acts as the central orchestrator for live migration of VM states. When a migration is initiated, CubeMaster receives the request via its HTTP/GRPC API at the `/sandboxes/:id/migrate` endpoint, as defined in `CubeMaster/api/services/sandbox/v1/sandbox.proto`. The control plane records the migration intent in the central database using the job DTO structures found in [`CubeMaster/pkg/templatecenter/job_dto.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/templatecenter/job_dto.go), and updates the sandbox's lifecycle state to *Migrating*. Throughout the process, the `cube-lifecycle-manager` (referenced in [`cube-lifecycle-manager/internal/discovery/discovery.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/cube-lifecycle-manager/internal/discovery/discovery.go)) monitors the operation and updates the sandbox's endpoint to point to the new host upon completion.

## Hypervisor State Machine and Memory Transfer

At the hypervisor layer, the **VMM** (Virtual Machine Monitor) component executes the migration protocol implemented in [`hypervisor/vmm/src/migration.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/vmm/src/migration.rs). This module defines a `MigrationState` enum that transitions through `Init`, `PreCopy`, `StopAndCopy`, `Resume`, and `Done` states. The core `run_migration` function drives the state machine, coordinating with the destination host via a TCP-based migration channel to stream the VM snapshot and memory pages.

### State Capture and Snapshotting

During the initial phase, the hypervisor pauses the guest VM only long enough to quiesce device I/O, then collects a complete VM snapshot comprising CPU registers, device configuration, and block-device state. This snapshot is immediately streamed to the destination host to establish the base state for the new VM instance.

### Incremental Memory Synchronization

Following the initial snapshot, the system enters the **pre-copy** phase. The hypervisor leverages **soft-dirty** bitmap tracking implemented in [`hypervisor/vmm/src/soft_dirty.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/vmm/src/soft_dirty.rs) to identify modified memory pages since the last transfer. The state machine iteratively sends these dirty pages to the destination while the source VM continues running, using the following logic:

1.  Collect dirty pages using the soft-dirty bitmap.
2.  Transmit the pages over the migration channel.
3.  Clear the dirty flags and repeat until the working set stabilizes.

When the host supports it, CubeSandbox utilizes **virtio-mem** for zero-copy ballooning optimizations, as documented in the live migration enhancements section of [`hypervisor/release-notes.md`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/release-notes.md). This incremental approach significantly reduces the amount of data that must transfer during the final stop-and-copy phase.

## Destination Coordination and Final Switchover

The destination host prepares by creating a new VM instance with an identical configuration and pre-allocating the required memory. As the source streams memory pages, the destination applies incremental updates in real-time. Once the final memory page transfers and the destination confirms receipt, the source VM stops, and the target VM resumes execution from the exact point of suspension. CubeMaster then finalizes the migration by updating the sandbox's endpoint and marking the state as *Running* again.

## Platform Constraints and Limitations

Live migration of VM states in CubeSandbox currently supports **x86_64 architectures only**; ARM64 hosts are restricted to cold migration, as noted in [`hypervisor/README.md`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/README.md). Additionally, migration across different major versions is explicitly disallowed to prevent schema and ABI mismatches between hypervisor instances. The implementation is designed to be **idempotent**—if a migration fails at any point, the source VM remains running and the operation can be safely retried without state corruption.

## Code Implementation Examples

### Triggering Migration via the Go SDK

The CubeSandbox SDK abstracts the underlying API calls. The following example demonstrates initiating a migration using the Go SDK:

```go
import "github.com/TencentCloud/CubeSandbox/sdk/go"

func migrateSandbox(sandboxID, destHost string) error {
    client := go.NewClient("https://api.cubesandbox.example")
    // The SDK exposes a MigrationRequest struct that maps to the
    // CubeMaster /sandboxes/:id/migrate endpoint.
    req := &go.MigrationRequest{
        DestinationHost: destHost,
        // optional: timeout, keep‑alive settings, etc.
    }
    _, err := client.Sandboxes.Migrate(sandboxID, req)
    return err
}

```

### Core Migration Logic in Rust

The internal state machine handles the complex coordination between source and destination. The following excerpt from [`hypervisor/vmm/src/migration.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/vmm/src/migration.rs) illustrates the migration flow:

```rust
// hypervisor/vmm/src/migration.rs (excerpt)
enum MigrationState {
    Init,
    PreCopy,
    StopAndCopy,
    Resume,
    Done,
}

fn run_migration(vm: &mut Vm, dst: &mut MigrationSink) -> Result<()> {
    // 1. Pause VM and snapshot registers
    vm.pause()?;
    let snap = vm.take_snapshot()?;
    dst.send_snapshot(snap)?;

    // 2. Pre‑copy dirty pages
    while vm.has_dirty_pages() {
        let pages = vm.collect_dirty_pages();
        dst.send_memory(pages)?;
        vm.clear_dirty();
    }

    // 3. Final stop‑and‑copy
    vm.stop()?;
    let final_pages = vm.collect_all_pages();
    dst.send_memory(final_pages)?;

    // 4. Resume on destination
    dst.finalize()?;
    Ok(())
}

```

## Summary

- CubeSandbox coordinates live migration of VM states through CubeMaster's control plane API and the hypervisor's state machine in [`hypervisor/vmm/src/migration.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/vmm/src/migration.rs).
- The system uses **soft-dirty** page tracking in [`hypervisor/vmm/src/soft_dirty.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/vmm/src/soft_dirty.rs) to enable efficient incremental memory transfer during the pre-copy phase, reducing final switchover time.
- Migration is currently **x86_64-only** and explicitly disallowed across different major versions to ensure ABI compatibility.
- The architecture is **idempotent**, ensuring source VMs remain running if migration fails, allowing for safe retry operations without service interruption.

## Frequently Asked Questions

### Does CubeSandbox support live migration on ARM64 architectures?

No, live migration of VM states is currently restricted to **x86_64** hosts. According to [`hypervisor/README.md`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/README.md), ARM64 hosts can only perform cold migration, which requires stopping the VM before transferring state.

### What happens if a live migration fails mid-transfer?

The migration implementation is **idempotent**. If the process fails at any stage, the source VM remains running and the target state is discarded. Operators can safely retry the migration without risking state corruption or service interruption on the source host.

### Can I migrate VMs between different CubeSandbox versions?

No, migration across different major versions is explicitly disallowed. The [`hypervisor/README.md`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/README.md) documentation notes that live migration is not supported across different versions to avoid schema and ABI mismatches between the source and destination hypervisors.

### How does the hypervisor minimize downtime during the final switchover?

The hypervisor minimizes downtime by using an **iterative pre-copy** algorithm. It transfers the majority of memory pages while the VM runs, then only pauses the VM briefly during the `StopAndCopy` phase to transfer the final dirty pages and CPU state. This final pause typically lasts only milliseconds, as implemented in [`hypervisor/vmm/src/migration.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/vmm/src/migration.rs).