Sandbox Migration with Cross-Node Pause and Resume in CubeSandbox: A Complete Guide

CubeSandbox migrates running sandboxes between nodes by pausing the source VM, serializing its full state (memory, CPU registers, and filesystem) into a snapshot, streaming it over the network, and resuming execution on the destination node.

CubeSandbox implements sandbox migration with cross-node pause and resume through a coordinated freeze-and-transfer mechanism managed by the CubeMaster control plane. This process allows lightweight virtual machines to move between physical hosts with only milliseconds of downtime, utilizing the internal vm-migration crate and REST-style APIs exposed via CubeAPI. The migration captures a consistent point-in-time snapshot of the entire execution environment, transmits it securely to the destination, and reconstructs the sandbox state to continue operation seamlessly.

The Cross-Node Migration Process

The migration flow consists of seven distinct phases orchestrated between the source node, destination node, and CubeMaster. Each phase maps to specific implementation files in the TencentCloud/CubeSandbox repository.

Step 1: Initiating Migration via CubeAPI

Migration begins when the scheduler or user triggers a VmSendMigration request. In CubeAPI/src/state.rs, the ApiRequest::VmSendMigration handler processes the request and invokes the migration logic on the source node.

The request specifies the destination URL (typically a Unix-domain socket or TCP endpoint) and sets the local parameter to false for cross-node transfers.

// Go client using the CubeAPI SDK
import "github.com/TencentCloud/CubeSandbox/sdk"

func MigrateSandbox(sandboxID, srcNode, dstNode string) error {
    payload := sdk.VmSendMigration{
        DestinationUrl: fmt.Sprintf("unix:///var/run/cube/%s/migration.sock", dstNode),
        Local:          false, // remote migration
    }
    _, err := sdk.NewClient(srcNode).VmSendMigration(sandboxID, payload)
    return err
}

Step 2: Pausing the VM Source

The hypervisor implements the Pausable trait to freeze the VM state. In hypervisor/vmm/src/vm.rs, the start_migration method signals the freeze operation:

fn start_migration(&mut self) -> Result<(), MigratableError> {
    self.memory_manager.lock().unwrap().start_migration()?;
    self.device_manager.lock().unwrap().start_migration()
}

This blocks new memory writes and flushes pending I/O operations, ensuring a consistent memory image. The memory_manager.start_migration() and device_manager.start_migration() calls in hypervisor/vmm/src/memory_manager.rs prevent further modifications during the snapshot window.

Step 3: Creating the Snapshot

With the VM paused, the system creates a vm_migration::Snapshot instance containing:

• SNAPSHOT_STATE_FILE: CPU registers and device state
• SNAPSHOT_CONFIG_FILE: VM configuration metadata
• memory_blob: Raw RAM image

The snapshot logic resides in hypervisor/vmm/src/migration.rs, which serializes these components to temporary files under /tmp/ch-migration-<name>-<nanos>.

// From hypervisor/vmm/src/lib.rs
let snapshot_config = vm_migration_config.clone();
let snapshot = vm.create_snapshot(SnapshotType::Full, &snapshot_config)?;

Step 4: Transferring the Snapshot

The source node opens a socket connection to the destination and streams the snapshot files using the vm_migration::protocol. This protocol includes chunked transmission with CRC32 verification to ensure data integrity over the network.

In hypervisor/vmm/src/lib.rs, the vm_send_migration function handles the socket creation and file streaming:

fn vm_send_migration(...) {
    // Opens Unix-domain or TCP socket based on destination URL
    // Streams snapshot files with CRC32 verification
}

Step 5: Receiving on the Destination

The target node executes VmReceiveMigration via its CubeAPI endpoint. In hypervisor/vmm/src/lib.rs, the vm_receive_migration function reads the incoming stream, reconstructs the memory image, and writes the state files to the local sandbox storage area.

// From hypervisor/vmm/src/lib.rs
let path = Self::socket_url_to_path(&receive_data_migration.receiver_url)?;
let snapshot = vm.receive_snapshot(&path)?;   // streams the files
vm.apply_snapshot(snapshot)?;                // restores memory & state

Step 6: Resuming the VM

After the snapshot is applied, the destination invokes the resume sequence. The complete_migration method in hypervisor/vmm/src/vm.rs re-enables memory writes and finalizes device state:

fn complete_migration(&mut self) -> Result<(), MigratableError> {
    self.memory_manager.lock().unwrap().complete_migration()?;
    self.device_manager.lock().unwrap().complete_migration()
}

This sequence transitions the sandbox from the frozen snapshot state to active execution, completing the pause and resume cycle.

Step 7: Cleanup and Verification

Temporary snapshot files are removed from /tmp/, and the source sandbox is shut down or moved to a stopped state. The CubeMaster control plane updates the sandbox metadata to reflect the new host location and monitors the migration status to trigger rollback if necessary.

Implementation Deep Dive: The Pause and Resume Hooks

The consistency guarantee relies on the Pausable trait implementation across the memory and device managers. When start_migration is called:

1. Memory Manager: Tracks dirty pages and blocks new allocations
2. Device Manager: Flushes pending I/O operations and pauses device emulation
3. VCPU Threads: Enter a quiescent state to prevent register modification

The complete_migration hooks perform the inverse operation, reactivating memory mapping and resuming device timers. This architecture ensures that the snapshot captures an exact point-in-time state without requiring guest operating system cooperation.

Summary

• Sandbox migration with cross-node pause and resume in CubeSandbox involves freezing the VM, serializing state to snapshot files, and transferring them to a destination node before resuming execution.
• The process is orchestrated through CubeAPI/src/state.rs handlers for VmSendMigration and VmReceiveMigration.
• Consistency is achieved through the Pausable trait implemented in hypervisor/vmm/src/vm.rs, coordinating memory_manager and device_manager freeze operations.
• State transfer uses the vm_migration::protocol with CRC32 verification, supporting both Unix-domain sockets (local) and TCP (remote) transports.
• Downtime is limited to the pause-and-snapshot window, typically milliseconds, while the bulk network transfer occurs while the VM is stopped.

Frequently Asked Questions

How long does the VM pause last during migration?

The pause duration is typically a few milliseconds, limited to the time required to capture memory and device state into the snapshot files. The bulk network transfer happens while the VM is already paused, so the total downtime depends on the snapshot creation speed rather than the network bandwidth.

What happens if the migration fails during the transfer?

CubeMaster monitors the migration status asynchronously. If the snapshot transfer fails or the destination node cannot apply the state, the control plane can retry the operation or roll back to the source node, which maintains the original paused state until the migration is confirmed successful.

Does CubeSandbox support live migration without pausing?

Currently, the implementation requires a full pause-and-resume cycle to ensure consistency. While the source code structure in hypervisor/vmm/src/migration.rs supports the foundation for live migration (streaming dirty pages), the master branch implements the freeze-based approach described here for guaranteed atomic snapshots.

How is the filesystem handled during cross-node migration?

The snapshot includes the sandbox's rootfs state stored in a copy-on-write image. The destination node reconstructs the full filesystem from the snapshot files, eliminating host-specific dependencies and allowing the workload to resume with its exact disk state intact.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}