# Sandbox Migration with Cross-Node Pause and Resume in CubeSandbox: A Complete Guide

> Master sandbox migration with cross-node pause and resume in CubeSandbox. Learn how to serialize, stream, and resume VM states for seamless transitions. Get your complete guide now.

- Repository: [Tencent Cloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox)
- Tags: migration-guide
- Published: 2026-07-16

---

**CubeSandbox migrates running sandboxes between nodes by pausing the source VM, serializing its full state (memory, CPU registers, and filesystem) into a snapshot, streaming it over the network, and resuming execution on the destination node.**

CubeSandbox implements sandbox migration with cross-node pause and resume through a coordinated freeze-and-transfer mechanism managed by the CubeMaster control plane. This process allows lightweight virtual machines to move between physical hosts with only milliseconds of downtime, utilizing the internal *vm-migration* crate and REST-style APIs exposed via CubeAPI. The migration captures a consistent point-in-time snapshot of the entire execution environment, transmits it securely to the destination, and reconstructs the sandbox state to continue operation seamlessly.

## The Cross-Node Migration Process

The migration flow consists of seven distinct phases orchestrated between the source node, destination node, and CubeMaster. Each phase maps to specific implementation files in the TencentCloud/CubeSandbox repository.

### Step 1: Initiating Migration via CubeAPI

Migration begins when the scheduler or user triggers a `VmSendMigration` request. In [`CubeAPI/src/state.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeAPI/src/state.rs), the `ApiRequest::VmSendMigration` handler processes the request and invokes the migration logic on the source node.

The request specifies the destination URL (typically a Unix-domain socket or TCP endpoint) and sets the `local` parameter to `false` for cross-node transfers.

```go
// Go client using the CubeAPI SDK
import "github.com/TencentCloud/CubeSandbox/sdk"

func MigrateSandbox(sandboxID, srcNode, dstNode string) error {
    payload := sdk.VmSendMigration{
        DestinationUrl: fmt.Sprintf("unix:///var/run/cube/%s/migration.sock", dstNode),
        Local:          false, // remote migration
    }
    _, err := sdk.NewClient(srcNode).VmSendMigration(sandboxID, payload)
    return err
}

```

### Step 2: Pausing the VM Source

The hypervisor implements the **Pausable** trait to freeze the VM state. In [`hypervisor/vmm/src/vm.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/vmm/src/vm.rs), the `start_migration` method signals the freeze operation:

```rust
fn start_migration(&mut self) -> Result<(), MigratableError> {
    self.memory_manager.lock().unwrap().start_migration()?;
    self.device_manager.lock().unwrap().start_migration()
}

```

This blocks new memory writes and flushes pending I/O operations, ensuring a consistent memory image. The `memory_manager.start_migration()` and `device_manager.start_migration()` calls in [`hypervisor/vmm/src/memory_manager.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/vmm/src/memory_manager.rs) prevent further modifications during the snapshot window.

### Step 3: Creating the Snapshot

With the VM paused, the system creates a `vm_migration::Snapshot` instance containing:
- **SNAPSHOT_STATE_FILE**: CPU registers and device state
- **SNAPSHOT_CONFIG_FILE**: VM configuration metadata
- **memory_blob**: Raw RAM image

The snapshot logic resides in [`hypervisor/vmm/src/migration.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/vmm/src/migration.rs), which serializes these components to temporary files under `/tmp/ch-migration-<name>-<nanos>`.

```rust
// From hypervisor/vmm/src/lib.rs
let snapshot_config = vm_migration_config.clone();
let snapshot = vm.create_snapshot(SnapshotType::Full, &snapshot_config)?;

```

### Step 4: Transferring the Snapshot

The source node opens a socket connection to the destination and streams the snapshot files using the `vm_migration::protocol`. This protocol includes chunked transmission with CRC32 verification to ensure data integrity over the network.

In [`hypervisor/vmm/src/lib.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/vmm/src/lib.rs), the `vm_send_migration` function handles the socket creation and file streaming:

```rust
fn vm_send_migration(...) {
    // Opens Unix-domain or TCP socket based on destination URL
    // Streams snapshot files with CRC32 verification
}

```

### Step 5: Receiving on the Destination

The target node executes `VmReceiveMigration` via its CubeAPI endpoint. In [`hypervisor/vmm/src/lib.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/vmm/src/lib.rs), the `vm_receive_migration` function reads the incoming stream, reconstructs the memory image, and writes the state files to the local sandbox storage area.

```rust
// From hypervisor/vmm/src/lib.rs
let path = Self::socket_url_to_path(&receive_data_migration.receiver_url)?;
let snapshot = vm.receive_snapshot(&path)?;   // streams the files
vm.apply_snapshot(snapshot)?;                // restores memory & state

```

### Step 6: Resuming the VM

After the snapshot is applied, the destination invokes the resume sequence. The `complete_migration` method in [`hypervisor/vmm/src/vm.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/vmm/src/vm.rs) re-enables memory writes and finalizes device state:

```rust
fn complete_migration(&mut self) -> Result<(), MigratableError> {
    self.memory_manager.lock().unwrap().complete_migration()?;
    self.device_manager.lock().unwrap().complete_migration()
}

```

This sequence transitions the sandbox from the frozen snapshot state to active execution, completing the **pause and resume** cycle.

### Step 7: Cleanup and Verification

Temporary snapshot files are removed from `/tmp/`, and the source sandbox is shut down or moved to a stopped state. The `CubeMaster` control plane updates the sandbox metadata to reflect the new host location and monitors the migration status to trigger rollback if necessary.

## Implementation Deep Dive: The Pause and Resume Hooks

The consistency guarantee relies on the **Pausable** trait implementation across the memory and device managers. When `start_migration` is called:

1. **Memory Manager**: Tracks dirty pages and blocks new allocations
2. **Device Manager**: Flushes pending I/O operations and pauses device emulation
3. **VCPU Threads**: Enter a quiescent state to prevent register modification

The `complete_migration` hooks perform the inverse operation, reactivating memory mapping and resuming device timers. This architecture ensures that the snapshot captures an exact point-in-time state without requiring guest operating system cooperation.

## Summary

- **Sandbox migration with cross-node pause and resume** in CubeSandbox involves freezing the VM, serializing state to snapshot files, and transferring them to a destination node before resuming execution.
- The process is orchestrated through [`CubeAPI/src/state.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeAPI/src/state.rs) handlers for `VmSendMigration` and `VmReceiveMigration`.
- **Consistency** is achieved through the **Pausable** trait implemented in [`hypervisor/vmm/src/vm.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/vmm/src/vm.rs), coordinating `memory_manager` and `device_manager` freeze operations.
- **State transfer** uses the `vm_migration::protocol` with CRC32 verification, supporting both Unix-domain sockets (local) and TCP (remote) transports.
- **Downtime** is limited to the pause-and-snapshot window, typically milliseconds, while the bulk network transfer occurs while the VM is stopped.

## Frequently Asked Questions

### How long does the VM pause last during migration?

The pause duration is typically a few milliseconds, limited to the time required to capture memory and device state into the snapshot files. The bulk network transfer happens while the VM is already paused, so the total downtime depends on the snapshot creation speed rather than the network bandwidth.

### What happens if the migration fails during the transfer?

CubeMaster monitors the migration status asynchronously. If the snapshot transfer fails or the destination node cannot apply the state, the control plane can retry the operation or roll back to the source node, which maintains the original paused state until the migration is confirmed successful.

### Does CubeSandbox support live migration without pausing?

Currently, the implementation requires a full pause-and-resume cycle to ensure consistency. While the source code structure in [`hypervisor/vmm/src/migration.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/vmm/src/migration.rs) supports the foundation for live migration (streaming dirty pages), the master branch implements the freeze-based approach described here for guaranteed atomic snapshots.

### How is the filesystem handled during cross-node migration?

The snapshot includes the sandbox's rootfs state stored in a copy-on-write image. The destination node reconstructs the full filesystem from the snapshot files, eliminating host-specific dependencies and allowing the workload to resume with its exact disk state intact.