How CubeSandbox Achieves Sub-60ms Cold Start and Ultra-Low Memory Overhead

CubeSandbox achieves sub-60ms cold starts and sub-5MiB memory overhead per instance by combining pre-snapshotted VM templates, XFS reflink cloning via CubeCoW, and direct memory restoration through RustVMM, eliminating traditional boot sequences and container image unpacking.

CubeSandbox is TencentCloud's open-source sandbox runtime engineered to launch isolated AI-Agent environments in a few tens of milliseconds while maintaining each instance's RAM footprint under 5MiB. By integrating a custom storage engine, a specialized Rust-based hypervisor, and a stateless control plane, the project redefines virtualization efficiency for high-density workloads.

Pre-Snapshotted Templates and Instant Restoration

Traditional virtual machines waste time booting kernels and probing devices. CubeSandbox bypasses this entirely by starting every sandbox from a template that contains a frozen root filesystem and a memory snapshot of a minimal Linux kernel plus essential libraries.

Frozen Filesystem and Memory Snapshots

Instead of unpacking OCI images or running a full OS boot sequence, CubeSandbox relies on templates that are ready to restore. As documented in [docs/architecture/overview.md](https://github.com/TencentCloud/CubeSandbox/blob/master/docs/architecture/overview.md), these templates capture the exact state of a running minimal environment, allowing the system to skip initialization scripts and device discovery.

The RustVMM Restore Path

The hypervisor (CubeHypervisor) uses the RustVMM library to inject memory snapshots directly into the VM's RAM region. In [hypervisor/vmm/src/vm.rs](https://github.com/TencentCloud/CubeSandbox/blob/master/hypervisor/vmm/src/vm.rs), the restore_vm function handles this restoration:

fn restore_vm(&mut self, mem_snapshot: &Path) -> Result<()> {
    // Load the memory snapshot directly into the guest's RAM region
    self.vm.set_memory_region(mem_snapshot)?;
    self.vm.resume()?;
    Ok(())
}

This approach is orders of magnitude faster than traditional boot sequences because it eliminates init scripts and device probing entirely.

CubeCoW Storage Engine with XFS Reflink

Storage operations often dominate cold-start latency. CubeSandbox solves this through CubeCoW, a Rust library that leverages the XFS FICLONE ioctl to perform O(1) reflink clones of both root filesystems and memory volumes.

Sub-Millisecond Volume Cloning

Cloning a 1GiB rootfs takes only microseconds and consumes no additional disk space because only metadata is copied. According to [cubecow/src/lib.rs](https://github.com/TencentCloud/CubeSandbox/blob/master/cubecow/src/lib.rs), the implementation allows Cubelet to instantiate new volumes instantly:

// Cubelet calls CubeCoW to clone the template volume
let rootfs = cubecow::clone_volume(&tmpl.rootfs, "sandbox-rootfs")?;
let mem   = cubecow::clone_volume(&tmpl.memory, "sandbox-mem")?;
shim.start_vm(rootfs, mem)?;

In [Cubelet/storage/local.go](https://github.com/TencentCloud/CubeSandbox/blob/master/Cubelet/storage/local.go), the node agent coordinates these operations to ensure local storage paths are prepared before the hypervisor attempts restoration.

Incremental Dirty-Page Tracking

Maintaining ultra-low memory overhead requires aggressive page sharing. CubeSandbox implements incremental dirty-page tracking to minimize per-instance RAM usage.

After the initial snapshot, only anonymous pages that change are written to new snapshots. Unchanged pages remain shared via reflinks across thousands of instances. This mechanism, managed in [hypervisor/vmm/src/memory_manager.rs](https://github.com/TencentCloud/CubeSandbox/blob/master/hypervisor/vmm/src/memory_manager.rs), ensures that subsequent starts reuse the same shared pages, keeping RAM usage per sandbox under 5MiB even when running thousands of concurrent agents on a single host.

Containerd Shim v2 Integration

CubeSandbox eliminates full container runtime overhead by implementing CubeShim, a Containerd Shim v2 interface written in Rust. Located in [agent/src/main.rs](https://github.com/TencentCloud/CubeSandbox/blob/master/agent/src/main.rs), the shim handles VSock communication and invokes the hypervisor's restore_vm path directly.

Unlike traditional container runtimes that manage cgroups and namespaces, CubeShim simply hands off the VM instantly after restoration succeeds. This removes layers of abstraction that typically add latency to sandbox creation.

Stateless Control Plane and Zero-Copy Networking

Redis-Backed Metadata

All sandbox metadata lives in Redis, while CubeAPI, CubeMaster, and Cubelet remain completely stateless. This design allows node-local agents to retrieve template information without remote lookups, reducing planning latency during cold starts.

eBPF-Based Networking

Network configuration happens in parallel through CubeVS, which uses eBPF programs to forward traffic directly between TAP devices and the host NIC. This bypasses iptables and bridge processing, ensuring network paths are ready the moment the VM restoration completes.

The Cold Start Flow Step-by-Step

The complete cold-start sequence that achieves sub-60ms latency works as follows:

1. Template Clone – Cubelet requests CubeCoW to reflink-clone the template's rootfs and memory volumes locally.
2. VM Restoration – CubeShim calls launch_vmm → create_vm → restore_vm, injecting the memory snapshot directly into the VM's RAM region via RustVMM.
3. Ready Signal – The restored VM opens a VSock listener, CubeShim reports success, and the sandbox becomes reachable instantly.

Because the memory snapshot is already loaded into the VM's address space and the disk image is a lightweight reflink clone, the entire sequence completes in under 60ms on typical x86_64 hardware.

Creating Sandboxes with the Go SDK

Developers can leverage these optimizations through the CubeSandbox Go SDK. Configuration specifies the minimal 4MiB memory allocation that enables the ultra-low overhead:

client, _ := cubesandbox.NewClient(cubesandbox.WithEndpoint("https://api.my-cube.dev"))
sandbox, _ := client.CreateSandbox(context.TODO(), cubesandbox.SandboxConfig{
    TemplateID: "template-ubuntu-lite",
    MemoryMiB:  4,                     // <5MiB per sandbox
    CPU:        0.1,
})
fmt.Println("Sandbox started:", sandbox.ID)

Summary

• Pre-snapshotted templates eliminate OS boot time by providing frozen filesystem and memory states ready for immediate restoration.
• CubeCoW uses XFS FICLONE ioctl to create O(1) reflink clones of volumes, enabling sub-millisecond storage preparation.
• RustVMM restore path injects memory snapshots directly into RAM regions, bypassing traditional boot sequences.
• Incremental dirty-page tracking shares unchanged memory pages across instances, keeping per-sandbox RAM under 5MiB.
• Containerd Shim v2 implementation in Rust eliminates container runtime overhead by directly invoking hypervisor restoration.
• Zero-copy networking via eBPF prepares network paths instantly without iptables or bridge processing.

Frequently Asked Questions

What makes CubeSandbox faster than traditional container cold starts?

Traditional containers must unpack image layers, initialize namespaces, and execute startup scripts. CubeSandbox replaces these steps with a single memory restoration operation using RustVMM and reflink-cloned storage volumes, reducing startup time from seconds to sub-60 milliseconds.

How does CubeCoW differ from standard copy-on-write storage?

Standard copy-on-write systems duplicate data blocks on first write, consuming I/O and space. CubeCoW leverages XFS reflinks (FICLONE ioctl) to create O(1) metadata-only clones that share physical blocks between templates and instances, consuming microseconds rather than seconds and no additional disk space for unchanged data.

Why does each sandbox consume less than 5MiB of RAM?

CubeSandbox uses incremental dirty-page tracking to share memory pages from the original template across all instances. Only modified anonymous pages consume new RAM, while code segments and read-only data remain shared via reflinks, enabling thousands of concurrent sandboxes on a single host.

Can CubeSandbox run on filesystems other than XFS?

The current implementation requires XFS for the reflink cloning capabilities used by CubeCoW. Other filesystems would need to support equivalent clone or reflink operations to achieve the same O(1) storage performance and ultra-low memory characteristics documented in the project.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}