# How CubeSandbox Achieves Sub-6ms Startup Latency with MicroVM Isolation

> Discover how CubeSandbox achieves sub-6ms startup latency using MicroVM isolation. Learn about its Firecracker architecture, copy-on-write images, lazy DMA, and dedicated TAP networking for fast execution.

- Repository: [Tencent Cloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox)
- Tags: internals
- Published: 2026-07-03

---

**CubeSandbox achieves sub-6 millisecond startup latency by combining a stripped-down Firecracker-based micro-VM architecture, copy-on-write overlay images, lazy DMA mapping, and dedicated TAP networking to eliminate boot-time overhead at every layer.**

TencentCloud/CubeSandbox delivers production-grade sandbox isolation through a purpose-built micro-VM stack that launches in under 6 milliseconds. Unlike traditional virtualization that incurs hundreds of milliseconds of boot overhead, this open-source platform leverages a minimal hypervisor, specialized kernel configurations, and intelligent storage overlays to achieve **sub-6ms startup latency with MicroVM isolation** that is orders of magnitude faster than conventional containers or VMs.

## Micro-VM Architecture and the Tiny VMM

### Firecracker-Based Minimal Hypervisor

At the core of CubeSandbox sits a **tiny VMM** derived from the Firecracker micro-VM project. Located in `hypervisor/vmm/src/` and launched via [`hypervisor/src/main.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/src/main.rs), this hypervisor eliminates legacy device emulation and user-space bloat. By stripping the virtualization layer to only essential components, the VMM executes fewer instructions during launch, directly contributing to the sub-6ms startup window.

### Minimal Linux Kernel Configuration

The platform utilizes a **minimal Linux kernel** configured in `configs/kernel-oc9.*.config`. This kernel includes only drivers required for sandbox workloads—specifically virtio-net, virtio-fs, and vsock—omitting unnecessary modules and hardware support. The reduced image size requires less memory to load and initializes faster during early boot, shaving critical milliseconds off the startup sequence.

## Copy-on-Write Image Management

### Overlay-Based Template System

CubeSandbox eliminates full image copies through an **overlay-based template system** implemented in [`Cubelet/storage/overlay/overlay.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/storage/overlay/overlay.go). A base template image loads once into memory, and each new sandbox mounts a lightweight copy-on-write overlay layer pointing to this base. When creating a sandbox, Cubelet simply mounts the overlay rather than copying gigabytes of data, reducing storage initialization to a matter of microseconds.

## Hardware-Optimized I/O and Memory

### Lazy PVDMA Mapping

The **PVDMA** (pass-through DMA) technique, implemented in [`hypervisor/vmm/src/memory_manager.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/vmm/src/memory_manager.rs), establishes DMA mappings for guest memory lazily rather than pre-pinning all pages at startup. This defers costly memory registration until devices actually access specific pages, preserving the tight sub-6ms latency budget that would otherwise be consumed by bulk memory setup operations.

### Virtio-Based Device Model

Instead of emulating legacy PCI hardware, CubeSandbox uses **virtio-based devices** including virtio-net, virtio-fs, and vsock, defined in `hypervisor/virtio-devices/src/`. This paravirtualized approach eliminates PCI initialization delays and legacy device probe sequences, providing high-throughput, low-latency I/O without the overhead of hardware emulation.

### Dedicated TAP Networking

Each sandbox receives its own **dedicated TAP device** via code in [`hypervisor/net_util/src/tap.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/net_util/src/tap.rs), avoiding shared bridges or software switches. This point-to-point networking architecture eliminates ARP storms and extra packet hops, ensuring network interfaces are ready immediately upon VM boot.

## Snapshot and Control Plane Optimizations

### Soft-Dirty Incremental Snapshots

For pause/resume operations, CubeSandbox implements **soft-dirty incremental snapshots** in [`hypervisor/vmm/src/soft_dirty.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/vmm/src/soft_dirty.rs). This mechanism tracks only memory pages dirtied since the last checkpoint, avoiding full-memory copies during state transitions. By reducing pause latency, the system maintains the overall fast lifecycle required for sub-6ms startup scenarios.

### Fast Binary API Protocol

The control plane uses a compact binary protocol via `micro_http` in `hypervisor/api/`, defined in [`openapi.yml`](https://github.com/TencentCloud/CubeSandbox/blob/main/openapi.yml). Replacing verbose JSON serialization with minimal binary messaging reduces control-plane latency, allowing CubeMaster to signal Cubelet to launch VMs with minimal overhead between API call and VMM execution.

## Orchestration Architecture

### CubeMaster and Cubelet Coordination

The startup flow orchestrates between **CubeMaster** and **Cubelet** components. CubeMaster selects the appropriate template and instructs Cubelet to provision a new micro-VM. Cubelet then mounts the storage overlay, attaches the dedicated TAP interface, and invokes the VMM entry point. This coordinated pipeline ensures that by the time [`hypervisor/src/main.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/src/main.rs) executes, all prerequisites are pre-staged and ready.

## Practical Implementation Examples

### Creating a Sandbox with the Go SDK

```go
import (
    "github.com/TencentCloud/CubeSandbox/sdk/go"
)

func main() {
    client := cubesandbox.NewClient("http://localhost:8080")
    // Use the "default" template with pre-loaded minimal Linux image
    sandbox, err := client.CreateSandbox(cubesandbox.CreateOpts{
        Template: "default",
        Resources: cubesandbox.Resources{
            MemoryMB:  128,
            CpuCount:  1,
        },
    })
    if err != nil {
        panic(err)
    }
    fmt.Printf("Sandbox %s started in %d ms\n", sandbox.ID, sandbox.StartupLatencyMs)
}

```

This example demonstrates requesting a sandbox using the overlay-based "default" template. The `StartupLatencyMs` field typically reports approximately 5 milliseconds on commodity hardware.

### Direct API Invocation

```bash
curl -X POST http://localhost:8080/v1/sandboxes \
     -H "Content-Type: application/json" \
     -d '{"template":"default","memory_mb":128,"cpu_count":1}'

```

The JSON response includes a `startup_latency_ms` field confirming **sub-6ms** performance.

### Inspecting Overlay Mounts

To verify the copy-on-write mechanism on the host:

```bash

# Check overlay mount for a specific sandbox ID

mount | grep /var/lib/cubelet/overlays/<sandbox-id>

```

The output shows an overlay filesystem with the base image as the lower layer, confirming no full copy occurred during startup.

## Summary

- **CubeSandbox** achieves sub-6ms startup by stripping the hypervisor to essentials in `hypervisor/vmm/src/` and [`hypervisor/src/main.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/src/main.rs).
- **Copy-on-write overlays** in [`Cubelet/storage/overlay/overlay.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/storage/overlay/overlay.go) eliminate image copy overhead by reusing base templates.
- **Lazy PVDMA** mapping in [`hypervisor/vmm/src/memory_manager.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/vmm/src/memory_manager.rs) defers memory page pinning until actual device access.
- **Virtio-based devices** and **dedicated TAP interfaces** remove legacy hardware initialization delays.
- **Soft-dirty snapshots** enable fast pause/resume without full memory copies.
- **Binary API protocols** minimize control-plane latency between CubeMaster and Cubelet.

## Frequently Asked Questions

### How does CubeSandbox compare to Firecracker for startup latency?

CubeSandbox builds upon the Firecracker micro-VM architecture but adds specialized optimizations including lazy PVDMA mapping and soft-dirty incremental snapshots that further reduce startup time. While Firecracker achieves sub-100ms boots, CubeSandbox's sub-6ms latency represents a significant optimization through aggressive kernel minimization and overlay-based storage.

### What kernel configurations enable the sub-6ms boot time?

The platform uses minimal kernel configurations located in `configs/kernel-oc9.*.config` that compile in only virtio-net, virtio-fs, and vsock drivers while excluding all unnecessary hardware support and modules. This reduced kernel image loads faster into memory and initializes fewer subsystems during early boot.

### Why does CubeSandbox use dedicated TAP devices instead of virtual bridges?

Each sandbox receives its own TAP device via [`hypervisor/net_util/src/tap.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/net_util/src/tap.rs) to create a direct point-to-point packet path. This eliminates the latency introduced by shared bridges, ARP resolution, and software switching, ensuring network interfaces are immediately available when the micro-VM starts.

### Can I use custom templates with the overlay storage system?

Yes, the overlay system in [`Cubelet/storage/overlay/overlay.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/storage/overlay/overlay.go) supports custom templates. You can create base images with specific toolchains or dependencies, and Cubelet will mount copy-on-write overlays for each new sandbox instance, maintaining the sub-6ms startup latency regardless of template complexity.