CubeSandbox Data Plane Components: Architecture and Implementation

The CubeSandbox data plane comprises seven specialized components—Cubelet, CubeShim, CubeHypervisor, CubeCoW, CubeVS, CubeEgress, and CubeProxy—that collectively handle VM lifecycle management, high-performance networking, storage snapshots, and L7 traffic control on each node.

The CubeSandbox data plane is the node-local runtime stack that executes and secures sandbox workloads in the TencentCloud/CubeSandbox project. Unlike the control plane (CubeAPI, CubeMaster, Redis, WebUI), this data plane operates autonomously on each host to ensure that management failures or upgrades never disrupt running sandboxes.

Core Components of the CubeSandbox Data Plane

The data plane architecture separates concerns across seven distinct components, each implemented with specific technologies to optimize for security, performance, and isolation.

Cubelet

Cubelet serves as the node-local scheduler and orchestration agent. It creates, runs, pauses, resumes, snapshots, and destroys sandbox VMs while driving the data-plane lifecycle. The component attaches TAP devices to sandboxes and installs network policies by coordinating with CubeVS.

According to the architecture documentation in docs/architecture/overview.md, Cubelet acts as the primary integration point that ties together TAP creation, VM launch via CubeShim/Hypervisor, and storage operations through CubeCoW.

CubeShim

CubeShim implements the containerd Shim v2 interface in Rust, bridging the container-runtime world to the MicroVM. It prepares resources, launches the VM, and provides vsock communication channels between the host and guest.

This component is crucial for maintaining compatibility with standard container orchestration while providing VM-level isolation.

CubeHypervisor

CubeHypervisor is the lightweight Virtual Machine Monitor (VMM) built on RustVMM + KVM. It runs the sandbox's kernel and virtual devices, providing the foundation for MicroVM-based isolation.

As listed in the architecture overview, this component represents the virtualization layer of the data plane.

CubeCoW

CubeCoW is the storage engine (implemented as a Rust library) that provides O(1) snapshot and clone operations. It utilizes the kernel FICLONE ioctl on XFS to manage root filesystem and memory volumes for the data plane.

All storage volumes for running sandboxes are managed through this engine, enabling instant cloning without data duplication.

CubeVS (Cube Virtual Switch)

CubeVS is a pure eBPF-based kernel-space network data plane. It implements three distinct eBPF programs:

• from_cube: Handles traffic originating from sandboxes
• from_world: Processes inbound traffic from external sources
• from_envoy: Manages traffic from the L7 proxy layer

These programs collectively handle SNAT/DNAT, per-sandbox L3/L4 policy enforcement, ARP proxy, and port-mapping. The implementation resides in CubeNet/cubevs/tap.go, where functions like AddTAPDevice register TAP metadata and apply eBPF policies.

CubeEgress

CubeEgress is a host-local L7 transparent proxy built on OpenResty and Lua. All outbound HTTP/HTTPS traffic from sandboxes passes through this component for:

• Domain filtering
• Credential injection
• Audit logging

The configuration and logic are defined in CubeEgress/nginx.conf and CubeEgress/lua/redactor.lua.

CubeProxy

CubeProxy functions as a reverse proxy (nginx/OpenResty) that routes inbound client traffic to the appropriate sandbox VM. It works in conjunction with CubeEgress to provide complete egress and ingress control.

Data Plane Operations and Code Examples

The following Go snippets demonstrate how these components interact during typical data plane operations.

Listing TAP Devices Managed by CubeVS

To inspect network interfaces currently managed by the eBPF virtual switch:

package main

import (
	"fmt"
	"log"

	"github.com/TencentCloud/CubeSandbox/CubeNet/cubevs"
)

func main() {
	taps, err := cubevs.ListTAPDevices()
	if err != nil {
		log.Fatalf("cannot list TAP devices: %v", err)
	}
	for _, t := range taps {
		fmt.Printf("TAP ifindex=%d ip=%s id=%s\n", t.Ifindex, t.IP, t.ID)
	}
}

This utilizes the ListTAPDevices function defined in CubeNet/cubevs/tap.go (lines 18-42).

Adding a TAP Device with Network Policy

To create a new TAP interface and apply security policies:

opts := cubevs.MVMOptions{
	AllowOut:   &[]string{"0.0.0.0/0"},
	AllowInternetAccess: nil, // defaults to false
}
err := cubevs.AddTAPDevice(5, net.ParseIP("10.240.0.2"), "sandbox-123", 1, opts)
if err != nil {
    log.Fatalf("failed to add TAP: %v", err)
}

The AddTAPDevice function (lines 45-55 in CubeNet/cubevs/tap.go) registers TAP metadata and applies eBPF network policies through the from_cube program.

Orchestrating Sandbox Launch

Cubelet coordinates the full lifecycle by chaining operations across components:

// Inside Cubelet (simplified)
if err := cubelet.AddTAPDevice(ifIdx, ip, sandboxID, cfg); err != nil {
    return err
}
if err := cubelet.StartVM(sandboxID); err != nil {
    return err
}

This sequence first creates the network interface via CubeVS, then boots the VM through CubeShim and CubeHypervisor.

Key Source Files and Implementation

The following files constitute the complete CubeSandbox data plane implementation:

• docs/architecture/overview.md — High-level description of data-plane components and responsibilities
• CubeNet/cubevs/tap.go — Core API for managing TAP devices and applying eBPF network policies
• CubeNet/cubevs/cubevs.go — Initialization of eBPF maps and loading of BPF programs
• CubeEgress/nginx.conf and CubeEgress/lua/redactor.lua — L7 egress proxy configuration and Lua logic for domain filtering
• CubeProxy/nginx.conf — Reverse-proxy configuration for inbound traffic routing
• Cubelet/ — Node-local agent tying together TAP creation, VM launch, and storage
• CubeShim/ — Rust shim implementing containerd Shim v2 with vsock communication
• CubeHypervisor/ — RustVMM-based VMM for MicroVM execution
• CubeCoW/ — Go/Rust library for O(1) snapshot/clone via XFS reflink

Summary

• The CubeSandbox data plane consists of seven specialized components: Cubelet, CubeShim, CubeHypervisor, CubeCoW, CubeVS, CubeEgress, and CubeProxy.
• CubeVS provides high-performance networking through eBPF programs (from_cube, from_world, from_envoy) that operate in kernel space.
• CubeCoW achieves O(1) storage snapshots using the FICLONE ioctl on XFS, eliminating copy overhead during VM cloning.
• The architecture maintains strict separation from the control plane, ensuring that CubeAPI or CubeMaster failures cannot affect running sandboxes.
• All components are implemented in Rust (CubeShim, CubeHypervisor, CubeCoW core) and Go (Cubelet, CubeVS), with OpenResty/Lua handling L7 traffic management.

Frequently Asked Questions

What is the difference between CubeVS and CubeEgress?

CubeVS operates at L3/L4 as an eBPF-based virtual switch handling packet forwarding, NAT, and network policies in kernel space. CubeEgress operates at L7 as an application-layer proxy (OpenResty/Lua) that inspects HTTP/HTTPS traffic for domain filtering, credential injection, and audit logging. While CubeVS handles all sandbox network traffic, CubeEgress specifically processes outbound web traffic.

How does CubeCoW achieve O(1) snapshot performance?

CubeCoW utilizes the XFS reflink feature via the kernel FICLONE ioctl. This creates copy-on-write clones of root filesystem and memory volumes without duplicating data blocks, allowing instantaneous snapshots regardless of volume size. The implementation in the CubeCoW/ directory provides both Go and Rust bindings for this functionality.

Is the CubeSandbox data plane independent of the control plane?

Yes, the data plane is designed as a node-local autonomous stack that functions independently from the control plane (CubeAPI, CubeMaster, Redis, WebUI). According to the architecture documentation, this design guarantees that upgrades, restarts, or failures in control plane components never impact sandboxes already running on a host.

What runtime interface does CubeShim implement?

CubeShim implements the containerd Shim v2 API in Rust. This interface allows CubeSandbox to integrate with standard Kubernetes and containerd deployments while providing VM-level isolation through MicroVMs. The shim handles resource preparation, VM launch, and vsock communication between host and guest.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}