# CubeSandbox Data Plane Components: Architecture and Implementation

> Explore the CubeSandbox data plane's seven components: Cubelet, CubeShim, CubeHypervisor, CubeCoW, CubeVS, CubeEgress, and CubeProxy. Learn how they manage VMs, networking, storage, and traffic.

- Repository: [Tencent Cloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox)
- Tags: architecture
- Published: 2026-07-08

---

**The CubeSandbox data plane comprises seven specialized components—Cubelet, CubeShim, CubeHypervisor, CubeCoW, CubeVS, CubeEgress, and CubeProxy—that collectively handle VM lifecycle management, high-performance networking, storage snapshots, and L7 traffic control on each node.**

The **CubeSandbox data plane** is the node-local runtime stack that executes and secures sandbox workloads in the TencentCloud/CubeSandbox project. Unlike the control plane (CubeAPI, CubeMaster, Redis, WebUI), this data plane operates autonomously on each host to ensure that management failures or upgrades never disrupt running sandboxes.

## Core Components of the CubeSandbox Data Plane

The data plane architecture separates concerns across seven distinct components, each implemented with specific technologies to optimize for security, performance, and isolation.

### Cubelet

**Cubelet** serves as the node-local scheduler and orchestration agent. It creates, runs, pauses, resumes, snapshots, and destroys sandbox VMs while driving the data-plane lifecycle. The component attaches **TAP devices** to sandboxes and installs network policies by coordinating with CubeVS.

According to the architecture documentation in [`docs/architecture/overview.md`](https://github.com/TencentCloud/CubeSandbox/blob/main/docs/architecture/overview.md), Cubelet acts as the primary integration point that ties together TAP creation, VM launch via CubeShim/Hypervisor, and storage operations through CubeCoW.

### CubeShim

**CubeShim** implements the **containerd Shim v2 interface** in Rust, bridging the container-runtime world to the MicroVM. It prepares resources, launches the VM, and provides **vsock communication** channels between the host and guest.

This component is crucial for maintaining compatibility with standard container orchestration while providing VM-level isolation.

### CubeHypervisor

**CubeHypervisor** is the lightweight Virtual Machine Monitor (VMM) built on **RustVMM + KVM**. It runs the sandbox's kernel and virtual devices, providing the foundation for MicroVM-based isolation.

As listed in the architecture overview, this component represents the virtualization layer of the data plane.

### CubeCoW

**CubeCoW** is the storage engine (implemented as a Rust library) that provides **O(1) snapshot and clone** operations. It utilizes the kernel `FICLONE` ioctl on XFS to manage root filesystem and memory volumes for the data plane.

All storage volumes for running sandboxes are managed through this engine, enabling instant cloning without data duplication.

### CubeVS (Cube Virtual Switch)

**CubeVS** is a pure **eBPF-based kernel-space network data plane**. It implements three distinct eBPF programs:

- **`from_cube`**: Handles traffic originating from sandboxes
- **`from_world`**: Processes inbound traffic from external sources  
- **`from_envoy`**: Manages traffic from the L7 proxy layer

These programs collectively handle SNAT/DNAT, per-sandbox L3/L4 policy enforcement, ARP proxy, and port-mapping. The implementation resides in [`CubeNet/cubevs/tap.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeNet/cubevs/tap.go), where functions like `AddTAPDevice` register TAP metadata and apply eBPF policies.

### CubeEgress

**CubeEgress** is a host-local **L7 transparent proxy** built on OpenResty and Lua. All outbound HTTP/HTTPS traffic from sandboxes passes through this component for:

- Domain filtering
- Credential injection  
- Audit logging

The configuration and logic are defined in [`CubeEgress/nginx.conf`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeEgress/nginx.conf) and [`CubeEgress/lua/redactor.lua`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeEgress/lua/redactor.lua).

### CubeProxy

**CubeProxy** functions as a **reverse proxy** (nginx/OpenResty) that routes inbound client traffic to the appropriate sandbox VM. It works in conjunction with CubeEgress to provide complete egress and ingress control.

## Data Plane Operations and Code Examples

The following Go snippets demonstrate how these components interact during typical data plane operations.

### Listing TAP Devices Managed by CubeVS

To inspect network interfaces currently managed by the eBPF virtual switch:

```go
package main

import (
	"fmt"
	"log"

	"github.com/TencentCloud/CubeSandbox/CubeNet/cubevs"
)

func main() {
	taps, err := cubevs.ListTAPDevices()
	if err != nil {
		log.Fatalf("cannot list TAP devices: %v", err)
	}
	for _, t := range taps {
		fmt.Printf("TAP ifindex=%d ip=%s id=%s\n", t.Ifindex, t.IP, t.ID)
	}
}

```

This utilizes the `ListTAPDevices` function defined in [`CubeNet/cubevs/tap.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeNet/cubevs/tap.go) (lines 18-42).

### Adding a TAP Device with Network Policy

To create a new TAP interface and apply security policies:

```go
opts := cubevs.MVMOptions{
	AllowOut:   &[]string{"0.0.0.0/0"},
	AllowInternetAccess: nil, // defaults to false
}
err := cubevs.AddTAPDevice(5, net.ParseIP("10.240.0.2"), "sandbox-123", 1, opts)
if err != nil {
    log.Fatalf("failed to add TAP: %v", err)
}

```

The `AddTAPDevice` function (lines 45-55 in [`CubeNet/cubevs/tap.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeNet/cubevs/tap.go)) registers TAP metadata and applies eBPF network policies through the `from_cube` program.

### Orchestrating Sandbox Launch

Cubelet coordinates the full lifecycle by chaining operations across components:

```go
// Inside Cubelet (simplified)
if err := cubelet.AddTAPDevice(ifIdx, ip, sandboxID, cfg); err != nil {
    return err
}
if err := cubelet.StartVM(sandboxID); err != nil {
    return err
}

```

This sequence first creates the network interface via CubeVS, then boots the VM through CubeShim and CubeHypervisor.

## Key Source Files and Implementation

The following files constitute the complete CubeSandbox data plane implementation:

- **[`docs/architecture/overview.md`](https://github.com/TencentCloud/CubeSandbox/blob/main/docs/architecture/overview.md)** — High-level description of data-plane components and responsibilities
- **[`CubeNet/cubevs/tap.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeNet/cubevs/tap.go)** — Core API for managing TAP devices and applying eBPF network policies
- **[`CubeNet/cubevs/cubevs.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeNet/cubevs/cubevs.go)** — Initialization of eBPF maps and loading of BPF programs
- **[`CubeEgress/nginx.conf`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeEgress/nginx.conf)** and **[`CubeEgress/lua/redactor.lua`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeEgress/lua/redactor.lua)** — L7 egress proxy configuration and Lua logic for domain filtering
- **[`CubeProxy/nginx.conf`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeProxy/nginx.conf)** — Reverse-proxy configuration for inbound traffic routing
- **`Cubelet/`** — Node-local agent tying together TAP creation, VM launch, and storage
- **`CubeShim/`** — Rust shim implementing containerd Shim v2 with vsock communication
- **`CubeHypervisor/`** — RustVMM-based VMM for MicroVM execution
- **`CubeCoW/`** — Go/Rust library for O(1) snapshot/clone via XFS reflink

## Summary

- The **CubeSandbox data plane** consists of seven specialized components: Cubelet, CubeShim, CubeHypervisor, CubeCoW, CubeVS, CubeEgress, and CubeProxy.
- **CubeVS** provides high-performance networking through eBPF programs (`from_cube`, `from_world`, `from_envoy`) that operate in kernel space.
- **CubeCoW** achieves O(1) storage snapshots using the `FICLONE` ioctl on XFS, eliminating copy overhead during VM cloning.
- The architecture maintains **strict separation** from the control plane, ensuring that CubeAPI or CubeMaster failures cannot affect running sandboxes.
- All components are implemented in **Rust** (CubeShim, CubeHypervisor, CubeCoW core) and **Go** (Cubelet, CubeVS), with **OpenResty/Lua** handling L7 traffic management.

## Frequently Asked Questions

### What is the difference between CubeVS and CubeEgress?

**CubeVS** operates at L3/L4 as an eBPF-based virtual switch handling packet forwarding, NAT, and network policies in kernel space. **CubeEgress** operates at L7 as an application-layer proxy (OpenResty/Lua) that inspects HTTP/HTTPS traffic for domain filtering, credential injection, and audit logging. While CubeVS handles all sandbox network traffic, CubeEgress specifically processes outbound web traffic.

### How does CubeCoW achieve O(1) snapshot performance?

CubeCoW utilizes the **XFS reflink** feature via the kernel `FICLONE` ioctl. This creates copy-on-write clones of root filesystem and memory volumes without duplicating data blocks, allowing instantaneous snapshots regardless of volume size. The implementation in the `CubeCoW/` directory provides both Go and Rust bindings for this functionality.

### Is the CubeSandbox data plane independent of the control plane?

Yes, the data plane is designed as a **node-local autonomous stack** that functions independently from the control plane (CubeAPI, CubeMaster, Redis, WebUI). According to the architecture documentation, this design guarantees that upgrades, restarts, or failures in control plane components never impact sandboxes already running on a host.

### What runtime interface does CubeShim implement?

CubeShim implements the **containerd Shim v2 API** in Rust. This interface allows CubeSandbox to integrate with standard Kubernetes and containerd deployments while providing VM-level isolation through MicroVMs. The shim handles resource preparation, VM launch, and vsock communication between host and guest.