# CubeVS eBPF Virtual Switch Architecture for Kernel-Level Network Isolation

> Explore the CubeVS eBPF virtual switch architecture for kernel-level network isolation. Achieve sub-microsecond latency and per-sandbox isolation with this high-performance solution.

- Repository: [Tencent Cloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox)
- Tags: architecture
- Published: 2026-07-11

---

**CubeVS is a high-performance eBPF-based virtual switch that attaches to the Linux Traffic Control (TC) subsystem to enforce per-sandbox network isolation entirely in kernel space, eliminating context-switch overhead and delivering sub-microsecond latency.**

CubeVS powers the network isolation layer in CubeSandbox, an open-source sandbox runtime maintained by TencentCloud. By leveraging eBPF programs attached to the TC subsystem, CubeVS intercepts and filters packets before they leave the kernel, providing robust security guarantees that cannot be bypassed even by privileged containers.

## eBPF Program Architecture and TC Hooks

CubeVS consists of three specialized eBPF programs that handle distinct traffic paths. These programs are generated at build time using **bpf2go** (as defined by the `//go:generate` directives in [`cubevs.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/cubevs.go)) and attached to **clsact** qdiscs on specific interfaces.

The three core programs are:

- **[`nodenic.bpf.c`](https://github.com/TencentCloud/CubeSandbox/blob/main/nodenic.bpf.c)** (`nodenic`): Handles bidirectional traffic between the host node's physical NIC and the sandbox-side virtual NIC (`cubegw0`).
- **[`localgw.bpf.c`](https://github.com/TencentCloud/CubeSandbox/blob/main/localgw.bpf.c)** (`localgw`): Processes egress packets leaving the sandbox ("world-side" processing).
- **[`mvmtap.bpf.c`](https://github.com/TencentCloud/CubeSandbox/blob/main/mvmtap.bpf.c)** (`mvmtap`): Handles ingress packets entering the sandbox ("in-gateway" processing).

After compilation, each program is pinned under `/sys/fs/bpf` and attached using TC constants defined in [`cubevs.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/cubevs.go) (lines 84-89), including `tcHandleClsact` and `tcFilterHandle`.

## Data-Plane Components and eBPF Maps

The data plane relies on several **BPF maps** created and managed by the Go helper library in the `cubevs` package. These maps store metadata, policy rules, and NAT configurations.

### Core Metadata Maps

- **`ifindex_to_mvmmeta`**: Stores per-TAP-device metadata including IP addresses, UUIDs, and version information.
- **`mvmip_to_ifindex`**: Provides fast reverse mapping from sandbox IPs to their corresponding interface indices.

### Network Policy Maps

- **`allow_out_v2`**: An LPM-Trie (Longest Prefix Match) storing `netPolicyValueV2` structures that whitelist egress CIDRs with optional L7-awareness flags.
- **`deny_out`**: An LPM-Trie of `uint32` values implementing blacklist-based egress filtering.
- **`dns_allow`**: A hash-of-maps structure storing hashed DNS names permitted for each sandbox.

### NAT and Port Mapping

- **`remote_port_mapping`** and **`local_port_mapping`**: Implement static port-NAT for external services.
- **`snat_iplist`**: Maintains a configurable pool of IPs used for source-NAT of outbound traffic.

The LPM-Trie inner maps are created on-demand by `initNetPolicy` (see [`netpolicy.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/netpolicy.go) lines 86-103) using `newInnerLPMMap` (lines 25-33), enabling efficient longest-prefix matching for CIDR-based policy enforcement.

## Packet Flow Through CubeVS

Traffic processing occurs entirely in kernel space through three distinct paths:

### Ingress Flow (Sandbox → Host)

Packets received on the TAP device enter the `mvmtap` eBPF program. The program:

1. Reads the `ifindex_to_mvmmeta` map to identify the source sandbox.
2. Consults the `allow_out_v2` or `deny_out` LPM-Trie maps to enforce egress policies.
3. Rewrites L2 headers to the host MAC address.
4. Redirects the packet to the world using `BPFRedirectFlagIngress`.

### Egress Flow (Host → Sandbox)

Outbound packets are intercepted by the `localgw` program attached to the `cubegw0` interface:

1. Performs SNAT using the `snat_iplist` map.
2. Checks policies against `allow_out_v2` or `deny_out` maps.
3. Looks up the destination TAP device via `mvmip_to_ifindex`.
4. Redirects the packet to the appropriate TAP interface.

### Node-NIC Flow

The `nodenic` program manages bidirectional forwarding between the host's physical NIC and sandbox virtual NICs, handling L2 rewrites and optional DNS filtering via tail-call programs (`dns_parse_chunk`, `dns_rev_chunk`).

## Isolation Guarantees and Security Model

CubeVS provides hardened isolation through several kernel-enforced mechanisms:

- **Per-Sandbox IP Isolation**: Each sandbox receives a unique `MVMInnerIP` mapped exclusively to its TAP interface index. The eBPF maps ensure packets route only to the TAP owning the destination IP.
- **CIDR-Based Egress Control**: LPM-Trie maps support fine-grained whitelist/blacklist semantics with L7-aware flags (e.g., `netPolicyFlagL7Required`).
- **DNS Filtering**: The `dns_allow` map stores hashed domain names checked by dedicated DNS tail-call programs, preventing DNS exfiltration.
- **SNAT Pool Protection**: The configurable SNAT pool ensures egress traffic originates from controlled addresses, preventing IP spoofing across sandboxes.

Because enforcement occurs in eBPF before packets reach the network stack, privileged containers cannot bypass these rules.

## Implementation and Code Examples

The `cubevs` package provides Go APIs for managing TAP devices and policies. The following examples demonstrate common operations.

### Creating a Sandbox TAP Device

```go
// Create a new sandbox TAP device and set its egress policies
params := cubevs.Params{
    MVMInnerIP: net.ParseIP("10.1.2.3"),
    MVMMacAddr: net.HardwareAddr{0x02, 0x42, 0xac, 0x11, 0x00, 0x02},
    // … other required fields …
}
vs, _ := cubevs.New(params)               // initialize CubeVS instance
opts := cubevs.MVMOptions{
    AllowOut:            &[]string{"0.0.0.0/0"},
    L7AllowOut:          &[]string{"10.0.0.0/8"},
    DenyOut:             &[]string{"192.168.0.0/16"},
}
_ = vs.AddTAPDevice(5, net.ParseIP("10.1.2.3"), "sandbox-01", 1, opts)

```

### Listing Managed TAP Devices

```go
// List all TAP devices currently managed by CubeVS
taps, _ := cubevs.ListTAPDevices()
for _, t := range taps {
    fmt.Printf("TAP %s – IP %s – ifindex %d\n", t.ID, t.IP, t.Ifindex)
}

```

### Removing a Sandbox TAP Device

```go
// Remove a sandbox TAP device (will also clean up its policy maps)
_ = cubevs.DelTAPDevice(5, net.ParseIP("10.1.2.3"))

```

### Key Source Files

| File | Description |
|------|-------------|
| [`cubevs.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/cubevs.go) | Central package definition, constant declarations, and map names. |
| [`tap.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/tap.go) | TAP-device lifecycle (list, add, delete, lookup) and metadata handling via `UpsertTAPDeviceMeta`. |
| [`netpolicy.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/netpolicy.go) | LPM-Trie map creation, inner-map management, CIDR parsing, and policy cleanup. |
| [`snat.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/snat.go) | SNAT-IP pool management and map updates. |
| [`miscs.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/miscs.go) | Helper functions for loading/pinning eBPF objects, constant rewriting, and tail-call setup. |
| [`src/localgw.bpf.c`](https://github.com/TencentCloud/CubeSandbox/blob/main/src/localgw.bpf.c), [`src/mvmtap.bpf.c`](https://github.com/TencentCloud/CubeSandbox/blob/main/src/mvmtap.bpf.c), [`src/nodenic.bpf.c`](https://github.com/TencentCloud/CubeSandbox/blob/main/src/nodenic.bpf.c) | The actual eBPF C programs compiled to BPF bytecode. |
| [`CubeNet/cubevs/cmd/cubevsmapdump/main.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeNet/cubevs/cmd/cubevsmapdump/main.go) | Utility to dump the BPF maps for debugging. |

## Summary

- CubeVS implements kernel-level network isolation using three eBPF programs (`nodenic`, `localgw`, `mvmtap`) attached to the Linux TC subsystem.
- The architecture relies on high-performance BPF maps including LPM-Tries for CIDR matching and hash maps for DNS and metadata storage.
- Packet processing occurs entirely in kernel space, delivering sub-microsecond latency while preventing bypass by privileged containers.
- The Go-based control plane in [`cubevs.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/cubevs.go) manages device lifecycle and policy enforcement through helpers like `UpsertTAPDeviceMeta` and `initNetPolicy`.

## Frequently Asked Questions

### How does CubeVS differ from traditional OVS or iptables-based isolation?

Traditional solutions rely on userspace packet processing or netfilter hooks that incur context-switch overhead and can be bypassed with sufficient privileges. CubeVS operates entirely within the kernel's TC subsystem using eBPF, eliminating context switches and providing tamper-proof enforcement that persists even inside privileged containers.

### What is the purpose of the LPM-Trie maps in CubeVS network policies?

The `allow_out_v2` and `deny_out` maps use LPM-Trie (Longest Prefix Match) inner maps created by `newInnerLPMMap` in [`netpolicy.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/netpolicy.go). This structure enables efficient O(log n) lookups for CIDR-based rules, allowing the eBPF program to rapidly determine whether a destination IP falls within an allowed or denied network range while maintaining minimal kernel memory overhead.

### Can CubeVS handle DNS-based filtering for egress control?

Yes. CubeVS implements DNS filtering through the `dns_allow` hash-of-maps structure and dedicated tail-call programs (`dns_parse_chunk`, `dns_rev_chunk`). When enabled, the eBPF programs hash queried domain names and verify them against the per-sandbox allow list before permitting the DNS request to leave the sandbox.

### How are the eBPF programs loaded and attached to network interfaces?

The control plane uses `bpf2go` (invoked via `//go:generate` directives in [`cubevs.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/cubevs.go)) to compile C source files into BPF bytecode. The Go runtime then loads these programs using constants like `tcHandleClsact` and `tcFilterHandle` (lines 84-89 of [`cubevs.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/cubevs.go)), pins them to `/sys/fs/bpf`, and attaches them to clsact qdiscs on the respective interfaces.