# CubeVS eBPF Virtual Switch Architecture: Kernel-Level Network Isolation in CubeSandbox

> Explore the CubeVS eBPF virtual switch architecture for kernel-level network isolation in CubeSandbox. Learn how eBPF programs and LPM-Trie maps achieve sub-microsecond latency for sandboxed workloads.

- Repository: [Tencent Cloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox)
- Tags: architecture
- Published: 2026-07-10

---

**CubeVS is a high-performance eBPF virtual switch that attaches to the Linux Traffic Control (TC) subsystem to provide kernel-level network isolation for sandboxed workloads, using three specialized eBPF programs and LPM-Trie maps to enforce per-sandbox policies with sub-microsecond latency.**

CubeSandbox relies on CubeVS to isolate network traffic at the kernel level, ensuring that even privileged containers cannot bypass security boundaries. The architecture eliminates context-switch overhead by processing packets entirely within kernel space using eBPF programs compiled with **bpf2go** and pinned under `/sys/fs/bpf`. This design establishes the CubeVS eBPF virtual switch architecture as a zero-trust networking layer for high-density container deployments.

## Core Architecture and eBPF Hook Points

The CubeVS virtual switch intercepts traffic at three distinct TC attachment points, each handled by a dedicated eBPF program.

### The Three-Point Hook Design

- **nodenic** ([`nodenic.bpf.c`](https://github.com/TencentCloud/CubeSandbox/blob/main/nodenic.bpf.c)): Handles bidirectional traffic between the host node’s physical NIC and the sandbox-side virtual NIC (`cubegw0`).
- **localgw** ([`localgw.bpf.c`](https://github.com/TencentCloud/CubeSandbox/blob/main/localgw.bpf.c)): Implements "world-side" processing for packets leaving the sandbox (egress path).
- **mvmtap** ([`mvmtap.bpf.c`](https://github.com/TencentCloud/CubeSandbox/blob/main/mvmtap.bpf.c)): Provides "in-gateway" processing for packets entering the sandbox (ingress path).

### Build-Time Generation and Loading

In [`cubevs.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/cubevs.go), the build process uses `//go:generate` directives to invoke **bpf2go**, converting the C source files into Go-embeddable BPF bytecode. After loading, each program is pinned under `/sys/fs/bpf` and attached to the appropriate TC qdisc using the `tcHandleClsact` and `tcFilterHandle` constants (see lines 84-89 of [`cubevs.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/cubevs.go)).

## Data-Plane Components and eBPF Maps

CubeVS maintains isolation state through several specialized eBPF maps managed by the Go helper library in the **cubevs** package.

**Metadata Management**
- `ifindex_to_mvmmeta`: Stores per-TAP-device metadata including IP, UUID, and version. The function `UpsertTAPDeviceMeta` in [`tap.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/tap.go) (lines 58-100) updates this map alongside the reverse IP mapping.

**IP to Interface Mapping**
- `mvmip_to_ifindex`: Enables fast reverse lookups from sandbox IPs to TAP interface indices, ensuring packets route to the correct destination.

**Port Address Translation**
- `remote_port_mapping`: Outbound static port-NAT for external services.
- `local_port_mapping`: Inbound static port-NAT for external access.

**Policy Enforcement Maps**
- `allow_out_v2`: An LPM-Trie of `netPolicyValueV2` structures that enforces CIDR-based egress whitelisting with optional L7 awareness (via `netPolicyFlagL7Required`).
- `deny_out`: An LPM-Trie of `uint32` values implementing blacklist semantics for blocked destinations.
- `dns_allow`: A hash-of-maps storing hashed DNS names per sandbox, accessed via `dnsAllowKey` and `dnsAllowValue` structs.

**Source NAT Pool**
- `snat_iplist`: Holds a configurable pool of IPs used for source-NAT of outbound traffic, preventing cross-sandbox IP spoofing.

The network-policy maps are created on-demand per-sandbox interface by `initNetPolicy` in [`netpolicy.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/netpolicy.go) (lines 86-103). Inner maps use LPM-Trie structures created via `newInnerLPMMap` (lines 25-33) to enable efficient longest-prefix matching for CIDR checks.

## Packet Flow and Processing Logic

All forwarding decisions occur in kernel space, avoiding userland context switches.

### Ingress Flow (Sandbox to Host)

Packets received on the TAP device trigger the `mvmtap` eBPF program:

1. Reads the `ifindex_to_mvmmeta` map to identify the sandbox context.
2. Applies egress policies by querying the `allow_out_v2` or `deny_out` inner LPM-Trie maps.
3. Rewrites L2 headers to the host MAC address.
4. Redirects the packet to the world using `BPFRedirectFlagIngress`.

### Egress Flow (Host to Sandbox)

Outbound packets are intercepted by the `localgw` program attached to the `cubegw0` interface:

1. Performs SNAT using the `snat_iplist` map to masquerade the source address.
2. Consults `allow_out_v2` or `deny_out` maps for policy enforcement.
3. Redirects to the appropriate TAP device identified via the `mvmip_to_ifindex` map.

### Node-NIC Bridging

The `nodenic` program provides bidirectional forwarding between the host node’s physical NIC and the sandbox NIC. It handles L2 rewrite and optional DNS filtering through tail-call programs (`dns_parse_chunk`, `dns_rev_chunk`, etc.).

## Isolation Guarantees and Security Model

CubeVS enforces strict isolation through eBPF map constraints that privileged containers cannot circumvent.

### Per-Sandbox IP Isolation

Each sandbox receives a unique `MVMInnerIP` mapped to its TAP ifindex. The eBPF maps guarantee that packets can only be forwarded to the TAP device that owns the destination IP, preventing traffic leakage between sandboxes.

### CIDR-Based Egress Control

The `allow_out_v2` and `deny_out` LPM-Trie maps implement whitelist/blacklist semantics for outbound connections. The LPM-Trie structure enables efficient matching against CIDR ranges, while the L7 flag allows for protocol-aware filtering.

### DNS Filtering and SNAT Protection

The `dns_allow` map stores hashed DNS names checked by dedicated tail-call programs, filtering DNS queries at the kernel level. The `snat_iplist` pool ensures all egress traffic originates from a controlled address range, mitigating IP spoofing attacks across the sandbox fleet.

## Working with CubeVS (Go API Examples)

Create a new sandbox TAP device and configure its egress policies:

```go
// Initialize CubeVS with sandbox parameters
params := cubevs.Params{
    MVMInnerIP: net.ParseIP("10.1.2.3"),
    MVMMacAddr: net.HardwareAddr{0x02, 0x42, 0xac, 0x11, 0x00, 0x02},
    // … other required fields …
}
vs, _ := cubevs.New(params)

// Add TAP device with network policies
opts := cubevs.MVMOptions{
    AllowOut:            &[]string{"0.0.0.0/0"},
    L7AllowOut:          &[]string{"10.0.0.0/8"},
    DenyOut:             &[]string{"192.168.0.0/16"},
}
_ = vs.AddTAPDevice(5, net.ParseIP("10.1.2.3"), "sandbox-01", 1, opts)

```

List all managed TAP devices:

```go
taps, _ := cubevs.ListTAPDevices()
for _, t := range taps {
    fmt.Printf("TAP %s – IP %s – ifindex %d\n", t.ID, t.IP, t.Ifindex)
}

```

Remove a sandbox TAP device and clean up its policy maps:

```go
_ = cubevs.DelTAPDevice(5, net.ParseIP("10.1.2.3"))

```

## Key Source Files and Implementation Details

The CubeVS implementation spans the following files in the `CubeNet/cubevs` package:

- **cubevs.go**: Central package definition, constant declarations (`tcHandleClsact`, `tcFilterHandle`), and map names.
- **tap.go**: TAP-device lifecycle management (list, add, delete, lookup) and metadata handling via `UpsertTAPDeviceMeta`.
- **netpolicy.go**: LPM-Trie map creation, inner-map management, CIDR parsing, and policy cleanup via `initNetPolicy` and `newInnerLPMMap`.
- **snat.go**: SNAT IP pool management and map updates.
- **miscs.go**: Helper functions for loading/pinning eBPF objects, constant rewriting, and tail-call setup.
- **src/localgw.bpf.c**, **src/mvmtap.bpf.c**, **src/nodenic.bpf.c**: The eBPF C programs compiled to BPF bytecode.
- **CubeNet/cubevs/cmd/cubevsmapdump/main.go**: Utility for dumping BPF maps to debug isolation policies.

## Summary

- **CubeVS** attaches to three TC hooks (`nodenic`, `localgw`, `mvmtap`) to intercept traffic before it leaves the kernel.
- **LPM-Trie maps** (`allow_out_v2`, `deny_out`) enable efficient CIDR-based egress filtering with sub-microsecond lookup times.
- **Per-sandbox isolation** is enforced through dedicated eBPF maps (`ifindex_to_mvmmeta`, `mvmip_to_ifindex`) that bind IPs to specific TAP devices.
- **Go API** provides lifecycle management via `AddTAPDevice`, `DelTAPDevice`, and `ListTAPDevices` in the `cubevs` package.
- **Kernel-level enforcement** prevents bypass by privileged containers, as policies are applied by eBPF programs before packets reach the network stack.

## Frequently Asked Questions

### How does CubeVS differ from traditional OVS or Linux bridges?

**CubeVS operates entirely within the kernel's eBPF subsystem**, eliminating the context-switch overhead and packet copying required by userspace switches like Open vSwitch (OVS). Unlike Linux bridges, which rely on the kernel's bridge module and netfilter hooks, CubeVS attaches directly to TC qdiscs (clsact) using eBPF programs, enabling policy enforcement at the earliest possible point in the packet lifecycle with minimal latency.

### What eBPF map types does CubeVS use for policy enforcement?

**CubeVS primarily uses LPM-Trie (Longest Prefix Match Trie) maps for CIDR-based policies** and hash-of-maps for DNS filtering. The `allow_out_v2` and `deny_out` maps utilize inner LPM-Trie structures created by `newInnerLPMMap` in [`netpolicy.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/netpolicy.go), allowing efficient matching of destination IPs against CIDR ranges. For DNS filtering, it uses a hash-of-maps structure (`dns_allow`) keyed by hashed DNS names.

### Can sandboxed workloads bypass CubeVS network policies?

**No, sandboxed workloads cannot bypass CubeVS policies even when running privileged containers.** Because the eBPF programs attach to TC hooks in the kernel's network stack, they process packets before they reach the container's network namespace. The enforcement occurs in kernel space, making it impossible for container-level privilege escalation to circumvent the `allow_out_v2`, `deny_out`, or `dns_allow` map lookups.

### How are the eBPF programs compiled and loaded in CubeSandbox?

**The eBPF programs are compiled at build time using bpf2go** and loaded by the Go runtime during sandbox initialization. The [`cubevs.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/cubevs.go) file contains `//go:generate` directives that compile [`localgw.bpf.c`](https://github.com/TencentCloud/CubeSandbox/blob/main/localgw.bpf.c), [`mvmtap.bpf.c`](https://github.com/TencentCloud/CubeSandbox/blob/main/mvmtap.bpf.c), and [`nodenic.bpf.c`](https://github.com/TencentCloud/CubeSandbox/blob/main/nodenic.bpf.c) into BPF bytecode. The resulting programs are pinned under `/sys/fs/bpf` and attached to TC qdiscs using constants defined in [`cubevs.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/cubevs.go) (lines 84-89), ensuring persistence across process restarts.