# CubeVS eBPF Virtual Switch Implementation: Kernel-Level Network Isolation in CubeSandbox

> Explore CubeVS, a high-performance eBPF virtual switch implementation enabling kernel-level network isolation for sandboxed workloads with fine-grained policy enforcement.

- Repository: [Tencent Cloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox)
- Tags: deep-dive
- Published: 2026-07-03

---

**CubeVS is a high-performance eBPF virtual switch that isolates sandboxed workloads at the kernel level by attaching TC hook programs to network interfaces and enforcing fine-grained policies through LPM-Trie maps pinned under `/sys/fs/bpf`.**

CubeVS powers the network isolation layer of TencentCloud's CubeSandbox, providing sub-microsecond latency packet processing without userspace context switches. This eBPF virtual switch implementation intercepts traffic at three strategic TC hook points to enforce per-sandbox CIDR policies, DNS filtering, and SNAT rules entirely within the kernel.

## Architecture and Hook Points

The CubeVS eBPF virtual switch attaches to the Linux **TC** (traffic control) subsystem using the `clsact` qdisc. According to the source code in [`CubeNet/cubevs/cubevs.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeNet/cubevs/cubevs.go), the implementation uses constants such as `tcHandleClsact` and `tcFilterHandle` (lines 84‑89) to manage attachment points.

Three specialized eBPF programs handle different traffic vectors:

- **[`nodenic.bpf.c`](https://github.com/TencentCloud/CubeSandbox/blob/main/nodenic.bpf.c)** – Handles bidirectional traffic between the host node's physical NIC and the sandbox-side virtual NIC (`cubegw0`).
- **[`localgw.bpf.c`](https://github.com/TencentCloud/CubeSandbox/blob/main/localgw.bpf.c)** – Processes egress packets leaving the sandbox (world-side forwarding).
- **[`mvmtap.bpf.c`](https://github.com/TencentCloud/CubeSandbox/blob/main/mvmtap.bpf.c)** – Handles ingress packets entering the sandbox (in-gateway processing).

These programs are generated at build time using **bpf2go**, as indicated by the `//go:generate` directives in [`CubeNet/cubevs/cubevs.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeNet/cubevs/cubevs.go). After compilation, each program is pinned under `/sys/fs/bpf` for persistent access across process restarts.

## Data Plane and eBPF Maps

The CubeVS data plane relies on several high-performance BPF maps to maintain state and enforce policies. These maps are created and managed by the Go helper library in the `cubevs` package.

### Metadata and Identity Maps

- **`ifindex_to_mvmmeta`** – Stores per-TAP-device metadata including IP addresses, UUIDs, and version information. The function `UpsertTAPDeviceMeta` in [`CubeNet/cubevs/tap.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeNet/cubevs/tap.go) (lines 58‑100) updates this map alongside the reverse IP mapping.
- **`mvmip_to_ifindex`** – Provides fast reverse lookup from sandbox IPs to their corresponding TAP interface indices.

### Network Policy Maps

- **`allow_out_v2`** – An LPM-Trie (longest prefix match) containing `netPolicyValueV2` structures that whitelist egress destinations with optional L7 awareness.
- **`deny_out`** – An LPM-Trie of `uint32` values implementing blacklist semantics for egress traffic.
- **`dns_allow`** – A hash-of-maps structure storing allowed DNS names per sandbox, checked by DNS tail-call programs (`dns_parse_chunk`, `dns_rev_chunk`).

### Translation and NAT Maps

- **`remote_port_mapping`** and **`local_port_mapping`** – Implement static port-NAT for external services.
- **`snat_iplist`** – Maintains a configurable pool of IPs used for source-NAT of outbound traffic.

The network policy maps are created on-demand per-sandbox interface by `initNetPolicy` (see [`CubeNet/cubevs/netpolicy.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeNet/cubevs/netpolicy.go) lines 86‑103). Inner maps use LPM-Trie structures created via `newInnerLPMMap` (lines 25‑33) to enable efficient CIDR matching.

## Packet Flow and Processing Logic

CubeVS processes packets entirely in kernel space through three distinct flows:

### Ingress Path (Sandbox → Host)

Packets received on the TAP device enter the `mvmtap` eBPF program. The program reads the `ifindex_to_mvmmeta` map to identify the sandbox context, consults the `allow_out_v2` or `deny_out` LPM-Trie maps for egress policy enforcement, and rewrites L2 headers to the host MAC before redirecting the packet to the world using `BPFRedirectFlagIngress`.

### Egress Path (Host → Sandbox)

Outbound packets are intercepted by the `localgw` program attached to the `cubegw0` interface. The program performs SNAT using the `snat_iplist` map, validates traffic against egress policies, and redirects to the appropriate TAP device identified via the `mvmip_to_ifindex` map.

### Node-NIC Path

The `nodenic` program provides bidirectional forwarding between the host node's physical NIC and sandbox NICs, handling L2 rewrite and optional DNS filtering at the entry point.

## Isolation Guarantees and Security Model

CubeVS provides hardened isolation through several kernel-enforced mechanisms:

- **Per-Sandbox IP Isolation** – Each sandbox receives a unique inner IP (`MVMInnerIP`) mapped to its TAP ifindex; the eBPF maps ensure packets can only reach the TAP owning the destination IP.
- **CIDR-Based Egress Control** – LPM-Trie maps implement whitelist/blacklist semantics with optional L7 flags (`netPolicyFlagL7Required`) for fine-grained outbound filtering.
- **DNS Filtering** – Hashed DNS names stored in the `dns_allow` map prevent sandboxed workloads from resolving unauthorized domains.
- **SNAT Pool Enforcement** – A dedicated pool of SNAT IPs prevents IP spoofing and ensures egress traffic originates from controlled address ranges.

Because policies are enforced by eBPF before packets exit the kernel, privileged containers cannot bypass these rules.

## Working with CubeVS

The `cubevs` package provides Go APIs to manage sandbox network interfaces and policies.

### Creating a Sandbox TAP Device

```go
// Initialize CubeVS instance with sandbox parameters
params := cubevs.Params{
    MVMInnerIP: net.ParseIP("10.1.2.3"),
    MVMMacAddr: net.HardwareAddr{0x02, 0x42, 0xac, 0x11, 0x00, 0x02},
    // ... other required fields ...
}
vs, _ := cubevs.New(params)

// Define egress policies with CIDR-based allow/deny rules
opts := cubevs.MVMOptions{
    AllowOut:   &[]string{"0.0.0.0/0"},
    L7AllowOut: &[]string{"10.0.0.0/8"},
    DenyOut:    &[]string{"192.168.0.0/16"},
}

// Add TAP device with index 5
_ = vs.AddTAPDevice(5, net.ParseIP("10.1.2.3"), "sandbox-01", 1, opts)

```

### Listing Managed TAP Devices

```go
// Retrieve all TAP devices managed by CubeVS
taps, _ := cubevs.ListTAPDevices()
for _, t := range taps {
    fmt.Printf("TAP %s – IP %s – ifindex %d\n", t.ID, t.IP, t.Ifindex)
}

```

### Removing a Sandbox Interface

```go
// Delete TAP device and clean up associated policy maps
_ = cubevs.DelTAPDevice(5, net.ParseIP("10.1.2.3"))

```

## Key Source Files

The implementation spans several critical files in the `CubeNet/cubevs` directory:

| File | Description |
|------|-------------|
| [`cubevs.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/cubevs.go) | Central package definition, constant declarations (`tcHandleClsact`, `tcFilterHandle`), and map names. |
| [`tap.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/tap.go) | TAP-device lifecycle management including `UpsertTAPDeviceMeta` for metadata handling. |
| [`netpolicy.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/netpolicy.go) | LPM-Trie map creation, inner-map management, and CIDR parsing via `initNetPolicy` and `newInnerLPMMap`. |
| [`snat.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/snat.go) | SNAT IP pool management and map updates. |
| [`miscs.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/miscs.go) | Helper functions for loading/pinning eBPF objects and tail-call setup. |
| [`src/localgw.bpf.c`](https://github.com/TencentCloud/CubeSandbox/blob/main/src/localgw.bpf.c), [`src/mvmtap.bpf.c`](https://github.com/TencentCloud/CubeSandbox/blob/main/src/mvmtap.bpf.c), [`src/nodenic.bpf.c`](https://github.com/TencentCloud/CubeSandbox/blob/main/src/nodenic.bpf.c) | The actual eBPF C programs compiled to BPF bytecode. |
| [`cmd/cubevsmapdump/main.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/cmd/cubevsmapdump/main.go) | Utility for dumping BPF maps for debugging. |

## Summary

- CubeVS implements kernel-level network isolation using eBPF programs attached to the TC subsystem at three hook points (`nodenic`, `localgw`, `mvmtap`).
- Network policies are enforced through LPM-Trie maps (`allow_out_v2`, `deny_out`) that support CIDR-based matching with optional L7 filtering.
- The data plane uses high-performance BPF maps for metadata (`ifindex_to_mvmmeta`), IP-to-interface mapping (`mvmip_to_ifindex`), and SNAT pools (`snat_iplist`).
- All packet processing occurs in kernel space via `CubeNet/cubevs` Go bindings, eliminating context-switch overhead and delivering sub-microsecond latency.
- Isolation guarantees include per-sandbox IP restrictions, DNS filtering through `dns_allow` maps, and IP anti-spoofing via dedicated SNAT pools.

## Frequently Asked Questions

### What is CubeVS and how does it differ from traditional virtual switches?

CubeVS is an eBPF-based virtual switch built into TencentCloud's CubeSandbox that operates at the kernel level rather than in userspace. Unlike traditional OVS or Linux bridge implementations that require packets to traverse the network stack and context-switch to a daemon, CubeVS programs attach directly to the TC subsystem and process packets within the kernel, resulting in sub-microsecond latency and stronger isolation guarantees.

### How does CubeVS enforce network policies at the kernel level?

CubeVS enforces policies using LPM-Trie (longest prefix match) BPF maps created by `initNetPolicy` in [`netpolicy.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/netpolicy.go). When a packet traverses the `mvmtap` or `localgw` programs, the eBPF code performs lookups in `allow_out_v2` or `deny_out` maps to determine if the destination CIDR is permitted. These checks occur before the packet reaches the host network stack, preventing bypass attempts even from privileged containers.

### What are the performance benefits of using eBPF for network isolation in CubeSandbox?

The CubeVS eBPF virtual switch eliminates userspace copies and context switches by processing traffic entirely within the kernel's TC hook points. According to the implementation in [`src/mvmtap.bpf.c`](https://github.com/TencentCloud/CubeSandbox/blob/main/src/mvmtap.bpf.c) and [`src/localgw.bpf.c`](https://github.com/TencentCloud/CubeSandbox/blob/main/src/localgw.bpf.c), packets are redirected using `BPFRedirectFlagIngress` and direct map lookups, achieving sub-microsecond latency for intra-node traffic while maintaining CIDR-based policy enforcement.

### How are the eBPF programs loaded and managed in CubeVS?

The eBPF programs are compiled using **bpf2go** via `//go:generate` directives in [`cubevs.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/cubevs.go), then loaded and pinned under `/sys/fs/bpf` by the Go helper library. The [`miscs.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/miscs.go) file handles object loading and pinning, while [`tap.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/tap.go) manages the lifecycle of TAP devices and their associated maps. This architecture ensures that eBPF resources persist independently of the userspace process lifecycle.