CubeVS eBPF Virtual Switch Architecture for Kernel-Level Network Isolation

CubeVS is a high-performance eBPF-based virtual switch that attaches to the Linux Traffic Control (TC) subsystem to enforce per-sandbox network isolation entirely in kernel space, eliminating context-switch overhead and delivering sub-microsecond latency.

CubeVS powers the network isolation layer in CubeSandbox, an open-source sandbox runtime maintained by TencentCloud. By leveraging eBPF programs attached to the TC subsystem, CubeVS intercepts and filters packets before they leave the kernel, providing robust security guarantees that cannot be bypassed even by privileged containers.

eBPF Program Architecture and TC Hooks

CubeVS consists of three specialized eBPF programs that handle distinct traffic paths. These programs are generated at build time using bpf2go (as defined by the //go:generate directives in cubevs.go) and attached to clsact qdiscs on specific interfaces.

The three core programs are:

• nodenic.bpf.c (nodenic): Handles bidirectional traffic between the host node's physical NIC and the sandbox-side virtual NIC (cubegw0).
• localgw.bpf.c (localgw): Processes egress packets leaving the sandbox ("world-side" processing).
• mvmtap.bpf.c (mvmtap): Handles ingress packets entering the sandbox ("in-gateway" processing).

After compilation, each program is pinned under /sys/fs/bpf and attached using TC constants defined in cubevs.go (lines 84-89), including tcHandleClsact and tcFilterHandle.

Data-Plane Components and eBPF Maps

The data plane relies on several BPF maps created and managed by the Go helper library in the cubevs package. These maps store metadata, policy rules, and NAT configurations.

Core Metadata Maps

• ifindex_to_mvmmeta: Stores per-TAP-device metadata including IP addresses, UUIDs, and version information.
• mvmip_to_ifindex: Provides fast reverse mapping from sandbox IPs to their corresponding interface indices.

Network Policy Maps

• allow_out_v2: An LPM-Trie (Longest Prefix Match) storing netPolicyValueV2 structures that whitelist egress CIDRs with optional L7-awareness flags.
• deny_out: An LPM-Trie of uint32 values implementing blacklist-based egress filtering.
• dns_allow: A hash-of-maps structure storing hashed DNS names permitted for each sandbox.

NAT and Port Mapping

• remote_port_mapping and local_port_mapping: Implement static port-NAT for external services.
• snat_iplist: Maintains a configurable pool of IPs used for source-NAT of outbound traffic.

The LPM-Trie inner maps are created on-demand by initNetPolicy (see netpolicy.go lines 86-103) using newInnerLPMMap (lines 25-33), enabling efficient longest-prefix matching for CIDR-based policy enforcement.

Packet Flow Through CubeVS

Traffic processing occurs entirely in kernel space through three distinct paths:

Ingress Flow (Sandbox → Host)

Packets received on the TAP device enter the mvmtap eBPF program. The program:

1. Reads the ifindex_to_mvmmeta map to identify the source sandbox.
2. Consults the allow_out_v2 or deny_out LPM-Trie maps to enforce egress policies.
3. Rewrites L2 headers to the host MAC address.
4. Redirects the packet to the world using BPFRedirectFlagIngress.

Egress Flow (Host → Sandbox)

Outbound packets are intercepted by the localgw program attached to the cubegw0 interface:

1. Performs SNAT using the snat_iplist map.
2. Checks policies against allow_out_v2 or deny_out maps.
3. Looks up the destination TAP device via mvmip_to_ifindex.
4. Redirects the packet to the appropriate TAP interface.

Node-NIC Flow

The nodenic program manages bidirectional forwarding between the host's physical NIC and sandbox virtual NICs, handling L2 rewrites and optional DNS filtering via tail-call programs (dns_parse_chunk, dns_rev_chunk).

Isolation Guarantees and Security Model

CubeVS provides hardened isolation through several kernel-enforced mechanisms:

• Per-Sandbox IP Isolation: Each sandbox receives a unique MVMInnerIP mapped exclusively to its TAP interface index. The eBPF maps ensure packets route only to the TAP owning the destination IP.
• CIDR-Based Egress Control: LPM-Trie maps support fine-grained whitelist/blacklist semantics with L7-aware flags (e.g., netPolicyFlagL7Required).
• DNS Filtering: The dns_allow map stores hashed domain names checked by dedicated DNS tail-call programs, preventing DNS exfiltration.
• SNAT Pool Protection: The configurable SNAT pool ensures egress traffic originates from controlled addresses, preventing IP spoofing across sandboxes.

Because enforcement occurs in eBPF before packets reach the network stack, privileged containers cannot bypass these rules.

Implementation and Code Examples

The cubevs package provides Go APIs for managing TAP devices and policies. The following examples demonstrate common operations.

Creating a Sandbox TAP Device

// Create a new sandbox TAP device and set its egress policies
params := cubevs.Params{
    MVMInnerIP: net.ParseIP("10.1.2.3"),
    MVMMacAddr: net.HardwareAddr{0x02, 0x42, 0xac, 0x11, 0x00, 0x02},
    // … other required fields …
}
vs, _ := cubevs.New(params)               // initialize CubeVS instance
opts := cubevs.MVMOptions{
    AllowOut:            &[]string{"0.0.0.0/0"},
    L7AllowOut:          &[]string{"10.0.0.0/8"},
    DenyOut:             &[]string{"192.168.0.0/16"},
}
_ = vs.AddTAPDevice(5, net.ParseIP("10.1.2.3"), "sandbox-01", 1, opts)

Listing Managed TAP Devices

// List all TAP devices currently managed by CubeVS
taps, _ := cubevs.ListTAPDevices()
for _, t := range taps {
    fmt.Printf("TAP %s – IP %s – ifindex %d\n", t.ID, t.IP, t.Ifindex)
}

Removing a Sandbox TAP Device

// Remove a sandbox TAP device (will also clean up its policy maps)
_ = cubevs.DelTAPDevice(5, net.ParseIP("10.1.2.3"))

Key Source Files

File	Description
cubevs.go	Central package definition, constant declarations, and map names.
tap.go	TAP-device lifecycle (list, add, delete, lookup) and metadata handling via UpsertTAPDeviceMeta.
netpolicy.go	LPM-Trie map creation, inner-map management, CIDR parsing, and policy cleanup.
snat.go	SNAT-IP pool management and map updates.
miscs.go	Helper functions for loading/pinning eBPF objects, constant rewriting, and tail-call setup.
src/localgw.bpf.c, src/mvmtap.bpf.c, src/nodenic.bpf.c	The actual eBPF C programs compiled to BPF bytecode.
CubeNet/cubevs/cmd/cubevsmapdump/main.go	Utility to dump the BPF maps for debugging.

Summary

• CubeVS implements kernel-level network isolation using three eBPF programs (nodenic, localgw, mvmtap) attached to the Linux TC subsystem.
• The architecture relies on high-performance BPF maps including LPM-Tries for CIDR matching and hash maps for DNS and metadata storage.
• Packet processing occurs entirely in kernel space, delivering sub-microsecond latency while preventing bypass by privileged containers.
• The Go-based control plane in cubevs.go manages device lifecycle and policy enforcement through helpers like UpsertTAPDeviceMeta and initNetPolicy.

Frequently Asked Questions

How does CubeVS differ from traditional OVS or iptables-based isolation?

Traditional solutions rely on userspace packet processing or netfilter hooks that incur context-switch overhead and can be bypassed with sufficient privileges. CubeVS operates entirely within the kernel's TC subsystem using eBPF, eliminating context switches and providing tamper-proof enforcement that persists even inside privileged containers.

What is the purpose of the LPM-Trie maps in CubeVS network policies?

The allow_out_v2 and deny_out maps use LPM-Trie (Longest Prefix Match) inner maps created by newInnerLPMMap in netpolicy.go. This structure enables efficient O(log n) lookups for CIDR-based rules, allowing the eBPF program to rapidly determine whether a destination IP falls within an allowed or denied network range while maintaining minimal kernel memory overhead.

Can CubeVS handle DNS-based filtering for egress control?

Yes. CubeVS implements DNS filtering through the dns_allow hash-of-maps structure and dedicated tail-call programs (dns_parse_chunk, dns_rev_chunk). When enabled, the eBPF programs hash queried domain names and verify them against the per-sandbox allow list before permitting the DNS request to leave the sandbox.

How are the eBPF programs loaded and attached to network interfaces?

The control plane uses bpf2go (invoked via //go:generate directives in cubevs.go) to compile C source files into BPF bytecode. The Go runtime then loads these programs using constants like tcHandleClsact and tcFilterHandle (lines 84-89 of cubevs.go), pins them to /sys/fs/bpf, and attaches them to clsact qdiscs on the respective interfaces.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}