CubeVS eBPF Virtual Switch Implementation: Kernel-Level Network Isolation in CubeSandbox

CubeVS is a high-performance eBPF virtual switch that isolates sandboxed workloads at the kernel level by attaching TC hook programs to network interfaces and enforcing fine-grained policies through LPM-Trie maps pinned under /sys/fs/bpf.

CubeVS powers the network isolation layer of TencentCloud's CubeSandbox, providing sub-microsecond latency packet processing without userspace context switches. This eBPF virtual switch implementation intercepts traffic at three strategic TC hook points to enforce per-sandbox CIDR policies, DNS filtering, and SNAT rules entirely within the kernel.

Architecture and Hook Points

The CubeVS eBPF virtual switch attaches to the Linux TC (traffic control) subsystem using the clsact qdisc. According to the source code in CubeNet/cubevs/cubevs.go, the implementation uses constants such as tcHandleClsact and tcFilterHandle (lines 84‑89) to manage attachment points.

Three specialized eBPF programs handle different traffic vectors:

• nodenic.bpf.c – Handles bidirectional traffic between the host node's physical NIC and the sandbox-side virtual NIC (cubegw0).
• localgw.bpf.c – Processes egress packets leaving the sandbox (world-side forwarding).
• mvmtap.bpf.c – Handles ingress packets entering the sandbox (in-gateway processing).

These programs are generated at build time using bpf2go, as indicated by the //go:generate directives in CubeNet/cubevs/cubevs.go. After compilation, each program is pinned under /sys/fs/bpf for persistent access across process restarts.

Data Plane and eBPF Maps

The CubeVS data plane relies on several high-performance BPF maps to maintain state and enforce policies. These maps are created and managed by the Go helper library in the cubevs package.

Metadata and Identity Maps

• ifindex_to_mvmmeta – Stores per-TAP-device metadata including IP addresses, UUIDs, and version information. The function UpsertTAPDeviceMeta in CubeNet/cubevs/tap.go (lines 58‑100) updates this map alongside the reverse IP mapping.
• mvmip_to_ifindex – Provides fast reverse lookup from sandbox IPs to their corresponding TAP interface indices.

Network Policy Maps

• allow_out_v2 – An LPM-Trie (longest prefix match) containing netPolicyValueV2 structures that whitelist egress destinations with optional L7 awareness.
• deny_out – An LPM-Trie of uint32 values implementing blacklist semantics for egress traffic.
• dns_allow – A hash-of-maps structure storing allowed DNS names per sandbox, checked by DNS tail-call programs (dns_parse_chunk, dns_rev_chunk).

Translation and NAT Maps

• remote_port_mapping and local_port_mapping – Implement static port-NAT for external services.
• snat_iplist – Maintains a configurable pool of IPs used for source-NAT of outbound traffic.

The network policy maps are created on-demand per-sandbox interface by initNetPolicy (see CubeNet/cubevs/netpolicy.go lines 86‑103). Inner maps use LPM-Trie structures created via newInnerLPMMap (lines 25‑33) to enable efficient CIDR matching.

Packet Flow and Processing Logic

CubeVS processes packets entirely in kernel space through three distinct flows:

Ingress Path (Sandbox → Host)

Packets received on the TAP device enter the mvmtap eBPF program. The program reads the ifindex_to_mvmmeta map to identify the sandbox context, consults the allow_out_v2 or deny_out LPM-Trie maps for egress policy enforcement, and rewrites L2 headers to the host MAC before redirecting the packet to the world using BPFRedirectFlagIngress.

Egress Path (Host → Sandbox)

Outbound packets are intercepted by the localgw program attached to the cubegw0 interface. The program performs SNAT using the snat_iplist map, validates traffic against egress policies, and redirects to the appropriate TAP device identified via the mvmip_to_ifindex map.

Node-NIC Path

The nodenic program provides bidirectional forwarding between the host node's physical NIC and sandbox NICs, handling L2 rewrite and optional DNS filtering at the entry point.

Isolation Guarantees and Security Model

CubeVS provides hardened isolation through several kernel-enforced mechanisms:

• Per-Sandbox IP Isolation – Each sandbox receives a unique inner IP (MVMInnerIP) mapped to its TAP ifindex; the eBPF maps ensure packets can only reach the TAP owning the destination IP.
• CIDR-Based Egress Control – LPM-Trie maps implement whitelist/blacklist semantics with optional L7 flags (netPolicyFlagL7Required) for fine-grained outbound filtering.
• DNS Filtering – Hashed DNS names stored in the dns_allow map prevent sandboxed workloads from resolving unauthorized domains.
• SNAT Pool Enforcement – A dedicated pool of SNAT IPs prevents IP spoofing and ensures egress traffic originates from controlled address ranges.

Because policies are enforced by eBPF before packets exit the kernel, privileged containers cannot bypass these rules.

Working with CubeVS

The cubevs package provides Go APIs to manage sandbox network interfaces and policies.

Creating a Sandbox TAP Device

// Initialize CubeVS instance with sandbox parameters
params := cubevs.Params{
    MVMInnerIP: net.ParseIP("10.1.2.3"),
    MVMMacAddr: net.HardwareAddr{0x02, 0x42, 0xac, 0x11, 0x00, 0x02},
    // ... other required fields ...
}
vs, _ := cubevs.New(params)

// Define egress policies with CIDR-based allow/deny rules
opts := cubevs.MVMOptions{
    AllowOut:   &[]string{"0.0.0.0/0"},
    L7AllowOut: &[]string{"10.0.0.0/8"},
    DenyOut:    &[]string{"192.168.0.0/16"},
}

// Add TAP device with index 5
_ = vs.AddTAPDevice(5, net.ParseIP("10.1.2.3"), "sandbox-01", 1, opts)

Listing Managed TAP Devices

// Retrieve all TAP devices managed by CubeVS
taps, _ := cubevs.ListTAPDevices()
for _, t := range taps {
    fmt.Printf("TAP %s – IP %s – ifindex %d\n", t.ID, t.IP, t.Ifindex)
}

Removing a Sandbox Interface

// Delete TAP device and clean up associated policy maps
_ = cubevs.DelTAPDevice(5, net.ParseIP("10.1.2.3"))

Key Source Files

The implementation spans several critical files in the CubeNet/cubevs directory:

File	Description
cubevs.go	Central package definition, constant declarations (tcHandleClsact, tcFilterHandle), and map names.
tap.go	TAP-device lifecycle management including UpsertTAPDeviceMeta for metadata handling.
netpolicy.go	LPM-Trie map creation, inner-map management, and CIDR parsing via initNetPolicy and newInnerLPMMap.
snat.go	SNAT IP pool management and map updates.
miscs.go	Helper functions for loading/pinning eBPF objects and tail-call setup.
src/localgw.bpf.c, src/mvmtap.bpf.c, src/nodenic.bpf.c	The actual eBPF C programs compiled to BPF bytecode.
cmd/cubevsmapdump/main.go	Utility for dumping BPF maps for debugging.

Summary

• CubeVS implements kernel-level network isolation using eBPF programs attached to the TC subsystem at three hook points (nodenic, localgw, mvmtap).
• Network policies are enforced through LPM-Trie maps (allow_out_v2, deny_out) that support CIDR-based matching with optional L7 filtering.
• The data plane uses high-performance BPF maps for metadata (ifindex_to_mvmmeta), IP-to-interface mapping (mvmip_to_ifindex), and SNAT pools (snat_iplist).
• All packet processing occurs in kernel space via CubeNet/cubevs Go bindings, eliminating context-switch overhead and delivering sub-microsecond latency.
• Isolation guarantees include per-sandbox IP restrictions, DNS filtering through dns_allow maps, and IP anti-spoofing via dedicated SNAT pools.

Frequently Asked Questions

What is CubeVS and how does it differ from traditional virtual switches?

CubeVS is an eBPF-based virtual switch built into TencentCloud's CubeSandbox that operates at the kernel level rather than in userspace. Unlike traditional OVS or Linux bridge implementations that require packets to traverse the network stack and context-switch to a daemon, CubeVS programs attach directly to the TC subsystem and process packets within the kernel, resulting in sub-microsecond latency and stronger isolation guarantees.

How does CubeVS enforce network policies at the kernel level?

CubeVS enforces policies using LPM-Trie (longest prefix match) BPF maps created by initNetPolicy in netpolicy.go. When a packet traverses the mvmtap or localgw programs, the eBPF code performs lookups in allow_out_v2 or deny_out maps to determine if the destination CIDR is permitted. These checks occur before the packet reaches the host network stack, preventing bypass attempts even from privileged containers.

What are the performance benefits of using eBPF for network isolation in CubeSandbox?

The CubeVS eBPF virtual switch eliminates userspace copies and context switches by processing traffic entirely within the kernel's TC hook points. According to the implementation in src/mvmtap.bpf.c and src/localgw.bpf.c, packets are redirected using BPFRedirectFlagIngress and direct map lookups, achieving sub-microsecond latency for intra-node traffic while maintaining CIDR-based policy enforcement.

How are the eBPF programs loaded and managed in CubeVS?

The eBPF programs are compiled using bpf2go via //go:generate directives in cubevs.go, then loaded and pinned under /sys/fs/bpf by the Go helper library. The miscs.go file handles object loading and pinning, while tap.go manages the lifecycle of TAP devices and their associated maps. This architecture ensures that eBPF resources persist independently of the userspace process lifecycle.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}