How CubeVS Achieves Kernel-Level Isolation with eBPF-Based Network Virtualization

CubeVS achieves kernel-level isolation by executing three tightly-coupled eBPF programs entirely within the Linux kernel that handle all packet processing, NAT, ACL enforcement, and session tracking without userspace involvement, ensuring each sandbox operates in isolation through dedicated TAP devices and pinned BPF maps.

CubeVS is the high-performance networking stack powering Tencent Cloud's CubeSandbox. Unlike traditional container networking that relies on userspace proxies or bridge-based switching, CubeVS implements complete per-sandbox virtual networks entirely inside the kernel using eBPF. This architecture eliminates context switches, removes userspace packet processing from the data path, and provides hardened isolation for multi-tenant workloads.

Core Architecture

CubeVS implements its network virtualization through three specialized eBPF programs that attach to the only points where traffic can cross a sandbox boundary. A Go control-plane library manages the lifecycle of these programs and their associated state maps.

The Three eBPF Data-Path Programs

The data-plane consists of three BPF programs written in C that attach to specific Traffic Control (TC) hooks:

• from_cube (located in CubeNet/src/mvmtap.bpf.c): Attaches to TC-ingress on each sandbox TAP device. This program handles ARP responses, enforces outbound ACLs, performs SNAT, manages session state, and redirects packets to the host NIC.

• from_world (located in CubeNet/src/nodenic.bpf.c): Attaches to TC-ingress on the host NIC. This program processes inbound traffic, performs reverse NAT using the session tables, and handles port-mapping for external access.

• from_envoy (located in CubeNet/src/localgw.bpf.c): Attaches to TC-egress on the cube-dev overlay device. This program DNATs traffic from the host-side proxy (Envoy) to the appropriate sandbox TAP.

These programs perform SNAT/DNAT, policy enforcement, L7-proxy selection, and session creation without ever leaving kernel space.

BPF Maps and Shared State

Nine pinned BPF maps under /sys/fs/bpf/ provide shared state between the three programs and the Go control plane:

• Device mapping: mvmip_to_ifindex and ifindex_to_mvmmeta map IP addresses to network interface indices.
• Session tables: egress_sessions and ingress_sessions store bidirectional connection state.
• SNAT pool: snat_iplist manages available source IP addresses for NAT.
• Policy enforcement: allow_out and deny_out are per-sandbox LPM-Trie structures that store ACL rules.
• Port mapping: remote_port_mapping and local_port_mapping handle port translation for external access.

Because these maps are pinned in the BPF filesystem, any program loaded later can instantly read or write the same data structures, creating a single source of truth that prevents race conditions.

Go Control Plane

The cubevs/ Go package drives the initialization and lifecycle management:

• Init(): Loads and pins the three eBPF object files, rewrites compile-time constants (IP, MAC, interface indices), and attaches programs to the correct TC hooks.
• AddTAPDevice() and DelTAPDevice(): Register and deregister sandbox TAP interfaces, populating device-IP maps and creating per-sandbox LPM-Trie ACLs.
• AttachFilter(): Creates a clsact qdisc on the TAP and attaches the from_cube program.
• SetSNATIPs(): Fills the SNAT pool map with available IP addresses.
• Background reaper: A goroutine periodically walks egress_sessions to clean up expired connections.

All these actions occur in userspace once during setup, after which the data-plane remains entirely in kernel.

How Kernel-Level Isolation Works

CubeVS provides kernel-level isolation through five complementary mechanisms that ensure sandboxes cannot bypass network policies or intercept traffic from other tenants.

Per-Sandbox TAP Devices

Each sandbox receives its own dedicated TAP interface. There is no shared bridge or software switch that aggregates traffic. The eBPF filter attached to each TAP (from_cube) processes only that sandbox's packets, ensuring that state (session tables and ACLs) is naturally isolated by interface index. This design prevents traffic leakage at the device level before any packet processing occurs.

In-Kernel Policy Evaluation

Outbound network policies are stored as per-sandbox LPM-Trie maps (allow_out and deny_out). The from_cube program evaluates these policy tries for every packet before performing NAT. Because policy decisions occur inside the kernel's BPF virtual machine, a malicious sandbox cannot bypass the rule set without exploiting a kernel vulnerability. This contrasts with iptables-based solutions where userspace processes might modify rules.

Stateful Connection Tracking

Bidirectional session state is maintained in two maps: egress_sessions for outbound connections and ingress_sessions for inbound replies. Reverse NAT for reply packets is performed by looking up the session in ingress_sessions inside from_world. Because sessions are scoped to a specific TAP's ifindex, cross-sandbox leakage is impossible—even if two sandboxes use identical internal IP addresses, the interface index distinguishes their sessions.

Zero Userspace Packet Processing

Traditional CNI plugins often forward packets to userspace proxies or daemons for policy enforcement. CubeVS implements NAT, ACL evaluation, L7-proxy selection, and ARP handling entirely within the three BPF programs. This eliminates context switches to userspace for the data path, removing a large attack surface that typical container networking solutions expose. Packets never leave the kernel between the sandbox TAP and the host NIC.

Atomic State via Pinned Maps

All eBPF programs share the same pinned maps in /sys/fs/bpf/, creating a single source of truth for NAT and ACL state. This design avoids synchronization race conditions that could arise from multiple independent programs trying to coordinate via userspace. When the Go control plane updates a policy or NAT entry, the change is immediately visible to all kernel-side programs without reloading or recompilation.

Implementation Examples

Setting Up a Sandbox Network with Go

The following example demonstrates how to initialize CubeVS and attach a sandbox TAP device:

import "github.com/TencentCloud/CubeSandbox/network-agent/cubevs"

func createSandbox(ifIdx int, ip string, id string) error {
    // Initialise CubeVS (once per host)
    if err := cubevs.Init(); err != nil {
        return err
    }

    // Register the TAP device and its metadata
    opts := cubevs.MVMOptions{
        AllowInternetAccess: false,      // deny all by default
        AllowOut:            []string{}, // add whitelist CIDRs if needed
        DenyOut:             []string{}, // add blacklist CIDRs if needed
    }
    if err := cubevs.AddTAPDevice(ifIdx, ip, id, "v1", opts); err != nil {
        return err
    }

    // Attach the per-TAP TC filter (from_cube)
    return cubevs.AttachFilter(ifIdx)
}

Packet Processing in from_cube

The from_cube program in CubeNet/src/mvmtap.bpf.c implements the core isolation logic:

/* mvmtap.bpf.c – core of the from_cube program (simplified) */
SEC("tc")
int from_cube(struct __sk_buff *skb) {
    // 1️⃣ ARP handling – reply to gateway ARP requests
    if (is_arp_request(skb)) {
        arp_reply(skb);
        return TC_ACT_OK;
    }

    // 2️⃣ Policy check – lookup per-TAP allow/deny LPM Tries
    if (!policy_allowed(skb)) {
        return TC_ACT_SHOT;   // drop packet inside kernel
    }

    // 3️⃣ NAT session handling – create / update egress_sessions map
    struct session_key key = build_key(skb);
    struct session_val *val = bpf_map_lookup_elem(&egress_sessions, &key);
    if (!val) {
        // allocate SNAT IP/port, store reverse entry in ingress_sessions
        allocate_snat(&key, &val);
        bpf_map_update_elem(&egress_sessions, &key, val, BPF_ANY);
    }

    // 4️⃣ SNAT rewrite – modify source IP/port, recalc checksums
    snat_rewrite(skb, val);

    // 5️⃣ Redirect packet to host NIC (eth0)
    return bpf_redirect(ETH_IFINDEX, 0);
}

Summary

• CubeVS implements network virtualization entirely inside the Linux kernel using three eBPF programs (from_cube, from_world, from_envoy) that attach to TC hooks on sandbox TAP devices, host NICs, and overlay devices.
• Kernel-level isolation is achieved through dedicated TAP devices per sandbox, in-kernel policy evaluation using LPM-Trie maps (allow_out/deny_out), and session tracking scoped to interface indices (egress_sessions/ingress_sessions).
• Zero userspace data-plane eliminates context switches and attack surfaces by performing NAT, ACL enforcement, and L7-proxy selection entirely in eBPF without packets leaving kernel space.
• Pinned BPF maps under /sys/fs/bpf/ provide atomic, race-free state synchronization between the Go control plane and the eBPF data-path programs.
• Source files defining this behavior include CubeNet/src/mvmtap.bpf.c, CubeNet/src/nodenic.bpf.c, and the network-agent/cubevs/ Go package.

Frequently Asked Questions

How does CubeVS differ from traditional CNI plugins like Flannel or Calico?

Traditional CNI plugins typically rely on Linux bridges, iptables rules, or userspace proxies (Envoy, sidecars) for packet forwarding and policy enforcement. CubeVS moves all packet processing—NAT, ACLs, session tracking, and L7-proxy selection—into eBPF programs that run entirely in kernel space. According to the CubeSandbox source code, this eliminates context switches and provides per-sandbox isolation through dedicated TAP devices rather than shared bridges, resulting in lower latency and stronger security boundaries.

Why are the BPF maps pinned to /sys/fs/bpf/ instead of kept private?

Pinning maps to the BPF filesystem allows multiple eBPF programs to share the same data structures across different attachment points. In CubeVS, the egress_sessions, ingress_sessions, and policy maps must be accessible by from_cube, from_world, and the Go control plane simultaneously. By pinning these maps, CubeVS ensures a single source of truth for NAT and ACL state, preventing race conditions that could occur if each program maintained separate copies synchronized through userspace.

How does CubeVS prevent cross-sandbox traffic leakage?

Cross-sandbox leakage is prevented through three mechanisms: (1) each sandbox has its own dedicated TAP device with no shared bridge, (2) session tables are keyed by interface index (ifindex) ensuring that even if two sandboxes use identical internal IPs, their sessions remain distinct, and (3) the from_cube program evaluates outbound policies using per-sandbox LPM-Trie maps before any NAT occurs, guaranteeing that a compromised sandbox cannot inject packets into another sandbox's network without passing through the kernel-enforced policy engine.

What happens when a network policy changes at runtime?

When policies change, the Go control plane updates the per-sandbox allow_out or deny_out LPM-Trie maps in /sys/fs/bpf/ using the cubevs.AddTAPDevice() or equivalent update functions. Because these maps are pinned and shared, the change is immediately effective for all packets processed by the from_cube program without requiring program reload or service restart. The eBPF data-path sees updated policy rules atomically on the next packet processed for that sandbox.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}