How CubeSandbox Implements Kernel-Level Network Isolation with CubeVS eBPF Virtual Switch

CubeSandbox achieves kernel-level network isolation by attaching eBPF programs to the Linux TC subsystem at three strategic hook points (nodenic, localgw, and mvmtap), which enforce CIDR-based policies, DNS filtering, and SNAT rules entirely in kernel space before packets reach the sandbox network interface.

CubeSandbox leverages the CubeVS eBPF virtual switch to provide high-performance, kernel-level network isolation for sandboxed workloads. This architecture eliminates context-switch overhead by processing packets directly in the kernel using maps and programs compiled with bpf2go. The implementation ensures that sandboxed containers cannot bypass network policies even when running with elevated privileges.

CubeVS eBPF Architecture

TC Hook Points and Program Attachment

CubeVS implements its data plane using three specialized eBPF programs generated at build time via //go:generate directives in cubevs.go. These programs attach to the Linux TC (traffic-control) subsystem using the clsact qdisc:

• nodenic.bpf.c – Handles bidirectional traffic between the host node's physical NIC and the sandbox-side virtual NIC (cubegw0).
• localgw.bpf.c – Processes egress packets leaving the sandbox (world-side processing).
• mvmtap.bpf.c – Processes ingress packets entering the sandbox (in-gateway processing).

Each program is pinned under /sys/fs/bpf after loading and attached using TC constants defined in cubevs.go (lines 84‑89), specifically tcHandleClsact and tcFilterHandle. This attachment ensures that all packets traversing the network interfaces are intercepted and processed by eBPF before reaching user space.

eBPF Maps for Network Isolation

The CubeVS control plane manages several BPF map types that store per-sandbox metadata and policies:

• ifindex_to_mvmmeta – Stores per-TAP-device metadata including IP addresses, UUIDs, and version information.
• mvmip_to_ifindex – Provides fast reverse lookup mapping sandbox IPs to their corresponding TAP interface indices.
• remote_port_mapping and local_port_mapping – Implement static port-NAT for external services, handling outbound and inbound traffic respectively.
• allow_out_v2 and deny_out – LPM-Trie (Longest Prefix Match) structures that store CIDR-based egress policies. The allow_out_v2 map uses netPolicyValueV2 values supporting L7-aware flags such as netPolicyFlagL7Required.
• dns_allow – A hash-of-maps structure storing hashed DNS names permitted for each sandbox, checked by DNS tail-call programs (dns_parse_chunk, dns_rev_chunk).
• snat_iplist – Maintains a configurable pool of IPs used for source-NAT of outbound traffic.

These maps are created and managed by the Go helper library in the cubevs package. For example, UpsertTAPDeviceMeta updates the metadata and reverse IP maps (see tap.go lines 58‑100), while initNetPolicy creates per-sandbox policy maps on-demand (see netpolicy.go lines 86‑103).

Packet Flow and Data Plane

Ingress Path (Sandbox to Host)

When a sandbox transmits packets, the traffic is received on the TAP device and passed to the mvmtap eBPF program. The execution flow follows these steps:

1. The program reads the ifindex_to_mvmmeta map to identify the source sandbox using the interface index.
2. It applies egress policies by consulting the allow_out_v2 or deny_out LPM-Trie maps for CIDR matching.
3. After policy validation, the program rewrites L2 headers to the host MAC address.
4. Finally, it redirects the packet to the world using BPFRedirectFlagIngress.

This entire path executes in kernel space, avoiding context switches and delivering sub-microsecond latency for intra-node traffic.

Egress Path (Host to Sandbox)

Outbound packets from external sources are intercepted by the localgw eBPF program attached to the cubegw0 interface:

1. The program performs SNAT using the snat_iplist map to ensure traffic originates from a controlled address range.
2. It consults allow_out_v2 or deny_out maps for policy enforcement based on destination CIDRs.
3. Using the mvmip_to_ifindex map, it identifies the target TAP device for the destination IP.
4. The packet is redirected to the appropriate TAP device, completing the delivery to the sandbox.

Node-NIC Bridging

The nodenic program provides bidirectional forwarding between the host node's physical NIC and the sandbox NIC. It handles L2 header rewriting and optional DNS filtering, ensuring that traffic between the host and sandboxes respects the same isolation policies enforced by the other hooks.

Implementing Security Policies

CIDR-Based Egress Control

Network policies are enforced using LPM-Trie maps that enable efficient longest-prefix matching for CIDR blocks. The initNetPolicy function in netpolicy.go (lines 86‑103) creates these maps on-demand for each sandbox interface. Inner maps are allocated using newInnerLPMMap (lines 25‑33), which configures the trie structure for high-performance lookups. The allow_out_v2 map supports both whitelist and blacklist semantics, with optional L7 inspection flags.

DNS Filtering

DNS-based isolation is implemented through the dns_allow hash-of-maps. The eBPF programs store hashed DNS names using dnsAllowKey and dnsAllowValue structures. Dedicated tail-call programs (dns_parse_chunk, dns_rev_chunk) parse DNS packets and validate queries against the allowed list, preventing sandboxes from resolving unauthorized domains.

SNAT Pool Management

The snat_iplist map maintains a small pool of IP addresses used for source-NAT of outbound traffic. This mechanism guarantees that egress traffic appears to originate from a controlled address range, preventing IP spoofing across sandboxes and ensuring that external services cannot directly address individual sandbox IPs.

Managing TAP Devices and Policies

The cubevs package provides Go APIs for managing sandbox network interfaces and their associated policies. The following examples demonstrate how to create, list, and remove TAP devices with specific egress policies.

To create a new sandbox TAP device and configure its egress policies:

params := cubevs.Params{
    MVMInnerIP: net.ParseIP("10.1.2.3"),
    MVMMacAddr: net.HardwareAddr{0x02, 0x42, 0xac, 0x11, 0x00, 0x02},
    // … other required fields …
}
vs, _ := cubevs.New(params)               // initialize CubeVS instance
opts := cubevs.MVMOptions{
    AllowOut:            &[]string{"0.0.0.0/0"},
    L7AllowOut:          &[]string{"10.0.0.0/8"},
    DenyOut:             &[]string{"192.168.0.0/16"},
}
_ = vs.AddTAPDevice(5, net.ParseIP("10.1.2.3"), "sandbox-01", 1, opts)

To list all TAP devices currently managed by CubeVS:

taps, _ := cubevs.ListTAPDevices()
for _, t := range taps {
    fmt.Printf("TAP %s – IP %s – ifindex %d\n", t.ID, t.IP, t.Ifindex)
}

To remove a sandbox TAP device and clean up its associated policy maps:

_ = cubevs.DelTAPDevice(5, net.ParseIP("10.1.2.3"))

These operations update the underlying eBPF maps—including ifindex_to_mvmmeta, mvmip_to_ifindex, and the policy tries—ensuring that the kernel data plane remains synchronized with the control plane state.

Key Source Files

• cubevs.go – Central package definition, map name constants, and TC attachment logic.
• tap.go – TAP device lifecycle management and metadata handling.
• netpolicy.go – LPM-Trie map creation, CIDR parsing, and policy lifecycle management.
• snat.go – SNAT IP pool management and map updates.
• miscs.go – Helper functions for loading and pinning eBPF objects, constant rewriting, and tail-call setup.
• src/localgw.bpf.c, src/mvmtap.bpf.c, src/nodenic.bpf.c – The kernel-space eBPF programs compiled to BPF bytecode.
• CubeNet/cubevs/cmd/cubevsmapdump/main.go – Debugging utility for inspecting BPF map contents.

Summary

• CubeVS attaches three eBPF programs (nodenic, localgw, mvmtap) to the TC subsystem for comprehensive packet interception.
• Network isolation is enforced using LPM-Trie maps for CIDR-based egress policies and hash maps for DNS filtering and port NAT.
• All packet processing occurs in kernel space, eliminating context-switch overhead and preventing privilege escalation bypasses.
• The Go control plane in the cubevs package manages BPF maps through APIs like UpsertTAPDeviceMeta and initNetPolicy.
• SNAT pools and per-sandbox IP-to-ifindex mappings prevent IP spoofing and ensure controlled egress addressing.

Frequently Asked Questions

How does CubeVS prevent sandboxed workloads from bypassing network policies?

CubeVS enforces policies through eBPF programs attached to the TC subsystem, which intercept packets before they reach the sandbox network interface. Because this enforcement occurs in kernel space, sandboxed containers—including those running with elevated privileges or in privileged mode—cannot bypass the allow_out_v2, deny_out, or dns_allow map lookups. The policies are evaluated for every packet before redirection to the TAP device or physical NIC.

What eBPF map types does CubeVS use for policy enforcement?

CubeVS utilizes LPM-Trie (Longest Prefix Match) maps for CIDR-based egress policies, enabling efficient matching of IP addresses against network ranges. For DNS filtering, it uses hash-of-maps structures. Per-device metadata is stored in hash maps, while port mappings use standard hash maps for NAT translations. These map types are created and managed through functions like newInnerLPMMap in netpolicy.go.

How does CubeVS handle DNS-based filtering?

DNS filtering is implemented through a dedicated dns_allow hash map that stores hashed DNS names permitted for each sandbox. When DNS packets traverse the CubeVS data plane, tail-call programs (dns_parse_chunk, dns_rev_chunk) parse the queries and validate them against the sandbox's allowed list. Unauthorized DNS queries are dropped before reaching external resolvers, preventing data exfiltration via DNS tunneling.

What is the performance impact of using eBPF for network isolation?

Because CubeVS processes packets entirely in kernel space using eBPF maps and TC hooks, it eliminates the context-switch overhead associated with userspace packet filtering. The architecture achieves sub-microsecond latency for intra-node traffic by using efficient LPM-Trie lookups for CIDR matching and direct packet redirection via BPFRedirectFlagIngress, making it suitable for high-throughput sandboxed workloads.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}