# How CubeSandbox Implements Kernel-Level Network Isolation with CubeVS eBPF Virtual Switch

> Discover how CubeSandbox uses CubeVS eBPF virtual switch and Linux TC subsystem for kernel-level network isolation. Enforce CIDR, DNS, and SNAT rules efficiently in kernel space.

- Repository: [Tencent Cloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox)
- Tags: how-to-guide
- Published: 2026-07-16

---

**CubeSandbox achieves kernel-level network isolation by attaching eBPF programs to the Linux TC subsystem at three strategic hook points (nodenic, localgw, and mvmtap), which enforce CIDR-based policies, DNS filtering, and SNAT rules entirely in kernel space before packets reach the sandbox network interface.**

CubeSandbox leverages the CubeVS eBPF virtual switch to provide high-performance, kernel-level network isolation for sandboxed workloads. This architecture eliminates context-switch overhead by processing packets directly in the kernel using maps and programs compiled with **bpf2go**. The implementation ensures that sandboxed containers cannot bypass network policies even when running with elevated privileges.

## CubeVS eBPF Architecture

### TC Hook Points and Program Attachment

CubeVS implements its data plane using three specialized eBPF programs generated at build time via `//go:generate` directives in **[`cubevs.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/cubevs.go)**. These programs attach to the Linux **TC** (traffic-control) subsystem using the `clsact` qdisc:

- **[`nodenic.bpf.c`](https://github.com/TencentCloud/CubeSandbox/blob/main/nodenic.bpf.c)** – Handles bidirectional traffic between the host node's physical NIC and the sandbox-side virtual NIC (`cubegw0`).
- **[`localgw.bpf.c`](https://github.com/TencentCloud/CubeSandbox/blob/main/localgw.bpf.c)** – Processes egress packets leaving the sandbox (world-side processing).
- **[`mvmtap.bpf.c`](https://github.com/TencentCloud/CubeSandbox/blob/main/mvmtap.bpf.c)** – Processes ingress packets entering the sandbox (in-gateway processing).

Each program is pinned under **`/sys/fs/bpf`** after loading and attached using TC constants defined in [`cubevs.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/cubevs.go) (lines 84‑89), specifically `tcHandleClsact` and `tcFilterHandle`. This attachment ensures that all packets traversing the network interfaces are intercepted and processed by eBPF before reaching user space.

### eBPF Maps for Network Isolation

The CubeVS control plane manages several BPF map types that store per-sandbox metadata and policies:

- **`ifindex_to_mvmmeta`** – Stores per-TAP-device metadata including IP addresses, UUIDs, and version information.
- **`mvmip_to_ifindex`** – Provides fast reverse lookup mapping sandbox IPs to their corresponding TAP interface indices.
- **`remote_port_mapping`** and **`local_port_mapping`** – Implement static port-NAT for external services, handling outbound and inbound traffic respectively.
- **`allow_out_v2`** and **`deny_out`** – LPM-Trie (Longest Prefix Match) structures that store CIDR-based egress policies. The `allow_out_v2` map uses `netPolicyValueV2` values supporting L7-aware flags such as `netPolicyFlagL7Required`.
- **`dns_allow`** – A hash-of-maps structure storing hashed DNS names permitted for each sandbox, checked by DNS tail-call programs (`dns_parse_chunk`, `dns_rev_chunk`).
- **`snat_iplist`** – Maintains a configurable pool of IPs used for source-NAT of outbound traffic.

These maps are created and managed by the Go helper library in the **cubevs** package. For example, `UpsertTAPDeviceMeta` updates the metadata and reverse IP maps (see **[`tap.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/tap.go)** lines 58‑100), while `initNetPolicy` creates per-sandbox policy maps on-demand (see **[`netpolicy.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/netpolicy.go)** lines 86‑103).

## Packet Flow and Data Plane

### Ingress Path (Sandbox to Host)

When a sandbox transmits packets, the traffic is received on the TAP device and passed to the **`mvmtap`** eBPF program. The execution flow follows these steps:

1. The program reads the **`ifindex_to_mvmmeta`** map to identify the source sandbox using the interface index.
2. It applies egress policies by consulting the **`allow_out_v2`** or **`deny_out`** LPM-Trie maps for CIDR matching.
3. After policy validation, the program rewrites L2 headers to the host MAC address.
4. Finally, it redirects the packet to the world using `BPFRedirectFlagIngress`.

This entire path executes in kernel space, avoiding context switches and delivering sub-microsecond latency for intra-node traffic.

### Egress Path (Host to Sandbox)

Outbound packets from external sources are intercepted by the **`localgw`** eBPF program attached to the `cubegw0` interface:

1. The program performs SNAT using the **`snat_iplist`** map to ensure traffic originates from a controlled address range.
2. It consults **`allow_out_v2`** or **`deny_out`** maps for policy enforcement based on destination CIDRs.
3. Using the **`mvmip_to_ifindex`** map, it identifies the target TAP device for the destination IP.
4. The packet is redirected to the appropriate TAP device, completing the delivery to the sandbox.

### Node-NIC Bridging

The **`nodenic`** program provides bidirectional forwarding between the host node's physical NIC and the sandbox NIC. It handles L2 header rewriting and optional DNS filtering, ensuring that traffic between the host and sandboxes respects the same isolation policies enforced by the other hooks.

## Implementing Security Policies

### CIDR-Based Egress Control

Network policies are enforced using LPM-Trie maps that enable efficient longest-prefix matching for CIDR blocks. The `initNetPolicy` function in **[`netpolicy.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/netpolicy.go)** (lines 86‑103) creates these maps on-demand for each sandbox interface. Inner maps are allocated using `newInnerLPMMap` (lines 25‑33), which configures the trie structure for high-performance lookups. The `allow_out_v2` map supports both whitelist and blacklist semantics, with optional L7 inspection flags.

### DNS Filtering

DNS-based isolation is implemented through the **`dns_allow`** hash-of-maps. The eBPF programs store hashed DNS names using `dnsAllowKey` and `dnsAllowValue` structures. Dedicated tail-call programs (`dns_parse_chunk`, `dns_rev_chunk`) parse DNS packets and validate queries against the allowed list, preventing sandboxes from resolving unauthorized domains.

### SNAT Pool Management

The **`snat_iplist`** map maintains a small pool of IP addresses used for source-NAT of outbound traffic. This mechanism guarantees that egress traffic appears to originate from a controlled address range, preventing IP spoofing across sandboxes and ensuring that external services cannot directly address individual sandbox IPs.

## Managing TAP Devices and Policies

The **cubevs** package provides Go APIs for managing sandbox network interfaces and their associated policies. The following examples demonstrate how to create, list, and remove TAP devices with specific egress policies.

To create a new sandbox TAP device and configure its egress policies:

```go
params := cubevs.Params{
    MVMInnerIP: net.ParseIP("10.1.2.3"),
    MVMMacAddr: net.HardwareAddr{0x02, 0x42, 0xac, 0x11, 0x00, 0x02},
    // … other required fields …
}
vs, _ := cubevs.New(params)               // initialize CubeVS instance
opts := cubevs.MVMOptions{
    AllowOut:            &[]string{"0.0.0.0/0"},
    L7AllowOut:          &[]string{"10.0.0.0/8"},
    DenyOut:             &[]string{"192.168.0.0/16"},
}
_ = vs.AddTAPDevice(5, net.ParseIP("10.1.2.3"), "sandbox-01", 1, opts)

```

To list all TAP devices currently managed by CubeVS:

```go
taps, _ := cubevs.ListTAPDevices()
for _, t := range taps {
    fmt.Printf("TAP %s – IP %s – ifindex %d\n", t.ID, t.IP, t.Ifindex)
}

```

To remove a sandbox TAP device and clean up its associated policy maps:

```go
_ = cubevs.DelTAPDevice(5, net.ParseIP("10.1.2.3"))

```

These operations update the underlying eBPF maps—including `ifindex_to_mvmmeta`, `mvmip_to_ifindex`, and the policy tries—ensuring that the kernel data plane remains synchronized with the control plane state.

## Key Source Files

- **[`cubevs.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/cubevs.go)** – Central package definition, map name constants, and TC attachment logic.
- **[`tap.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/tap.go)** – TAP device lifecycle management and metadata handling.
- **[`netpolicy.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/netpolicy.go)** – LPM-Trie map creation, CIDR parsing, and policy lifecycle management.
- **[`snat.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/snat.go)** – SNAT IP pool management and map updates.
- **[`miscs.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/miscs.go)** – Helper functions for loading and pinning eBPF objects, constant rewriting, and tail-call setup.
- **[`src/localgw.bpf.c`](https://github.com/TencentCloud/CubeSandbox/blob/main/src/localgw.bpf.c)**, **[`src/mvmtap.bpf.c`](https://github.com/TencentCloud/CubeSandbox/blob/main/src/mvmtap.bpf.c)**, **[`src/nodenic.bpf.c`](https://github.com/TencentCloud/CubeSandbox/blob/main/src/nodenic.bpf.c)** – The kernel-space eBPF programs compiled to BPF bytecode.
- **[`CubeNet/cubevs/cmd/cubevsmapdump/main.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeNet/cubevs/cmd/cubevsmapdump/main.go)** – Debugging utility for inspecting BPF map contents.

## Summary

- CubeVS attaches three eBPF programs (`nodenic`, `localgw`, `mvmtap`) to the TC subsystem for comprehensive packet interception.
- Network isolation is enforced using LPM-Trie maps for CIDR-based egress policies and hash maps for DNS filtering and port NAT.
- All packet processing occurs in kernel space, eliminating context-switch overhead and preventing privilege escalation bypasses.
- The Go control plane in the `cubevs` package manages BPF maps through APIs like `UpsertTAPDeviceMeta` and `initNetPolicy`.
- SNAT pools and per-sandbox IP-to-ifindex mappings prevent IP spoofing and ensure controlled egress addressing.

## Frequently Asked Questions

### How does CubeVS prevent sandboxed workloads from bypassing network policies?

CubeVS enforces policies through eBPF programs attached to the TC subsystem, which intercept packets before they reach the sandbox network interface. Because this enforcement occurs in kernel space, sandboxed containers—including those running with elevated privileges or in privileged mode—cannot bypass the `allow_out_v2`, `deny_out`, or `dns_allow` map lookups. The policies are evaluated for every packet before redirection to the TAP device or physical NIC.

### What eBPF map types does CubeVS use for policy enforcement?

CubeVS utilizes **LPM-Trie** (Longest Prefix Match) maps for CIDR-based egress policies, enabling efficient matching of IP addresses against network ranges. For DNS filtering, it uses **hash-of-maps** structures. Per-device metadata is stored in hash maps, while port mappings use standard hash maps for NAT translations. These map types are created and managed through functions like `newInnerLPMMap` in [`netpolicy.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/netpolicy.go).

### How does CubeVS handle DNS-based filtering?

DNS filtering is implemented through a dedicated **`dns_allow`** hash map that stores hashed DNS names permitted for each sandbox. When DNS packets traverse the CubeVS data plane, tail-call programs (`dns_parse_chunk`, `dns_rev_chunk`) parse the queries and validate them against the sandbox's allowed list. Unauthorized DNS queries are dropped before reaching external resolvers, preventing data exfiltration via DNS tunneling.

### What is the performance impact of using eBPF for network isolation?

Because CubeVS processes packets entirely in kernel space using eBPF maps and TC hooks, it eliminates the context-switch overhead associated with userspace packet filtering. The architecture achieves sub-microsecond latency for intra-node traffic by using efficient LPM-Trie lookups for CIDR matching and direct packet redirection via `BPFRedirectFlagIngress`, making it suitable for high-throughput sandboxed workloads.