How CubeVS Provides eBPF-Based Networking for Secure Sandbox Isolation

CubeVS implements high-performance, per-sandbox network isolation by loading three specialized eBPF programs into the host kernel that intercept TAP device traffic, enforce ACLs via LPM-Trie maps, and manage NAT sessions without iptables.

CubeVS serves as the kernel-level networking hypervisor for TencentCloud's CubeSandbox project, replacing traditional container networking with an eBPF-based data plane. By moving packet forwarding, access control, and DNS learning logic directly into kernel-space BPF programs, CubeVS eliminates the overhead of user-space packet processing and context switches. The architecture relies on a Go control plane that manipulates BPF maps to synchronize policies, while the eBPF data plane handles every packet at native kernel speed.

TAP Device Architecture and Entry Points

Each sandbox in CubeSandbox receives dedicated network access through a TAP device that represents the sandbox's virtual NIC on the host. When the network-agent creates a sandbox, it invokes AddTAPDevice or UpsertTAPDevice in cubevs/cubevs.go to establish this interface.

The TAP device attaches to the eBPF programs as the primary entry point for all traffic originating from the sandbox. In cubevs.go, the UpsertTAPDevice method configures the device and loads the compiled eBPF ELF object, patching global variables with the map file descriptors before attaching the programs to the TAP interface.

The Three Core eBPF Programs

CubeVS distributes networking logic across three specialized BPF programs that handle distinct traffic directions:

From-Sandbox Traffic (mvmtap.bpf.c)

The from_cube logic (implemented in CubeNet/src/mvmtap.bpf.c) processes packets leaving the sandbox through the TAP device. This program performs:

• LPM-Trie lookups on the allow_out_v2 map to enforce L3/L4 allow/deny policies
• DNS learning: Intercepts UDP port 53 queries, records pending queries in BPF maps, and inserts temporary IP entries with TTL upon receiving DNS responses
• Session creation for TCP/UDP flows, SNAT rewriting using the configured SNAT pool
• L7 redirection: Sets the L7_REQUIRED flag and redirects packets to the cube-dev dummy device for user-space proxy inspection

To-Sandbox Traffic (localgw.bpf.c)

The to_cube logic (implemented in CubeNet/src/localgw.bpf.c) handles inbound packets destined for sandboxes. This program restores original destination addresses through reverse NAT using the sessions map, then forwards the packet to the appropriate TAP device based on the flow state established during outbound processing.

Bridge and ARP Handling (nodenic.bpf.c)

The bridge logic (implemented in CubeNet/src/nodenic.bpf.c) generates ARP replies and forwards Ethernet frames not destined for any specific sandbox. This ensures the sandbox can communicate with the host and external networks without requiring a separate bridge device in the traditional Linux networking stack.

BPF Maps for Policy and Session State

The eBPF programs share state through BPF maps declared in CubeNet/src/cubevs.h and managed by the Go control plane in cubevs/map.go. The critical maps include:

• allow_out_v2: LPM-Trie storing CIDR-based allow/deny rules for outbound connections
• dns_allow: LPM-Trie containing hashed domain names permitted for DNS learning
• netpolicy: Bit-field flags encoding L4 port allow/deny configurations and special handling flags like L7_REQUIRED
• sessions: Per-flow state tracking TCP/UDP NAT mappings, reverse IP translations, and connection metadata

Data Plane Processing Flow

When a packet arrives on the TAP device, the eBPF data plane executes a deterministic pipeline:

1. Policy Enforcement: The program reads the allow_out_v2 map; if the destination IP fails the LPM-Trie lookup, the program returns TC_ACT_SHOT to drop the packet silently
2. DNS Interception: For UDP port 53 traffic, the program checks dns_allow and records queries in the pending query map for response correlation
3. NAT Processing: Allowed packets undergo SNAT rewriting using IPs from the SNAT pool configured in MVMOptions, with reverse flow state written atomically to the sessions map
4. L7 Redirection: If the packet matches a domain requiring deep inspection, the L7_REQUIRED flag from netpolicy triggers redirection to cube-dev, where the CubeEgress user-space proxy performs application-layer analysis

This entire pipeline executes in kernel context without traversing user-space, achieving packet processing latencies measured in microseconds.

Control Plane Integration

The Go library in cubevs/cubevs.go provides the interface between user-defined policies and the eBPF data plane.

Policy Translation

In network-agent/internal/service/local_service.go, the network-agent converts CubeNetworkConfig specifications into MVMOptions structs. These options contain:

• L3/L4 allow/deny lists
• DNS-allow entries (hashed domains)
• NAT configuration and SNAT pools
• Port-mapping definitions

The control plane writes these options into the corresponding BPF maps using helpers defined in cubevs/netpolicy.go and cubevs/dnspolicy.go.

Map Management API

The cubevs package exposes functions to insert, delete, and query map entries with automatic retry logic. Key operations include:

• UpsertTAPDevice: Creates or updates TAP devices and synchronizes map state
• DumpBusinessMaps: Retrieves diagnostic data from maps like allow_out_v2 for troubleshooting
• UpdateDnsAllow: Inserts hashed domain entries into the dns_allow map

Implementation Examples

Creating a TAP Device and Pushing Policy

The following Go code demonstrates how the network-agent creates a sandbox TAP interface and pushes initial ACL rules:

import (
    "github.com/tencentcloud/CubeSandbox/CubeNet/cubevs"
    "github.com/tencentcloud/CubeSandbox/network-agent/internal/service/types"
    "net"
)

// Build MVM options from high-level network config
opts := cubevs.MVMOptions{
    AllowOutV2: []cubevs.LpmKey{
        {PrefixLen: 32, Addr: ipToUint32(net.ParseIP("203.0.113.10"))},
    },
    DnsAllow: []cubevs.DnsAllowKey{
        {DomainHash: hashDomain("example.com")},
    },
    // Additional SNAT pool and port-mapping configuration
}

// Define the TAP device
tap := cubevs.TAPDevice{
    SandboxID: "sandbox-12345",
    IfIndex:   0,  // Kernel allocates interface index
    IP:        net.ParseIP("10.0.0.2"),
    MAC:       net.HardwareAddr{0x02, 0x42, 0xac, 0x11, 0x00, 0x02},
    Options:   opts,
}

// Create or update the device and load eBPF programs
if err := cubevs.UpsertTAPDevice(tap); err != nil {
    log.Fatalf("failed to configure CubeVS: %v", err)
}

Inspecting BPF Maps

To debug policy enforcement, use DumpBusinessMaps to retrieve current map contents:

// Dump specific business maps for diagnostics
opts := cubevs.DumpOptions{MapNames: []string{"allow_out_v2"}}
dump, err := cubevs.DumpBusinessMaps(opts)
if err != nil {
    log.Fatalf("dump error: %v", err)
}
fmt.Println(string(dump))

Adding DNS-Allow Rules

Domain-based filtering requires hashing the domain before insertion:

domain := "api.example.com"
hash := cubevs.HashDomain(domain)  // Must match eBPF hashing algorithm

key := cubevs.DnsAllowKey{DomainHash: hash}
val := cubevs.DnsAllowValue{ExpiresAtNs: 0}  // 0 indicates permanent rule

if err := cubevs.UpdateDnsAllow(key, val); err != nil {
    log.Fatalf("failed to add DNS rule: %v", err)
}

Summary

• CubeVS replaces traditional iptables-based networking with eBPF programs attached to per-sandbox TAP devices, eliminating context switches and reducing latency
• Three specialized BPF programs handle distinct traffic directions: mvmtap.bpf.c for outbound traffic and DNS learning, localgw.bpf.c for inbound NAT reversals, and nodenic.bpf.c for layer-2 bridging
• LPM-Trie maps (allow_out_v2, dns_allow) enable efficient longest-prefix matching for CIDR and domain-based policies directly in the kernel
• The Go control plane in cubevs.go translates high-level CubeNetworkConfig into MVMOptions, then updates BPF maps atomically without reloading programs
• DNS learning occurs in the eBPF data plane, where UDP port 53 traffic triggers automatic insertion of resolved IPs into temporary allow lists with TTL management
• L7 inspection redirects specific flows to the cube-dev dummy device, enabling seamless integration with user-space proxies while maintaining kernel-fast path for allowed traffic

Frequently Asked Questions

How does CubeVS handle DNS traffic with eBPF?

CubeVS intercepts DNS queries at the eBPF level in mvmtap.bpf.c by filtering UDP port 53 traffic. When a query matches an entry in the dns_allow LPM-Trie map, the program records the pending query in a BPF map. Upon receiving the DNS response, the eBPF program extracts the resolved IP addresses and inserts them into temporary allow-list entries with automatic TTL expiration. This eliminates the need for a separate DNS proxy process and allows immediate network policy enforcement based on domain names rather than just IP addresses.

What is the difference between the three eBPF programs in CubeVS?

The three programs partition networking responsibilities by traffic direction. mvmtap.bpf.c (from-sandbox) handles outbound packets, enforcing ACLs, performing DNS learning, and executing SNAT. localgw.bpf.c (to-sandbox) processes inbound packets, restoring original destination addresses through reverse NAT using the sessions map. nodenic.bpf.c (bridge) manages layer-2 operations including ARP reply generation and forwarding Ethernet frames to external networks. This separation allows each program to optimize for its specific code path while sharing state through common BPF maps.

How does CubeVS achieve network isolation without iptables?

CubeVS eliminates iptables by implementing all filtering, NAT, and forwarding logic directly in eBPF programs attached to the sandbox's TAP device. When a packet arrives, the eBPF program consults BPF maps (like allow_out_v2 for CIDR rules and netpolicy for port restrictions) and returns TC_ACT_SHOT to drop denied packets or TC_ACT_OK to forward allowed traffic. Because this executes in kernel context without traversing netfilter hooks or user-space, it provides more granular per-sandbox policies with significantly lower overhead than iptables rules.

Can CubeVS policies be updated without disrupting existing sandboxes?

Yes, CubeVS supports dynamic policy updates through BPF map modifications. The Go control plane calls functions like UpsertTAPDevice or UpdateDnsAllow to atomically update map entries while the eBPF programs continue running. Since the programs remain loaded and only the map contents change, existing TCP sessions and NAT flows persist during policy updates. The control plane handles retries and rollbacks in cubevs.go to ensure consistency between the desired configuration in MVMOptions and the actual kernel state.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}