How CubeSandbox Avoids iptables and OVS for Networking: A Pure eBPF Architecture

CubeSandbox eliminates iptables and Open vSwitch entirely by implementing CubeVS, a custom eBPF networking stack that handles NAT, session tracking, and policy enforcement directly in kernel space using three specialized BPF programs attached to TAP devices and host interfaces.

The TencentCloud/CubeSandbox project replaces the traditional container networking stack with a high-performance eBPF datapath. By avoiding iptables rules and OVS bridges entirely, CubeSandbox achieves deterministic per-sandbox policy enforcement and scalable NAT without the rule explosion typical of multi-tenant environments.

The CubeVS eBPF Architecture

CubeSandbox implements a dedicated networking layer called CubeVS that resides entirely in the kernel. Instead of relying on the Linux netfilter framework or Open vSwitch datapath, CubeVS uses three small BPF programs attached directly to network interfaces:

• mvmtap.bpf.c – Attached to the TAP device for each sandbox
• nodenic.bpf.c – Attached to the host NIC
• localgw.bpf.c – Attached to the internal "cube-dev" interface

These programs perform SNAT/DNAT, session tracking, ARP proxy, policy checks, and L7 proxy steering entirely within kernel space, eliminating the need for iptables chains.

BPF Maps Replace iptables Rules

Traditional container networking stores policy state in iptables rules, which can scale poorly. CubeSandbox stores all per-sandbox policy state and NAT tables in pinned BPF maps that the eBPF programs consult on-the-fly:

• allow_out – LPM trie for outbound allow-list CIDRs
• deny_out – LPM trie for outbound deny-list CIDRs
• egress_sessions – Session tracking for NAT connections
• ifindex_to_mvmmeta – Maps interface indices to sandbox metadata
• mvmip_to_ifindex – Resolves sandbox IPs to TAP interface indices

This map-based approach replaces the classic iptables rule database and avoids the performance degradation caused by linear rule evaluation in multi-tenant hosts.

Go Control Plane

The Go control plane in the cubevs package manages the eBPF lifecycle without invoking iptables or configuring OVS bridges. Key functions include:

• cubevs.Init – Loads the three BPF objects and rewrites interface-specific constants
• cubevs.AddTAPDevice – Registers new TAP devices with the BPF maps
• cubevs.SetSNATIPs – Configures SNAT IP pools for the BPF watermark allocator
• cubevs.UpsertTAPDevice – Updates per-sandbox policy configurations

According to the source code in network-agent/internal/service/local_service.go, these functions only load, pin, and rewrite constants in the BPF objects, never executing iptables commands.

No Linux Bridge or OVS

Each sandbox receives an exclusive TAP device, and traffic never passes through a shared Linux bridge or OVS bridge. The transition from TAP to host is handled by the from_cube BPF filter attached via TC (Traffic Control), not by a bridge layer.

As documented in docs/architecture/overview.md, this design ensures there are "no iptables rules, no Linux Bridge, no OVS – pure eBPF at each boundary."

Minimal TPROXY Exception

The only iptables involvement in CubeSandbox is a single host-wide rule to install TPROXY for HTTP/HTTPS traffic that must be sent to the user-space CubeEgress proxy. This rule does not implement NAT or policy enforcement; those functions remain in eBPF.

The network-agent/internal/service/cube_router.go file documents this exception, showing that only these minimal rules are created:

iptables -t mangle -A PREROUTING -p tcp -m tcp --dport 80 -j TPROXY \
    --on-port 15001 --tproxy-mark 0x1/0xffffffff

iptables -t mangle -A PREROUTING -p tcp -m tcp --dport 443 -j TPROXY \
    --on-port 15001 --tproxy-mark 0x1/0xffffffff

These rules are installed once per host via deploy/one-click/scripts/systemd/cube-egress-net-start.sh, not per-sandbox.

Scalable NAT Without iptables MASQUERADE

SNAT port allocation is performed by the BPF program itself using a lock-protected watermark pool. This approach completely avoids the iptables -j MASQUERADE target, which would generate thousands of rules in a multi-tenant host and create a bottleneck during connection setup.

The docs/architecture/network.md documentation explains that this Scalable NAT design allows CubeSandbox to handle high connection rates without the overhead of netfilter rule management.

Implementation Examples

Below are practical code examples showing how CubeSandbox configures networking entirely through eBPF.

Initializing CubeVS

The network-agent initializes the eBPF datapath without creating any iptables rules:

// network-agent/internal/service/local_service.go
params := cubevs.Params{
    MVMInnerIP:          mvmInnerIP,
    MVMMacAddr:          mvmMacAddr,
    MVMGatewayIP:        mvmGatewayIP,
    Cubegw0Ifindex:      uint32(cdev.Index),
    Cubegw0IP:           cdev.IP,
    Cubegw0MacAddr:      cdev.Mac,
    EgressSrcMacAddr:    egressSrcMac,
    EgressDstMacAddr:    egressDstMac,
    EgressRedirectFlags: egressRedirectFlags,
    CubeRouterIfindex:   cubeRouterIfindex,
    NodeIfindex:         uint32(device.Index),
    NodeIP:              device.IP,
    NodeMacAddr:         device.Mac,
    NodeGatewayMacAddr:  device.GatewayMac,
}
if err := cubevs.Init(params); err != nil {
    return nil, err}

This loads the three BPF objects, rewrites constants for IPs and MAC addresses, and attaches the from_cube TC filter to the TAP device.

Adding a Sandbox TAP Device

New sandboxes are registered with the eBPF maps directly:

// network-agent/internal/service/local_service.go
if err := cubevs.AddTAPDevice(tap.Index, tap.IP, sandboxID); err != nil {
    return fmt.Errorf("add TAP failed: %w", err)
}

This updates the ifindex_to_mvmmeta and mvmip_to_ifindex maps, allowing the BPF program to resolve the sandbox's TAP and apply per-sandbox policies.

Updating Policy Maps

Policy changes are written directly to BPF maps without touching iptables:

// network-agent/internal/service/local_service.go
if err := cubevs.UpsertTAPDevice(tap.Index, tap.IP, sandboxID, cfg); err != nil {
    return fmt.Errorf("upsert TAP failed: %w", err)
}

The cfg parameter contains merged allow_out and deny_out CIDR lists that the function writes into the per-sandbox LPM tries. The eBPF program consults these maps for every outbound packet.

Summary

• CubeSandbox replaces iptables and OVS with CubeVS, a pure eBPF networking stack using three BPF programs (mvmtap.bpf.c, nodenic.bpf.c, localgw.bpf.c).
• BPF maps (allow_out, deny_out, egress_sessions) store policy and NAT state, eliminating iptables rule chains.
• Exclusive TAP devices per sandbox avoid Linux bridges and OVS bridges entirely.
• The Go control plane in network-agent/internal/service/local_service.go manages the eBPF lifecycle via the cubevs package without invoking iptables.
• Scalable NAT uses a BPF-based watermark allocator instead of iptables MASQUERADE.
• Only a single host-wide TPROXY rule uses iptables, solely for redirecting HTTP/HTTPS to the user-space proxy.

Frequently Asked Questions

What replaces iptables NAT in CubeSandbox?

SNAT and DNAT are implemented entirely in eBPF. The localgw.bpf.c and nodenic.bpf.c programs handle address translation using BPF maps for session tracking. Port allocation uses a lock-protected watermark pool in the BPF program, avoiding the performance overhead of iptables MASQUERADE rules.

Does CubeSandbox use any iptables at all?

Only for TPROXY redirection. A single host-wide iptables rule in the mangle table redirects HTTP/HTTPS traffic (ports 80/443) to the CubeEgress proxy using TPROXY. This rule does not perform NAT or policy enforcement; all other networking functions remain in eBPF.

How does CubeSandbox handle network policy without iptables?

Network policies are enforced via BPF LPM trie maps. The allow_out and deny_out maps store CIDR-based rules that the eBPF program checks against every outbound packet. This provides O(1) lookup time regardless of the number of sandboxes, unlike iptables which requires linear rule evaluation.

Why is OVS not used in CubeSandbox?

CubeSandbox assigns exclusive TAP devices directly to each sandbox and uses TC filters to attach eBPF programs to these devices. Traffic flows directly from the TAP through the eBPF datapath to the host NIC without traversing a shared bridge or OVS datapath, reducing latency and complexity.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}