# How CubeSandbox Avoids iptables and OVS for Networking: A Pure eBPF Architecture

> Discover how CubeSandbox eliminates iptables and OVS with a pure eBPF networking stack. Learn how CubeVS handles NAT, session tracking, and policy enforcement in kernel space.

- Repository: [Tencent Cloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox)
- Tags: internals
- Published: 2026-07-08

---

**CubeSandbox eliminates iptables and Open vSwitch entirely by implementing CubeVS, a custom eBPF networking stack that handles NAT, session tracking, and policy enforcement directly in kernel space using three specialized BPF programs attached to TAP devices and host interfaces.**

The TencentCloud/CubeSandbox project replaces the traditional container networking stack with a high-performance eBPF datapath. By avoiding iptables rules and OVS bridges entirely, CubeSandbox achieves deterministic per-sandbox policy enforcement and scalable NAT without the rule explosion typical of multi-tenant environments.

## The CubeVS eBPF Architecture

CubeSandbox implements a dedicated networking layer called **CubeVS** that resides entirely in the kernel. Instead of relying on the Linux netfilter framework or Open vSwitch datapath, CubeVS uses three small BPF programs attached directly to network interfaces:

- **[`mvmtap.bpf.c`](https://github.com/TencentCloud/CubeSandbox/blob/main/mvmtap.bpf.c)** – Attached to the TAP device for each sandbox
- **[`nodenic.bpf.c`](https://github.com/TencentCloud/CubeSandbox/blob/main/nodenic.bpf.c)** – Attached to the host NIC
- **[`localgw.bpf.c`](https://github.com/TencentCloud/CubeSandbox/blob/main/localgw.bpf.c)** – Attached to the internal "cube-dev" interface

These programs perform **SNAT/DNAT, session tracking, ARP proxy, policy checks, and L7 proxy steering** entirely within kernel space, eliminating the need for iptables chains.

## BPF Maps Replace iptables Rules

Traditional container networking stores policy state in iptables rules, which can scale poorly. CubeSandbox stores all per-sandbox policy state and NAT tables in **pinned BPF maps** that the eBPF programs consult on-the-fly:

- **`allow_out`** – LPM trie for outbound allow-list CIDRs
- **`deny_out`** – LPM trie for outbound deny-list CIDRs
- **`egress_sessions`** – Session tracking for NAT connections
- **`ifindex_to_mvmmeta`** – Maps interface indices to sandbox metadata
- **`mvmip_to_ifindex`** – Resolves sandbox IPs to TAP interface indices

This map-based approach replaces the classic iptables rule database and avoids the performance degradation caused by linear rule evaluation in multi-tenant hosts.

## Go Control Plane

The Go control plane in the `cubevs` package manages the eBPF lifecycle without invoking iptables or configuring OVS bridges. Key functions include:

- **`cubevs.Init`** – Loads the three BPF objects and rewrites interface-specific constants
- **`cubevs.AddTAPDevice`** – Registers new TAP devices with the BPF maps
- **`cubevs.SetSNATIPs`** – Configures SNAT IP pools for the BPF watermark allocator
- **`cubevs.UpsertTAPDevice`** – Updates per-sandbox policy configurations

According to the source code in [`network-agent/internal/service/local_service.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/network-agent/internal/service/local_service.go), these functions only load, pin, and rewrite constants in the BPF objects, never executing `iptables` commands.

## No Linux Bridge or OVS

Each sandbox receives an exclusive TAP device, and traffic never passes through a shared Linux bridge or OVS bridge. The transition from TAP to host is handled by the `from_cube` BPF filter attached via TC (Traffic Control), not by a bridge layer.

As documented in [`docs/architecture/overview.md`](https://github.com/TencentCloud/CubeSandbox/blob/main/docs/architecture/overview.md), this design ensures there are "no iptables rules, no Linux Bridge, no OVS – pure eBPF at each boundary."

## Minimal TPROXY Exception

The only iptables involvement in CubeSandbox is a **single host-wide rule** to install TPROXY for HTTP/HTTPS traffic that must be sent to the user-space CubeEgress proxy. This rule does not implement NAT or policy enforcement; those functions remain in eBPF.

The [`network-agent/internal/service/cube_router.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/network-agent/internal/service/cube_router.go) file documents this exception, showing that only these minimal rules are created:

```bash
iptables -t mangle -A PREROUTING -p tcp -m tcp --dport 80 -j TPROXY \
    --on-port 15001 --tproxy-mark 0x1/0xffffffff

iptables -t mangle -A PREROUTING -p tcp -m tcp --dport 443 -j TPROXY \
    --on-port 15001 --tproxy-mark 0x1/0xffffffff

```

These rules are installed once per host via [`deploy/one-click/scripts/systemd/cube-egress-net-start.sh`](https://github.com/TencentCloud/CubeSandbox/blob/main/deploy/one-click/scripts/systemd/cube-egress-net-start.sh), not per-sandbox.

## Scalable NAT Without iptables MASQUERADE

SNAT port allocation is performed by the BPF program itself using a lock-protected watermark pool. This approach completely avoids the iptables `-j MASQUERADE` target, which would generate thousands of rules in a multi-tenant host and create a bottleneck during connection setup.

The [`docs/architecture/network.md`](https://github.com/TencentCloud/CubeSandbox/blob/main/docs/architecture/network.md) documentation explains that this **Scalable NAT** design allows CubeSandbox to handle high connection rates without the overhead of netfilter rule management.

## Implementation Examples

Below are practical code examples showing how CubeSandbox configures networking entirely through eBPF.

### Initializing CubeVS

The `network-agent` initializes the eBPF datapath without creating any iptables rules:

```go
// network-agent/internal/service/local_service.go
params := cubevs.Params{
    MVMInnerIP:          mvmInnerIP,
    MVMMacAddr:          mvmMacAddr,
    MVMGatewayIP:        mvmGatewayIP,
    Cubegw0Ifindex:      uint32(cdev.Index),
    Cubegw0IP:           cdev.IP,
    Cubegw0MacAddr:      cdev.Mac,
    EgressSrcMacAddr:    egressSrcMac,
    EgressDstMacAddr:    egressDstMac,
    EgressRedirectFlags: egressRedirectFlags,
    CubeRouterIfindex:   cubeRouterIfindex,
    NodeIfindex:         uint32(device.Index),
    NodeIP:              device.IP,
    NodeMacAddr:         device.Mac,
    NodeGatewayMacAddr:  device.GatewayMac,
}
if err := cubevs.Init(params); err != nil {
    return nil, err}

```

This loads the three BPF objects, rewrites constants for IPs and MAC addresses, and attaches the `from_cube` TC filter to the TAP device.

### Adding a Sandbox TAP Device

New sandboxes are registered with the eBPF maps directly:

```go
// network-agent/internal/service/local_service.go
if err := cubevs.AddTAPDevice(tap.Index, tap.IP, sandboxID); err != nil {
    return fmt.Errorf("add TAP failed: %w", err)
}

```

This updates the `ifindex_to_mvmmeta` and `mvmip_to_ifindex` maps, allowing the BPF program to resolve the sandbox's TAP and apply per-sandbox policies.

### Updating Policy Maps

Policy changes are written directly to BPF maps without touching iptables:

```go
// network-agent/internal/service/local_service.go
if err := cubevs.UpsertTAPDevice(tap.Index, tap.IP, sandboxID, cfg); err != nil {
    return fmt.Errorf("upsert TAP failed: %w", err)
}

```

The `cfg` parameter contains merged `allow_out` and `deny_out` CIDR lists that the function writes into the per-sandbox LPM tries. The eBPF program consults these maps for every outbound packet.

## Summary

- **CubeSandbox** replaces iptables and OVS with **CubeVS**, a pure eBPF networking stack using three BPF programs ([`mvmtap.bpf.c`](https://github.com/TencentCloud/CubeSandbox/blob/main/mvmtap.bpf.c), [`nodenic.bpf.c`](https://github.com/TencentCloud/CubeSandbox/blob/main/nodenic.bpf.c), [`localgw.bpf.c`](https://github.com/TencentCloud/CubeSandbox/blob/main/localgw.bpf.c)).
- **BPF maps** (`allow_out`, `deny_out`, `egress_sessions`) store policy and NAT state, eliminating iptables rule chains.
- **Exclusive TAP devices** per sandbox avoid Linux bridges and OVS bridges entirely.
- The Go control plane in [`network-agent/internal/service/local_service.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/network-agent/internal/service/local_service.go) manages the eBPF lifecycle via the `cubevs` package without invoking iptables.
- **Scalable NAT** uses a BPF-based watermark allocator instead of iptables MASQUERADE.
- Only a **single host-wide TPROXY rule** uses iptables, solely for redirecting HTTP/HTTPS to the user-space proxy.

## Frequently Asked Questions

### What replaces iptables NAT in CubeSandbox?

**SNAT and DNAT are implemented entirely in eBPF.** The [`localgw.bpf.c`](https://github.com/TencentCloud/CubeSandbox/blob/main/localgw.bpf.c) and [`nodenic.bpf.c`](https://github.com/TencentCloud/CubeSandbox/blob/main/nodenic.bpf.c) programs handle address translation using BPF maps for session tracking. Port allocation uses a lock-protected watermark pool in the BPF program, avoiding the performance overhead of iptables MASQUERADE rules.

### Does CubeSandbox use any iptables at all?

**Only for TPROXY redirection.** A single host-wide iptables rule in the mangle table redirects HTTP/HTTPS traffic (ports 80/443) to the CubeEgress proxy using TPROXY. This rule does not perform NAT or policy enforcement; all other networking functions remain in eBPF.

### How does CubeSandbox handle network policy without iptables?

**Network policies are enforced via BPF LPM trie maps.** The `allow_out` and `deny_out` maps store CIDR-based rules that the eBPF program checks against every outbound packet. This provides O(1) lookup time regardless of the number of sandboxes, unlike iptables which requires linear rule evaluation.

### Why is OVS not used in CubeSandbox?

**CubeSandbox assigns exclusive TAP devices directly to each sandbox** and uses TC filters to attach eBPF programs to these devices. Traffic flows directly from the TAP through the eBPF datapath to the host NIC without traversing a shared bridge or OVS datapath, reducing latency and complexity.