How to Implement Route-Aware Egress with Policy Routing in CubeSandbox

CubeSandbox enables route-aware egress by configuring the network-agent with three specific parameters—CubeRouterEnable, CubeRouterCIDR, and CubeRouterMacAddr—which instruct the CubeRouter service to install host-side policy routing rules that direct sandbox traffic out specific network interfaces instead of the default NAT path.

By default, CubeSandbox NATs all egress traffic to the host IP attached to the sandbox's virtual Ethernet (veth) pair. When deploying multi-NIC or multi-VPC environments, route-aware egress with policy routing allows granular control over outbound traffic paths by assigning each sandbox a unique host-side routing table and dedicated egress interface.

Architecture and Components

CubeVS BPF Datapath

The CubeVS component (CubeNet/cubevs/*.go) loads a BPF program that rewrites the sandbox's source MAC and IP addresses, then directs packets to the host-side "egress" port. This establishes the packet interception point before policy routing rules apply.

CubeRouter Policy Routing Engine

Located in network-agent/internal/service/cube_router.go, the CubeRouter service installs host-side routing rules and static ARP entries. When enabled, it creates per-sandbox routing tables and policy rules that select these tables based on the sandbox's source MAC address (EgressSrcMacAddr).

Configuration Knobs

The feature is controlled through the Config struct defined in network-agent/internal/service/config.go (lines 53-57). Three fields enable route-aware egress:

• CubeRouterEnable: Boolean toggle to activate the feature
• CubeRouterCIDR: The CIDR range of the host-side egress interface (e.g., 10.0.0.0/24)
• CubeRouterMacAddr: The MAC address the host uses on the egress interface

When CubeRouterEnable is true, the agent performs four steps for each sandbox:

1. Creates a host-side egress veth pair via local_service.go
2. Sets up a unique routing table (ID derived from sandbox IP hash) routing toward CubeRouterCIDR
3. Installs a policy rule selecting this table for packets originating from the sandbox's source MAC
4. Adds an ARP entry on the egress NIC mapping the sandbox's virtual MAC to its IP

Enabling Route-Aware Egress

Configure the network-agent TOML

Edit /etc/cubelet/network.toml to enable policy routing:

[plugins."io.cubelet.internal.v1.network"]
eth_name = "eth0"
cube_router_enable = true
cube_router_cidr = "10.0.0.0/24"
cube_router_mac_addr = "22:90:6f:cf:cf:cf"

The agent parses these fields in LoadConfigFromCubeletTOML (lines 53-63 of config.go). Restart the network-agent to apply changes.

Verify Host-Side Routing Tables

Check that per-sandbox tables exist using the table ID offset (0x1000-based):

ip rule list | grep sandbox

Expected output shows rules like:

2000: from all fwmark 0x1c9d lookup 0x14b1

Validate Traffic Paths

Confirm traffic exits via the intended NIC:

tcpdump -i eth0 -e -n host <sandbox-ip>

Packets should display the MAC address defined in cube_router_mac_addr with the sandbox's source IP.

Code Implementation Examples

Creating a Config Programmatically

Instantiate the network-agent with route-aware egress enabled:

cfg := service.DefaultConfig()
cfg.CubeRouterEnable = true
cfg.CubeRouterCIDR = "10.0.0.0/24"
cfg.CubeRouterMacAddr = "22:90:6f:cf:cf:cf"
agent, _ := service.New(cfg) // New() is the service entry point

Source: service/config.go lines 53-57 (definition) and lines 58-81 (defaults).

Calculating Sandbox Routing Table IDs

The CubeRouter uses a hash of the sandbox's IP address to generate unique table IDs. Access this logic via MakeTableID:

func getRoutingTableID(sandboxIP string) (int, error) {
    ip := net.ParseIP(sandboxIP)
    if ip == nil {
        return 0, fmt.Errorf("invalid IP %s", sandboxIP)
    }
    // See cube_router.go near line 407 for hash algorithm details
    return service.MakeTableID(ip), nil
}

Pushing L7 Egress Rules Alongside Route Awareness

Route-aware egress operates orthogonally to L7 policies pushed to CubeEgress:

rule := &sdk.EgressRule{
    Name: "allow-http",
    Match: &sdk.EgressRuleMatch{
        Host: "example.com",
    },
    Action: &sdk.EgressRuleAction{
        Allow: true,
    },
}
err := client.PutEgressPolicy(ctx, "sandbox-1234", []*sdk.EgressRule{rule})

Source: sdk/go/policy.go for rule definitions; cubeegress_push.go for the push flow.

Troubleshooting Common Issues

No Egress on Intended Interface

If CubeRouterEnable is false or omitted, traffic defaults to standard NAT. Verify cube_router_enable = true in the TOML configuration.

"Network is Unreachable" Errors

This occurs when CubeRouterCIDR does not cover the host-side gateway address. Ensure the CIDR matches the egress NIC's subnet exactly.

Duplicate IP Rule Conflicts

Rarely, two sandboxes may share the same sandboxIP causing duplicate rule errors. Verify unique IP allocation through CubeNet/cubevs/ipam.

CubeEgress API Errors

Misconfigured CubeEgressAdminURL causes API errors but does not affect routing. Leave CubeEgressAdminURL empty if you only require route-aware egress without L7 policies.

Summary

• Route-aware egress requires enabling CubeRouterEnable and configuring CubeRouterCIDR and CubeRouterMacAddr in network-agent/internal/service/config.go
• The CubeRouter service (cube_router.go) creates per-sandbox routing tables using hashed IDs from sandbox IPs
• Policy rules select tables based on source MAC addresses, allowing traffic to egress specific host interfaces
• L7 egress policies via CubeEgress operate independently and can run parallel to route-aware routing
• Verification uses standard Linux tools: ip rule list and tcpdump

Frequently Asked Questions

What is the relationship between CubeRouter and CubeEgress?

CubeRouter handles L3/L4 policy routing at the host level, determining which physical interface handles sandbox traffic. CubeEgress handles L7 application-layer filtering. They operate independently according to the CubeSandbox source code; you can use route-aware egress without enabling CubeEgress.

How does CubeSandbox calculate unique routing table IDs?

The system hashes the sandbox's IP address to generate a unique table ID offset (basis around 0x1000). See the makeTableID function in network-agent/internal/service/cube_router.go near line 407 for the specific hashing algorithm.

Can I implement route-aware egress without the CubeEgress service?

Yes. Route-aware egress depends only on the CubeRouter configuration parameters (CubeRouterEnable, CubeRouterCIDR, CubeRouterMacAddr). The CubeEgress API and CubeEgressAdminURL are only required if you need L7 egress filtering.

Which configuration file controls policy routing parameters?

The network-agent reads route-aware egress settings from the Cubelet TOML configuration, typically located at /etc/cubelet/network.toml under the [plugins."io.cubelet.internal.v1.network"] section, parsed by LoadConfigFromCubeletTOML in config.go.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}