# How to Implement Route-Aware Egress with Policy Routing in CubeSandbox

> Implement route-aware egress in CubeSandbox using policy routing. Configure network-agent parameters to direct sandbox traffic via specific interfaces, bypassing default NAT for enhanced control.

- Repository: [Tencent Cloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox)
- Tags: how-to-guide
- Published: 2026-07-15

---

**CubeSandbox enables route-aware egress by configuring the network-agent with three specific parameters—`CubeRouterEnable`, `CubeRouterCIDR`, and `CubeRouterMacAddr`—which instruct the CubeRouter service to install host-side policy routing rules that direct sandbox traffic out specific network interfaces instead of the default NAT path.**

By default, CubeSandbox NATs all egress traffic to the host IP attached to the sandbox's virtual Ethernet (veth) pair. When deploying multi-NIC or multi-VPC environments, **route-aware egress with policy routing** allows granular control over outbound traffic paths by assigning each sandbox a unique host-side routing table and dedicated egress interface.

## Architecture and Components

### CubeVS BPF Datapath

The `CubeVS` component (`CubeNet/cubevs/*.go`) loads a BPF program that rewrites the sandbox's source MAC and IP addresses, then directs packets to the host-side "egress" port. This establishes the packet interception point before policy routing rules apply.

### CubeRouter Policy Routing Engine

Located in [`network-agent/internal/service/cube_router.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/network-agent/internal/service/cube_router.go), the **CubeRouter** service installs host-side routing rules and static ARP entries. When enabled, it creates per-sandbox routing tables and policy rules that select these tables based on the sandbox's source MAC address (`EgressSrcMacAddr`).

### Configuration Knobs

The feature is controlled through the `Config` struct defined in [`network-agent/internal/service/config.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/network-agent/internal/service/config.go) (lines 53-57). Three fields enable route-aware egress:

- **`CubeRouterEnable`**: Boolean toggle to activate the feature
- **`CubeRouterCIDR`**: The CIDR range of the host-side egress interface (e.g., `10.0.0.0/24`)
- **`CubeRouterMacAddr`**: The MAC address the host uses on the egress interface

When `CubeRouterEnable` is **true**, the agent performs four steps for each sandbox:

1. Creates a host-side egress veth pair via [`local_service.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/local_service.go)
2. Sets up a unique routing table (ID derived from sandbox IP hash) routing toward `CubeRouterCIDR`
3. Installs a policy rule selecting this table for packets originating from the sandbox's source MAC
4. Adds an ARP entry on the egress NIC mapping the sandbox's virtual MAC to its IP

## Enabling Route-Aware Egress

### Configure the network-agent TOML

Edit [`/etc/cubelet/network.toml`](https://github.com/TencentCloud/CubeSandbox/blob/main//etc/cubelet/network.toml) to enable policy routing:

```toml
[plugins."io.cubelet.internal.v1.network"]
eth_name = "eth0"
cube_router_enable = true
cube_router_cidr = "10.0.0.0/24"
cube_router_mac_addr = "22:90:6f:cf:cf:cf"

```

The agent parses these fields in `LoadConfigFromCubeletTOML` (lines 53-63 of [`config.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/config.go)). Restart the network-agent to apply changes.

### Verify Host-Side Routing Tables

Check that per-sandbox tables exist using the table ID offset (0x1000-based):

```bash
ip rule list | grep sandbox

```

Expected output shows rules like:

```

2000: from all fwmark 0x1c9d lookup 0x14b1

```

### Validate Traffic Paths

Confirm traffic exits via the intended NIC:

```bash
tcpdump -i eth0 -e -n host <sandbox-ip>

```

Packets should display the MAC address defined in `cube_router_mac_addr` with the sandbox's source IP.

## Code Implementation Examples

### Creating a Config Programmatically

Instantiate the network-agent with route-aware egress enabled:

```go
cfg := service.DefaultConfig()
cfg.CubeRouterEnable = true
cfg.CubeRouterCIDR = "10.0.0.0/24"
cfg.CubeRouterMacAddr = "22:90:6f:cf:cf:cf"
agent, _ := service.New(cfg) // New() is the service entry point

```

*Source:* [`service/config.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/service/config.go) lines 53-57 (definition) and lines 58-81 (defaults).

### Calculating Sandbox Routing Table IDs

The CubeRouter uses a hash of the sandbox's IP address to generate unique table IDs. Access this logic via `MakeTableID`:

```go
func getRoutingTableID(sandboxIP string) (int, error) {
    ip := net.ParseIP(sandboxIP)
    if ip == nil {
        return 0, fmt.Errorf("invalid IP %s", sandboxIP)
    }
    // See cube_router.go near line 407 for hash algorithm details
    return service.MakeTableID(ip), nil
}

```

### Pushing L7 Egress Rules Alongside Route Awareness

Route-aware egress operates orthogonally to L7 policies pushed to CubeEgress:

```go
rule := &sdk.EgressRule{
    Name: "allow-http",
    Match: &sdk.EgressRuleMatch{
        Host: "example.com",
    },
    Action: &sdk.EgressRuleAction{
        Allow: true,
    },
}
err := client.PutEgressPolicy(ctx, "sandbox-1234", []*sdk.EgressRule{rule})

```

*Source:* [`sdk/go/policy.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/sdk/go/policy.go) for rule definitions; [`cubeegress_push.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/cubeegress_push.go) for the push flow.

## Troubleshooting Common Issues

### No Egress on Intended Interface

If `CubeRouterEnable` is false or omitted, traffic defaults to standard NAT. Verify `cube_router_enable = true` in the TOML configuration.

### "Network is Unreachable" Errors

This occurs when `CubeRouterCIDR` does not cover the host-side gateway address. Ensure the CIDR matches the egress NIC's subnet exactly.

### Duplicate IP Rule Conflicts

Rarely, two sandboxes may share the same `sandboxIP` causing duplicate rule errors. Verify unique IP allocation through `CubeNet/cubevs/ipam`.

### CubeEgress API Errors

Misconfigured `CubeEgressAdminURL` causes API errors but does not affect routing. Leave `CubeEgressAdminURL` empty if you only require route-aware egress without L7 policies.

## Summary

- **Route-aware egress** requires enabling `CubeRouterEnable` and configuring `CubeRouterCIDR` and `CubeRouterMacAddr` in [`network-agent/internal/service/config.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/network-agent/internal/service/config.go)
- The **CubeRouter** service ([`cube_router.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/cube_router.go)) creates per-sandbox routing tables using hashed IDs from sandbox IPs
- **Policy rules** select tables based on source MAC addresses, allowing traffic to egress specific host interfaces
- **L7 egress policies** via CubeEgress operate independently and can run parallel to route-aware routing
- Verification uses standard Linux tools: `ip rule list` and `tcpdump`

## Frequently Asked Questions

### What is the relationship between CubeRouter and CubeEgress?

**CubeRouter** handles L3/L4 policy routing at the host level, determining which physical interface handles sandbox traffic. **CubeEgress** handles L7 application-layer filtering. They operate independently according to the CubeSandbox source code; you can use route-aware egress without enabling CubeEgress.

### How does CubeSandbox calculate unique routing table IDs?

The system hashes the sandbox's IP address to generate a unique table ID offset (basis around 0x1000). See the `makeTableID` function in [`network-agent/internal/service/cube_router.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/network-agent/internal/service/cube_router.go) near line 407 for the specific hashing algorithm.

### Can I implement route-aware egress without the CubeEgress service?

Yes. Route-aware egress depends only on the CubeRouter configuration parameters (`CubeRouterEnable`, `CubeRouterCIDR`, `CubeRouterMacAddr`). The CubeEgress API and `CubeEgressAdminURL` are only required if you need L7 egress filtering.

### Which configuration file controls policy routing parameters?

The network-agent reads route-aware egress settings from the Cubelet TOML configuration, typically located at [`/etc/cubelet/network.toml`](https://github.com/TencentCloud/CubeSandbox/blob/main//etc/cubelet/network.toml) under the `[plugins."io.cubelet.internal.v1.network"]` section, parsed by `LoadConfigFromCubeletTOML` in [`config.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/config.go).