How to Configure Per-Sandbox Network Policies with CubeVS

CubeVS enforces Layer 7 egress rules per sandbox by serializing a NetworkPolicy object into the sandbox metadata, which the network-agent pushes to the CubeEgress control plane and enforces via eBPF maps in the TAP plugin.

TencentCloud/CubeSandbox provides granular network isolation through CubeVS (Cube Virtual Sandbox), allowing administrators to define fine-grained outbound traffic rules that override cluster-wide defaults. Configuring these policies requires understanding the data flow from the SDK through the network-agent to the eBPF-based enforcement layer. This guide explains the exact mechanism using source file references from the repository.

Understanding CubeVS Network Policy Architecture

CubeVS implements per-sandbox network policies through three coordinated components. The network-agent receives policy definitions from the Cube-API and maintains an in-memory policyKnown flag to track whether a specific sandbox has a custom policy. According to network-agent/internal/service/local_service.go, this flag prevents stale policies from being reapplied during sandbox recovery.

The CubeEgress control plane handles the actual rule distribution. When the network-agent pushes a policy, it sends a JSON payload to CubeEgress’s administrative endpoint, which then compiles the rules into eBPF programs. The Cubelet TAP plugin (Cubelet/network/plugin_tap.go) reads these rules from eBPF maps and intercepts every packet exiting the sandbox, applying allow or deny logic at the L7 level.

Network Policy Structure and Fields

When creating a sandbox, the SDK accepts a NetworkPolicy description that maps to the internal representation in network-agent/internal/service/config.go. The policy supports the following fields:

• dns – Optional list of DNS servers for the sandbox to use.
• allow_out – CIDR ranges permitted for outbound traffic (comma-separated or list format).
• deny_out – CIDR ranges explicitly blocked from outbound traffic.
• net_type – Network classification (e.g., public or private).

If no policy is supplied, the system returns an empty map, instructing CubeEgress that the sandbox has no L7 policy and should fall back to cluster-wide defaults.

Configuration Flow from SDK to eBPF

The policy enforcement path follows a strict sequence from user definition to kernel-level filtering:

1. SDK Serialization – The Go, Python, or Node.js SDK encodes the NetworkPolicy into the sandbox creation request’s metadata field.
2. API Processing – Cube-API forwards the metadata to the network-agent running on the target node.
3. Config Building – The agent parses the metadata into an internal Config struct in network-agent/internal/service/config.go, setting policyKnown to true.
4. Policy Push – The agent’s cubeegress_push.go issues a PUT /admin/v1/policies/<sandbox_ip> request with a JSON body defined in network-agent/internal/cubeegress/wire.go.
5. eBPF Installation – CubeEgress installs the rules into eBPF maps that the TAP driver references.
6. Runtime Enforcement – The Cubelet TAP plugin (Cubelet/network/plugin_tap.go) checks every egress packet against the loaded policy, dropping connections that match a deny_out rule or fail to match an allow_out rule.

Implementing Per-Sandbox Policies with SDKs

Go SDK

The Go SDK defines the policy structure in sdk/go/policy.go. Pass the serialized policy via the metadata map when calling Sandbox.Create:

import (
    "github.com/TencentCloud/CubeSandbox/sdk/go"
    "github.com/TencentCloud/CubeSandbox/sdk/go/policy"
)

func main() {
    client := cubesandbox.NewClient("http://cube-api.example.com")
    
    netPol := policy.NetworkPolicy{
        AllowOut: []string{"10.0.0.0/8"},
        DenyOut:  []string{"0.0.0.0/0"},
    }
    
    sandbox, err := client.Sandbox.Create(
        cubesandbox.SandboxCreateRequest{
            Image:    "ubuntu:20.04",
            Metadata: map[string]string{"network-policy": netPol.String()},
        })
    // The sandbox now runs with the specified egress restrictions
}

Python SDK

In sdk/python/cubesandbox/_policy.py, the NetworkPolicy class provides the same functionality:

from cubesandbox import CubeSandbox, NetworkPolicy

cs = CubeSandbox("http://cube-api.example.com")
policy = NetworkPolicy(
    allow_out=["10.0.0.0/8"],
    deny_out=["0.0.0.0/0"]
)

sandbox = cs.sandbox.create(
    image="ubuntu:20.04",
    metadata={"network-policy": policy.to_json()}
)

Node.js SDK

The TypeScript implementation in sdk/node/src/policy.ts exposes a similar interface:

import { CubeSandbox } from "@tencentcloud/cubesandbox";
import { NetworkPolicy } from "@tencentcloud/cubesandbox/policy";

const cs = new CubeSandbox("http://cube-api.example.com");
const policy = new NetworkPolicy({
  allowOut: ["10.0.0.0/8"],
  denyOut:  ["0.0.0.0/0"]
});

const sandbox = await cs.sandbox.create({
  image: "ubuntu:20.04",
  metadata: { "network-policy": policy.toJSON() }
});

Legacy Metadata Key

Older clients may use the simplified "network-policy" string value. The network-agent treats values like "deny-all" as shortcuts and automatically expands them into full policies that block all outbound traffic. This behavior is tested in sdk/python/tests/test_sandbox.py where meta = {"network-policy": "deny-all"} is validated.

Runtime Enforcement and Cluster Defaults

The Cubelet TAP plugin reads node-level annotations to establish baseline behaviors when no per-sandbox policy exists. As implemented in Cubelet/network/plugin_tap.go, the plugin checks for constants defined in Cubelet/pkg/constants/const.go:

• MasterAnnotationNetworkPolicyBlockAll – When set to "true", blocks all egress by default.
• MasterAnnotationNetworkPolicyAllowPublicServices – When set to "true", permits traffic to public service endpoints.
• MasterAnnotationNetworkPolicyDefault – Applies the repository-wide default policy.

Per-sandbox policies always override these defaults because the network-agent only pushes a configuration to CubeEgress when policyKnown is true. If a sandbox is restored after a node reboot and the recovered state lacks a known policy, the agent logs a warning and leaves the policy untouched, preventing accidental application of stale rules.

Summary

• CubeVS enforces per-sandbox egress rules through the network-agent, CubeEgress, and Cubelet TAP plugin chain.
• The NetworkPolicy struct accepts allow_out, deny_out, dns, and net_type fields to define traffic rules.
• Policies are serialized into metadata during sandbox creation and pushed to CubeEgress via PUT /admin/v1/policies/<sandbox_ip> as defined in wire.go.
• Go, Python, and Node.js SDKs provide native policy objects that handle JSON serialization automatically.
• Per-sandbox policies override cluster defaults because the policyKnown flag ensures only explicitly defined policies are pushed to the eBPF enforcement layer.

Frequently Asked Questions

What happens if I don't specify a network policy when creating a sandbox?

If no NetworkPolicy is provided, the SDK sends an empty configuration map. CubeEgress interprets this as an absence of L7 policy, causing the sandbox to inherit cluster-wide defaults from Cubelet annotations (such as block-all or allow-public-services modes).

How do per-sandbox policies interact with cluster-wide network defaults?

Per-sandbox policies take precedence over cluster defaults. The network-agent only pushes a policy to CubeEgress when the policyKnown flag is true. If this flag is false, the agent does not transmit any rules, allowing the TAP plugin to apply the default behaviors configured via node annotations.

What is the JSON format expected by the CubeEgress admin API?

According to network-agent/internal/cubeegress/wire.go, the payload must include a policy_id (set to the sandbox IP) and a rules array containing objects with id (either "allow_out" or "deny_out") and cidr arrays listing the IP ranges.

Can the network-agent recover policies after a node restart?

During sandbox restoration, the network-agent checks its internal state in network-agent/internal/service/local_service.go. If the recovered state lacks a known policy (policyKnown is false), the agent does not replay any configuration and logs a warning, ensuring that unknown or stale policies are not accidentally enforced.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}