How CubeEgress Secures Outbound Traffic in CubeSandbox: A Four-Stage Defense

CubeEgress secures outbound traffic through a four-stage defense-in-depth chain that combines kernel-level eBPF filtering, TPROXY redirection, and an OpenResty-based L7 proxy to enforce fine-grained policies while keeping secrets out of the sandbox.

CubeEgress is the per-host transparent egress gateway in the TencentCloud/CubeSandbox repository that enforces Layer 7 security policies on every outbound request. Unlike simple firewall rules, it implements an inline proxy architecture that intercepts TCP traffic destined for ports 80 and 443, allowing administrators to define precise allow-lists based on host, SNI, path, and HTTP method while ensuring sensitive credentials never reach the sandbox environment.

The Four-Stage Security Architecture

Stage 1: Policy Definition via CubeMaster

Users express security policies through the SDK or API, which CubeMaster stores as protobuf CubeNetworkConfig messages. These configurations include fields like allow_out, allow_out_v2, and detailed rules arrays that specify allowed CIDRs, DNS entries, and granular L7 match criteria including host, SNI, path, scheme, and action types (audit, inject, deny). The policies reside in the template database before being pushed to individual nodes.

Stage 2: Kernel-Side Enforcement with eBPF

The CubeVS eBPF program extracts only the network-reachable components of each L7 rule—specifically match.host and match.sni—and populates allow_out_v2 with an L7_REQUIRED flag. When a sandbox initiates a TCP connection to ports 80 or 443 matching a flagged entry, the kernel marks the socket for inspection and redirects it via TPROXY rules to the CubeEgress container. Non-HTTP traffic bypasses the proxy entirely, following a fast-path NAT route. This selective capture logic is documented in docs/guide/network-policy.md and implemented in the eBPF map layout within the source code.

Stage 3: L7 Proxy Enforcement

CubeEgress runs an OpenResty (nginx + Lua) proxy that receives the redirected TCP streams on the host network. The Lua modules—specifically CubeEgress/lua/policy.lua and CubeEgress/lua/access_phase.lua—parse HTTP and HTTPS traffic (extracting SNI for TLS) and apply the complete rule set. When a request violates policy, the proxy returns an immediate HTTP 403 without contacting the upstream service, guaranteeing that no data leaves the cluster. Valid requests proceed only after matching an allow rule.

Stage 4: Auditing and Credential Protection

Every request processed by CubeEgress generates structured JSON logs at /data/log/cube-egress/access.jsonl, capturing timestamps, sandbox IDs, rule hits, and redacted request/response data. The CubeEgress/lua/audit.lua module handles this logging. Critically, secrets are never exposed to the sandbox: the proxy performs credential injection by adding required headers (e.g., Authorization, x-api-key) after the sandbox dispatches the request. The per-sandbox CA used for TLS interception is generated by CubeEgress/gen-ca.sh and baked into the container image, with signing logic residing in CubeEgress/lua/cert_signer.lua.

Key Security Guarantees

CubeEgress provides several hardened security guarantees:

• Fail-Closed Startup: Until the policy fully loads and bootstrap_status equals "ready", CubeEgress rejects all non-audit traffic with a 403 status, eliminating windows where outbound traffic might bypass controls. This behavior is noted in docs/zh/changelog/v0.5.0.md.

• Selective Traffic Capture: Only TCP flows to ports 80 and 443 with the L7_REQUIRED flag enter the proxy. All other traffic, including UDP and non-HTTP ports, follows kernel NAT directly, minimizing performance overhead.

• Policy-Driven Routing: The eBPF maps egress_sessions and ingress_sessions track connection state, ensuring only packets belonging to allowed sessions forward through the data plane, preventing spoofed packets from bypassing the proxy. See the architecture diagram in docs/zh/architecture/network.md.

• Credential Isolation: Secrets stored in CubeMaster or external vaults are injected by the proxy at the final hop, satisfying the requirement that sensitive credentials never leave the host environment.

Configuration Examples

The following examples demonstrate how to define L7 rules that leverage CubeEgress's credential injection and auditing capabilities.

Python SDK:

# Define an L7 rule that injects an API key without exposing it to the sandbox

sandbox = client.sandbox.create(
    name="my-sandbox",
    allow_internet_access=True,
    network={
        "rules": [{
            "match": {"host": "api.openai.com"},
            "action": {"inject": {"headers": {"Authorization": "Bearer ${API_KEY}"}}}
        }]
    })

Go SDK:

// Same rule with audit logging enabled
network := cubenet.NetworkConfig{
    Rules: []cubenet.Rule{{
        Match: cubenet.Match{Host: "api.anthropic.com"},
        Action: cubenet.Action{Inject: &cubenet.Inject{
            Headers: map[string]string{"x-api-key": "${ANTHROPIC_API_KEY}"},
        }, Audit: true},
    }},
}
client.CreateSandbox(ctx, "my-sandbox", network, true)

Verification:

# Verify that requests without matching rules are blocked

curl -s https://example.com  # Returns 403 Forbidden - CubeEgress

Core Implementation Files

Component	File Path	Purpose
Proxy Configuration	CubeEgress/nginx.conf	OpenResty entry-point configuration
Policy Engine	CubeEgress/lua/policy.lua	Core rule matching (allow/deny/audit/inject)
Access Handler	CubeEgress/lua/access_phase.lua	HTTP/HTTPS request processing logic
Audit Logger	CubeEgress/lua/audit.lua	JSONL logging to /data/log/cube-egress/access.jsonl
Certificate Authority	CubeEgress/gen-ca.sh	Root CA generation for per-sandbox leaf certificates
TLS Signer	CubeEgress/lua/cert_signer.lua	Generates per-sandbox TLS certificates
TPROXY Setup	CubeEgress/scripts/cube-proxy-iptables-init.sh	Kernel rules for L7 traffic redirection

Summary

• CubeEgress implements a four-stage chain spanning policy definition, eBPF kernel filtering, OpenResty L7 proxying, and secure auditing.
• Traffic to ports 80/443 with the L7_REQUIRED flag is transparently redirected via TPROXY to the CubeEgress container.
• The fail-closed startup behavior ensures no traffic bypasses policy enforcement during initialization.
• Credential injection ensures secrets never reach the sandbox, with headers added by the proxy after the request originates.
• Implementation relies on specific Lua modules in CubeEgress/lua/ and eBPF programs that maintain session state in egress_sessions maps.

Frequently Asked Questions

What happens if CubeEgress fails to load its policy?

If the bootstrap_status is not "ready", CubeEgress operates in fail-closed mode, returning HTTP 403 Forbidden for all non-audit traffic. This prevents any outbound connections from proceeding until the security policy is fully loaded and validated, as implemented in the OpenResty startup logic.

How does CubeEgress handle HTTPS traffic without breaking TLS?

CubeEgress generates per-sandbox leaf certificates signed by a dedicated CA created via CubeEgress/gen-ca.sh. The proxy uses SNI (Server Name Indication) to inspect the target host for routing decisions, terminating TLS when necessary for L7 inspection and then re-encrypting traffic to the upstream. The certificate signing logic resides in CubeEgress/lua/cert_signer.lua.

Can non-HTTP traffic bypass the CubeEgress proxy?

Yes. Only TCP traffic destined for ports 80 and 443 that matches an L7_REQUIRED flag in the eBPF maps is redirected via TPROXY to the OpenResty proxy. All other traffic—including UDP, plain IP, and TCP on non-standard ports—follows the kernel NAT fast-path and never touches the user-space proxy, as documented in docs/guide/network-policy.md.

Where are the audit logs stored and what do they contain?

Audit logs are written as structured JSON Lines to /data/log/cube-egress/access.jsonl. Each entry includes the timestamp, sandbox ID, matched rule, destination host, and optional redacted request/response details. The CubeEgress/lua/audit.lua module formats these entries, ensuring comprehensive observability without exposing injected credentials.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}