How to Debug Sandbox Network Connectivity Issues with CubeVS: A Complete Guide

Debug CubeVS network issues by verifying the network-agent process health, inspecting Unix-socket API responses at /v1/network/get, confirming TAP device attachment via file descriptor handover, and validating routing inside the sandbox namespace using standard Linux tools.

CubeVS (Cube Virtual Sandbox) isolates workloads inside micro-VMs, with all network operations handled by the network-agent service. When sandbox connectivity fails, the root cause typically lies in the agent's configuration, the TAP device attachment, or policy enforcement. This guide walks through the exact debugging workflow based on the TencentCloud/CubeSandbox source code.

Understanding the CubeVS Network Architecture

According to network-agent/docs/ARCHITECTURE.md, the network-agent runs outside the sandbox and manages connectivity through a TAP device attached to the sandbox's virtual NIC. The agent exposes a Unix-socket REST API (/v1/network/*) and handles file descriptor (FD) handover for the TAP interface, making it the central control point for all sandbox traffic.

Common Network Failure Layers

Connectivity issues usually stem from four specific layers:

• Agent configuration: Misconfigured eth-name or state directory permissions defined in network-agent/docs/CONFIGURATION.md
• API communication: Socket permission issues or health-check failures at the /healthz endpoint
• Policy enforcement: Incorrect allow_out CIDRs or malformed rules documented in network-agent/docs/API.md
• TAP device layer: Missing file descriptor handover or namespace synchronization failures between the host and sandbox

Step-by-Step Debugging Workflow

Step 1: Verify the Network Agent Process

Confirm the agent is running and accessible:

ps -ef | grep network-agent

# Or run manually for debugging:

sudo ./bin/network-agent --eth-name=eth0

If the process is missing, the sandbox cannot initialize its network interface.

Step 2: Check Agent Health via Unix Socket

The agent exposes a health endpoint on a Unix socket:

curl --unix-socket /tmp/cube/network-agent.sock http://localhost/healthz

Expected output: {"status":"OK"}

If this fails, the agent is not listening or has permission issues preventing socket access.

Step 3: Inspect Applied Network Policies

Query the current policy for your sandbox using the /v1/network/get endpoint:

curl --unix-socket /tmp/cube/network-agent.sock \
     -X POST http://localhost/v1/network/get \
     -d '{"sandboxId":"sandbox-001"}' | jq .

Verify that allow_out includes your target destination CIDR. Missing entries here result in silent packet drops.

Step 4: Validate TAP Device Attachment

Ensure the TAP file descriptor was handed over correctly by calling the ensure endpoint:

curl --unix-socket /tmp/cube/network-agent.sock \
     -X POST http://localhost/v1/network/ensure \
     -d '{"sandboxId":"sandbox-001","network":{"allow_out":["0.0.0.0/0"]}}'

Check agent logs for "TAP fd handed to" messages to confirm the handover succeeded.

Step 5: Examine Sandbox Internal Network

Enter the sandbox using cubectl exec and verify interfaces:

cubectl exec sandbox-001 -- /bin/bash

# Inside sandbox:

ip a
ip r
ping -c 3 8.8.8.8

Look for missing eth0 interfaces, absent default routes, or destination unreachable errors indicating policy blocks.

Step 6: Review Agent Logs

Check for specific error patterns that indicate configuration failures:

journalctl -u network-agent -f

# Or for manual execution:

tail -f /var/log/network-agent/network-agent.log

Watch for "state directory not writable" or "network interface not found" errors that prevent proper initialization.

Practical Debugging Commands

Here are complete snippets for common debugging tasks:

Check agent health:

curl --unix-socket /tmp/cube/network-agent.sock http://localhost/healthz

Retrieve sandbox policy:

curl --unix-socket /tmp/cube/network-agent.sock \
     -X POST http://localhost/v1/network/get \
     -d '{"sandboxId":"sandbox-001"}' | jq .

Apply permissive policy (debug only):

curl --unix-socket /tmp/cube/network-agent.sock \
     -X POST http://localhost/v1/network/ensure \
     -d '{"sandboxId":"sandbox-001","network":{"allow_out":["0.0.0.0/0"]}}'

Enter sandbox namespace:

cubectl exec sandbox-001 -- /bin/bash
ip a && ip r

Key Source Files for Reference

• network-agent/docs/ARCHITECTURE.md: Describes the overall design, TAP device handling, and FD server mechanics
• network-agent/docs/API.md: Documents the REST/gRPC endpoint specifications (/v1/network/*) including request/response schemas
• network-agent/docs/CONFIGURATION.md: Lists command-line flags like --eth-name and TOML configuration sections for state directories
• examples/route-aware-egress/README.md: Provides concrete routing policy examples and their effects on outbound traffic
• docker/Dockerfile.cube-base: Shows default socket paths and image configuration used by the agent

Summary

• Debug connectivity by verifying the network-agent process is running and responding on its Unix socket at /tmp/cube/network-agent.sock
• Use /v1/network/get to inspect active policies and confirm allow_out CIDRs permit your traffic
• Validate TAP device attachment through file descriptor handover logs and the /v1/network/ensure endpoint
• Enter the sandbox with cubectl exec to check internal routing, interfaces, and DNS resolution
• Review agent logs for permission errors or missing interface configurations that prevent proper network setup

Frequently Asked Questions

Why does my sandbox have no eth0 interface?

This indicates the TAP device was not attached. Verify the network-agent handed over the file descriptor by checking the logs for "TAP fd handed to" messages and ensure the /v1/network/ensure endpoint returned successfully without permission errors.

How do I check if network policies are blocking my traffic?

Query the /v1/network/get endpoint via the Unix socket to retrieve the active allow_out CIDRs and rules defined in the policy. If your destination is not included in the allowed list, the agent will drop the packets at the policy enforcement layer before they reach the TAP device.

What permissions does the network-agent socket require?

The Unix socket at /tmp/cube/network-agent.sock must be readable and writable by the user running the Cube daemon or your debugging commands. Permission denied errors typically indicate the agent started with different user privileges than the client attempting to connect.

Can I debug the network-agent without systemd?

Yes. Run the binary directly in the foreground with verbose logging using sudo ./bin/network-agent --eth-name=eth0. This streams logs to stdout and allows you to see real-time FD handover and policy application events without relying on journalctl.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}