How to Troubleshoot CubeSandbox Issues: Complete Diagnostic Guide

CubeSandbox failures almost always trace to five specific areas: XFS filesystem requirements, network CIDR conflicts, cgroup v2 controller configuration, host-mount permissions, or egress policy restrictions, each producing distinct error signatures in component logs.

CubeSandbox is a multi-component system that isolates AI-agent workloads with hardware-level virtualization using KVM micro-VMs. When troubleshooting CubeSandbox issues, you must trace symptoms through the REST gateway, cluster orchestrator, node agent, and network stack to locate the root cause. Understanding the architecture helps you navigate from error messages in journalctl to the specific source files and configuration parameters that need adjustment.

Core Components and Where Failures Originate

CubeSandbox consists of specialized services that each generate unique failure signatures when misconfigured.

Component	Responsibility	Typical Symptoms	Source Reference
CubeAPI	High-concurrency REST gateway implementing the E2B-compatible API	HTTP 502/504, malformed JSON responses, API version mismatches	CubeAPI/src/routes.rs and CubeAPI/src/openapi.rs
CubeMaster	Cluster orchestrator scheduling sandbox requests to Cubelets	Nodes reported NotReady, scheduler errors, database connection failures	CubeMaster/pkg/base/dao and configs/single-node/cubemaster.yaml
Cubelet	Local node agent creating and monitoring sandbox instances	Sandbox creation hangs, missing TAP devices, out-of-disk errors	Cubelet/storage/pool.go and configs/single-node/cubelet.yaml
CubeVS	eBPF-based virtual switch enforcing network isolation	Egress traffic bypasses security proxy, packets dropped unexpectedly	Architecture docs in docs/architecture/network.md
CubeEgress	OpenResty gateway for domain allow-listing and audit logging	DNS resolution failures, "context deadline exceeded", credential leakage warnings	CubeEgress/lua/policy.lua
CubeHypervisor & CubeShim	KVM micro-VM manager and container-runtime shim	VM boot failures, "failed to attach vCPU", sandbox crashes	CubeHypervisor sources and CubeShim/docs/shimapi/README.md

Common CubeSandbox Failure Modes

Five specific misconfigurations account for the majority of deployment issues. Each generates predictable error patterns that you can verify against the source code.

XFS Filesystem Requirements

The Cubelet stores writable layers under /data/cubelet and requires an XFS filesystem with reflink support. Deploying on ext4 causes the one-click pre-flight check to abort with the error "not XFS" as documented in docs/guide/troubleshooting/deployment.md.

To resolve this, mount a loop-back XFS image at the required path:

# Create a 200 GiB sparse file

dd if=/dev/zero of=/data/cubelet.img bs=1M count=0 seek=200000

# Format it as XFS with reflink support

mkfs.xfs -f /data/cubelet.img

# Create mount point and mount

mkdir -p /data/cubelet
mount -o loop /data/cubelet.img /data/cubelet

# Verify filesystem type

df -T /data/cubelet

The storage path is defined in configs/single-node/cubelet.yaml.

Network CIDR Overlap

The default sandbox CIDR (192.168.0.0/18) frequently clashes with host LAN ranges, causing template-creation timeouts or port-probing failures. This is documented in docs/guide/troubleshooting/local-network-cidr-conflict.md.

Change the CIDR and clean residual interfaces:

# Update the environment configuration

sed -i 's/^CUBE_SANDBOX_NETWORK_CIDR=.*/CUBE_SANDBOX_NETWORK_CIDR=10.200.0.0\/16/' /etc/cubesandbox/env

# Delete old TAP devices

ip link delete cube-dev

# Restart services

systemctl restart cubelet cube-master

Stale TAP devices (cube-dev, z*) indicate incomplete shutdowns and must be removed before restarting.

cgroup v2 CPU Controller

Ubuntu and Debian images often lack the cpu controller enabled for child cgroups. This causes Cubelet CPU quotas to be ignored and generates Invalid argument errors when applying resource limits, as noted in docs/guide/troubleshooting/deployment.md.

Enable the controller via GRUB configuration:

# Add the unified cgroup flag

sudo sed -i 's/^GRUB_CMDLINE_LINUX="/GRUB_CMDLINE_LINUX="systemd.unified_cgroup_hierarchy=1 /' /etc/default/grub

# Update GRUB and reboot

sudo update-grub && sudo reboot

# Verify controller presence

ls /sys/fs/cgroup/cpu

Alternatively, use OpenCloudOS 9 or another distribution that enables the controller by default.

Host-Mount Permission Denied

When a sandbox attempts to bind-mount a host directory, the kernel rejects the operation if the directory lacks rwx permissions for the cubelet user. This generates permission denied errors in Cubelet/storage/pool.go operations.

Fix the permissions:

# Adjust ownership for the cubelet user

sudo chown -R cubelet:cubelet /var/www
sudo chmod -R 775 /var/www

# Verify access from cubelet perspective

sudo -u cubelet ls -l /var/www

Reference the host-mount permissions guide in docs/guide/troubleshooting/host-mount-permissions.md for volume management alternatives using the local.go storage backend.

Egress Policy Blocking

CubeEgress enforces strict domain allow-listing via CubeEgress/lua/policy.lua. Sandboxes attempting to reach external services not in the policy generate "403 Forbidden" entries in the audit log and fail with connection timeouts.

Add the required domain to configs/single-node/network-agent.yaml:

egress:
  allowlist:
    - "api.openai.com"
    - "my.trusted.service"

Then reload the network agent:

systemctl restart network-agent

Audit records are written to /var/log/cube-egress in JSON format.

Systematic Debugging Checklist

Follow this sequence to isolate CubeSandbox issues efficiently:

1. Verify component health – Run systemctl status cube* on every node and check journalctl -u cubelet for error codes.
2. Check pre-flight diagnostics – The one-click installer prints a summary; look specifically for XFS, cgroup, or CIDR warnings.
3. Inspect network interfaces – Execute ip link show to verify cube-dev and TAP devices only exist when Cube is running.
4. Examine storage backends – Confirm /data/cubelet is on XFS using xfs_info /data/cubelet.
5. Review egress logs – Search /var/log/cube-egress for the sandbox ID and blocked domains to identify policy violations.

Summary

Troubleshooting CubeSandbox effectively requires mapping symptoms to the six core components: CubeAPI, CubeMaster, Cubelet, CubeVS, CubeEgress, and CubeHypervisor. The five most common issues involve:

• XFS filesystem requirements for storage pools (docs/guide/troubleshooting/deployment.md)
• Network CIDR conflicts with host LAN ranges
• cgroup v2 CPU controller availability on Ubuntu/Debian
• Host-mount permissions requiring cubelet user access
• Egress policy domain allow-listing restrictions in CubeEgress/lua/policy.lua

Use the component-specific log locations and configuration files referenced in the TencentCloud/CubeSandbox repository to move from error symptoms to verified fixes.

Frequently Asked Questions

Why does CubeSandbox report "not XFS" during installation?

The Cubelet component stores writable layers under /data/cubelet and depends on XFS with reflink support for copy-on-write operations. ext4 or other filesystems lack the necessary reflink capabilities, causing the pre-flight check in the installation script to abort. Mount a dedicated XFS image at that path to satisfy the requirement.

How do I fix "Invalid argument" errors when starting sandboxes on Ubuntu?

This error indicates the cgroup v2 CPU controller is not enabled for child cgroups, which prevents Cubelet from applying CPU quotas. Edit /etc/default/grub to add systemd.unified_cgroup_hierarchy=1, run update-grub, and reboot. The controller should then appear under /sys/fs/cgroup/cpu.

Where does CubeSandbox store audit logs for blocked network connections?

The CubeEgress component writes JSON audit records to /var/log/cube-egress. When a sandbox attempts to reach a domain not in the allow-list, the component logs the sandbox ID, requested domain, and 403 status code in this file. Check these logs before modifying configs/single-node/network-agent.yaml.

What causes HTTP 502 errors from the CubeAPI gateway?

HTTP 502/504 errors typically indicate the CubeAPI (Rust-based REST gateway) cannot reach downstream services, usually CubeMaster or Cubelet instances. Check CubeAPI/src/routes.rs for timeout configurations, verify the cluster orchestrator is healthy via systemctl status cubemaster, and ensure network policies in CubeVS are not dropping packets between components.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}