# How to Troubleshoot CubeSandbox Issues: Complete Diagnostic Guide

> Troubleshoot CubeSandbox issues by diagnosing XFS, network CIDR, cgroup v2, host-mount permissions, or egress policies. This guide pinpoints common failure points for quick resolution.

- Repository: [Tencent Cloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox)
- Tags: how-to-guide
- Published: 2026-07-14

---

**CubeSandbox failures almost always trace to five specific areas: XFS filesystem requirements, network CIDR conflicts, cgroup v2 controller configuration, host-mount permissions, or egress policy restrictions, each producing distinct error signatures in component logs.**

CubeSandbox is a multi-component system that isolates AI-agent workloads with hardware-level virtualization using KVM micro-VMs. When troubleshooting CubeSandbox issues, you must trace symptoms through the REST gateway, cluster orchestrator, node agent, and network stack to locate the root cause. Understanding the architecture helps you navigate from error messages in `journalctl` to the specific source files and configuration parameters that need adjustment.

## Core Components and Where Failures Originate

CubeSandbox consists of specialized services that each generate unique failure signatures when misconfigured.

| Component | Responsibility | Typical Symptoms | Source Reference |
| --- | --- | --- | --- |
| **CubeAPI** | High-concurrency REST gateway implementing the E2B-compatible API | HTTP 502/504, malformed JSON responses, API version mismatches | [`CubeAPI/src/routes.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeAPI/src/routes.rs) and [`CubeAPI/src/openapi.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeAPI/src/openapi.rs) |
| **CubeMaster** | Cluster orchestrator scheduling sandbox requests to Cubelets | Nodes reported *NotReady*, scheduler errors, database connection failures | `CubeMaster/pkg/base/dao` and [`configs/single-node/cubemaster.yaml`](https://github.com/TencentCloud/CubeSandbox/blob/main/configs/single-node/cubemaster.yaml) |
| **Cubelet** | Local node agent creating and monitoring sandbox instances | Sandbox creation hangs, missing TAP devices, out-of-disk errors | [`Cubelet/storage/pool.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/storage/pool.go) and [`configs/single-node/cubelet.yaml`](https://github.com/TencentCloud/CubeSandbox/blob/main/configs/single-node/cubelet.yaml) |
| **CubeVS** | eBPF-based virtual switch enforcing network isolation | Egress traffic bypasses security proxy, packets dropped unexpectedly | Architecture docs in [`docs/architecture/network.md`](https://github.com/TencentCloud/CubeSandbox/blob/main/docs/architecture/network.md) |
| **CubeEgress** | OpenResty gateway for domain allow-listing and audit logging | DNS resolution failures, "context deadline exceeded", credential leakage warnings | [`CubeEgress/lua/policy.lua`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeEgress/lua/policy.lua) |
| **CubeHypervisor & CubeShim** | KVM micro-VM manager and container-runtime shim | VM boot failures, "failed to attach vCPU", sandbox crashes | `CubeHypervisor` sources and [`CubeShim/docs/shimapi/README.md`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeShim/docs/shimapi/README.md) |

## Common CubeSandbox Failure Modes

Five specific misconfigurations account for the majority of deployment issues. Each generates predictable error patterns that you can verify against the source code.

### XFS Filesystem Requirements

The **Cubelet** stores writable layers under `/data/cubelet` and requires an **XFS filesystem with reflink support**. Deploying on ext4 causes the one-click pre-flight check to abort with the error *"not XFS"* as documented in [`docs/guide/troubleshooting/deployment.md`](https://github.com/TencentCloud/CubeSandbox/blob/main/docs/guide/troubleshooting/deployment.md).

To resolve this, mount a loop-back XFS image at the required path:

```bash

# Create a 200 GiB sparse file

dd if=/dev/zero of=/data/cubelet.img bs=1M count=0 seek=200000

# Format it as XFS with reflink support

mkfs.xfs -f /data/cubelet.img

# Create mount point and mount

mkdir -p /data/cubelet
mount -o loop /data/cubelet.img /data/cubelet

# Verify filesystem type

df -T /data/cubelet

```

The storage path is defined in [`configs/single-node/cubelet.yaml`](https://github.com/TencentCloud/CubeSandbox/blob/main/configs/single-node/cubelet.yaml).

### Network CIDR Overlap

The default sandbox CIDR (`192.168.0.0/18`) frequently clashes with host LAN ranges, causing template-creation timeouts or port-probing failures. This is documented in [`docs/guide/troubleshooting/local-network-cidr-conflict.md`](https://github.com/TencentCloud/CubeSandbox/blob/main/docs/guide/troubleshooting/local-network-cidr-conflict.md).

Change the CIDR and clean residual interfaces:

```bash

# Update the environment configuration

sed -i 's/^CUBE_SANDBOX_NETWORK_CIDR=.*/CUBE_SANDBOX_NETWORK_CIDR=10.200.0.0\/16/' /etc/cubesandbox/env

# Delete old TAP devices

ip link delete cube-dev

# Restart services

systemctl restart cubelet cube-master

```

Stale TAP devices (`cube-dev`, `z*`) indicate incomplete shutdowns and must be removed before restarting.

### cgroup v2 CPU Controller

Ubuntu and Debian images often lack the `cpu` controller enabled for child cgroups. This causes Cubelet CPU quotas to be ignored and generates `Invalid argument` errors when applying resource limits, as noted in [`docs/guide/troubleshooting/deployment.md`](https://github.com/TencentCloud/CubeSandbox/blob/main/docs/guide/troubleshooting/deployment.md).

Enable the controller via GRUB configuration:

```bash

# Add the unified cgroup flag

sudo sed -i 's/^GRUB_CMDLINE_LINUX="/GRUB_CMDLINE_LINUX="systemd.unified_cgroup_hierarchy=1 /' /etc/default/grub

# Update GRUB and reboot

sudo update-grub && sudo reboot

# Verify controller presence

ls /sys/fs/cgroup/cpu

```

Alternatively, use OpenCloudOS 9 or another distribution that enables the controller by default.

### Host-Mount Permission Denied

When a sandbox attempts to bind-mount a host directory, the kernel rejects the operation if the directory lacks `rwx` permissions for the **cubelet user**. This generates permission denied errors in [`Cubelet/storage/pool.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/storage/pool.go) operations.

Fix the permissions:

```bash

# Adjust ownership for the cubelet user

sudo chown -R cubelet:cubelet /var/www
sudo chmod -R 775 /var/www

# Verify access from cubelet perspective

sudo -u cubelet ls -l /var/www

```

Reference the host-mount permissions guide in [`docs/guide/troubleshooting/host-mount-permissions.md`](https://github.com/TencentCloud/CubeSandbox/blob/main/docs/guide/troubleshooting/host-mount-permissions.md) for volume management alternatives using the [`local.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/local.go) storage backend.

### Egress Policy Blocking

**CubeEgress** enforces strict domain allow-listing via [`CubeEgress/lua/policy.lua`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeEgress/lua/policy.lua). Sandboxes attempting to reach external services not in the policy generate *"403 Forbidden"* entries in the audit log and fail with connection timeouts.

Add the required domain to [`configs/single-node/network-agent.yaml`](https://github.com/TencentCloud/CubeSandbox/blob/main/configs/single-node/network-agent.yaml):

```yaml
egress:
  allowlist:
    - "api.openai.com"
    - "my.trusted.service"

```

Then reload the network agent:

```bash
systemctl restart network-agent

```

Audit records are written to `/var/log/cube-egress` in JSON format.

## Systematic Debugging Checklist

Follow this sequence to isolate CubeSandbox issues efficiently:

1. **Verify component health** – Run `systemctl status cube*` on every node and check `journalctl -u cubelet` for error codes.
2. **Check pre-flight diagnostics** – The one-click installer prints a summary; look specifically for `XFS`, `cgroup`, or `CIDR` warnings.
3. **Inspect network interfaces** – Execute `ip link show` to verify `cube-dev` and TAP devices only exist when Cube is running.
4. **Examine storage backends** – Confirm `/data/cubelet` is on XFS using `xfs_info /data/cubelet`.
5. **Review egress logs** – Search `/var/log/cube-egress` for the sandbox ID and blocked domains to identify policy violations.

## Summary

Troubleshooting CubeSandbox effectively requires mapping symptoms to the six core components: **CubeAPI**, **CubeMaster**, **Cubelet**, **CubeVS**, **CubeEgress**, and **CubeHypervisor**. The five most common issues involve:

- **XFS filesystem** requirements for storage pools ([`docs/guide/troubleshooting/deployment.md`](https://github.com/TencentCloud/CubeSandbox/blob/main/docs/guide/troubleshooting/deployment.md))
- **Network CIDR** conflicts with host LAN ranges
- **cgroup v2** CPU controller availability on Ubuntu/Debian
- **Host-mount permissions** requiring cubelet user access
- **Egress policy** domain allow-listing restrictions in [`CubeEgress/lua/policy.lua`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeEgress/lua/policy.lua)

Use the component-specific log locations and configuration files referenced in the TencentCloud/CubeSandbox repository to move from error symptoms to verified fixes.

## Frequently Asked Questions

### Why does CubeSandbox report "not XFS" during installation?

The Cubelet component stores writable layers under `/data/cubelet` and depends on XFS with reflink support for copy-on-write operations. ext4 or other filesystems lack the necessary reflink capabilities, causing the pre-flight check in the installation script to abort. Mount a dedicated XFS image at that path to satisfy the requirement.

### How do I fix "Invalid argument" errors when starting sandboxes on Ubuntu?

This error indicates the **cgroup v2 CPU controller** is not enabled for child cgroups, which prevents Cubelet from applying CPU quotas. Edit `/etc/default/grub` to add `systemd.unified_cgroup_hierarchy=1`, run `update-grub`, and reboot. The controller should then appear under `/sys/fs/cgroup/cpu`.

### Where does CubeSandbox store audit logs for blocked network connections?

The **CubeEgress** component writes JSON audit records to `/var/log/cube-egress`. When a sandbox attempts to reach a domain not in the allow-list, the component logs the sandbox ID, requested domain, and 403 status code in this file. Check these logs before modifying [`configs/single-node/network-agent.yaml`](https://github.com/TencentCloud/CubeSandbox/blob/main/configs/single-node/network-agent.yaml).

### What causes HTTP 502 errors from the CubeAPI gateway?

HTTP 502/504 errors typically indicate the **CubeAPI** (Rust-based REST gateway) cannot reach downstream services, usually **CubeMaster** or **Cubelet** instances. Check [`CubeAPI/src/routes.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeAPI/src/routes.rs) for timeout configurations, verify the cluster orchestrator is healthy via `systemctl status cubemaster`, and ensure network policies in **CubeVS** are not dropping packets between components.