# Debugging Sandbox Startup Failures and VM Crashes in CubeSandbox: A Complete Guide

> Troubleshoot CubeSandbox startup failures and VM crashes. Learn to check systemd journals, network-agent logs, and Firecracker VM logs for precise root cause analysis.

- Repository: [Tencent Cloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox)
- Tags: how-to-guide
- Published: 2026-07-15

---

**You can diagnose CubeSandbox startup failures by checking systemd journals for bootstrap errors, verifying TAP pool warmup in network-agent logs, and inspecting Firecracker VM creation logs in `/data/log/cubelet/` to pinpoint exactly which lifecycle phase failed.**

CubeSandbox orchestrates lightweight MicroVMs using Firecracker, attaching networking via network-agent and storage via cubelet, with the `envd` data-plane handling runtime operations. When a sandbox fails to start or crashes unexpectedly, the root cause could exist in any of these distinct layers—from missing configuration files to exhausted TAP device pools or corrupt snapshot metadata. Understanding the specific failure points in the CubeSandbox architecture allows operators to quickly identify whether the issue lies in bootstrap validation, VM creation, or post-startup health monitoring.

## Understanding the Sandbox Startup Lifecycle

CubeSandbox initializes a sandbox through several distinct phases, each with specific failure modes and log locations. The process begins with bootstrap validation in [`deploy/one-click/scripts/common.sh`](https://github.com/TencentCloud/CubeSandbox/blob/main/deploy/one-click/scripts/common.sh), where the `validate_cubelet_cubecow_deps` function verifies required binaries and environment variables like `CUBE_SANDBOX_DOMAIN`.

Once bootstrap passes, the system warms up networking resources. The [`network-agent/internal/service/tap_lifecycle.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/network-agent/internal/service/tap_lifecycle.go) pre-creates TAP devices (default pool of 500) before Firecracker attempts VM creation. Finally, the `envd` binary must start and respond on port `49983` before the sandbox reaches a healthy state.

## Common Failure Points and How to Diagnose Them

### Bootstrap and Configuration Validation

Early failures occur when required environment variables are missing or configuration files are malformed. The cubelet service validates dependencies during initialization and aborts if [`config.toml`](https://github.com/TencentCloud/CubeSandbox/blob/main/config.toml) is invalid or required binaries are missing.

Check for these issues by examining the systemd journal for the cubelet service:

```bash
journalctl -u cube-sandbox-cubelet.service -b

```

Also inspect the runtime log directory at `/data/log/cubelet/` for bootstrap error messages. Common indicators include missing `CUBE_API_PUBLIC_HOST` or `CUBE_PROXY_HOST_PORT` settings in your `.one-click.env` file.

### Network Agent TAP Pool Initialization

If the network-agent cannot warm up its TAP device pool, sandbox creation will stall indefinitely. The `warmupTapPoolBackground` routine in [`network-agent/internal/service/tap_lifecycle.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/network-agent/internal/service/tap_lifecycle.go) handles this pre-allocation, and failures appear as explicit errors in the network-agent logs.

Monitor these issues in real-time with:

```bash
journalctl -u cube-sandbox-network-agent.service -f

```

Races between network-agent and cubelet can cause intermittent startup failures if the TAP pool is not ready when VM creation begins.

### VM Creation via Firecracker

Firecracker MicroVM creation fails when kernel files are missing, disks are full, or CPU features like XSAVE are unavailable. The [`cubelet/services/cubebox/update.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/cubelet/services/cubebox/update.go) file handles these operations and returns `VMCreateFailed` errors when Firecracker cannot launch.

Locate these failures in `/data/log/cubelet/` by searching for "firecracker" and "VM creation" keywords. The [`Cubelet/storage/local.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/storage/local.go) component also writes relevant error details during storage attachment failures.

### Envd Data-Plane Startup

The `envd` data-plane must listen on port `49983` and respond to health checks before the sandbox becomes ready. If the binary is missing from the image or fails to start, the readiness probe defined in [`docs/guide/troubleshooting/templates.md`](https://github.com/TencentCloud/CubeSandbox/blob/main/docs/guide/troubleshooting/templates.md) will timeout.

Check both cubelet and envd service statuses simultaneously:

```bash
journalctl -u cube-sandbox-cubelet.service -u cube-sandbox-envd.service

```

### Snapshot and Rollback State

After VM crashes, leftover snapshot metadata can prevent new sandbox creation. The [`CubeMaster/pkg/templatecenter/snapshot_ops.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/templatecenter/snapshot_ops.go) reconciles snapshots on startup, logging "reconcileSnapshotDefinitionTimeouts" when it encounters corrupt state.

Inspect the CubeMaster logs to identify these issues:

```bash
journalctl -u cube-sandbox-cubemaster.service

```

### Post-Startup VM Crashes

VMs may exit after successful launch due to OOM conditions or seccomp violations. The [`cubelet/services/cubebox/update.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/cubelet/services/cubebox/update.go) watches VMM processes and writes "VM crashed" entries when it detects unexpected exits. Check for automatic snapshot cleanup triggers in the same logs.

If crashes persist, examine potential core dumps in `/run/firecracker/` and verify the Firecracker PID status.

## Step-by-Step Debugging Workflow

Follow this systematic approach to isolate startup failures:

1. **Check systemd status** across all components: `systemctl status cube-sandbox-<component>.service`
2. **Read startup logs** from the journal for bootstrap phase errors, then check `/data/log/<component>/` for runtime details
3. **Identify the failure point** by searching logs for keywords: *bootstrap*, *tap*, *firecracker*, *envd*, or *snapshot*
4. **Verify configuration** in `.one-click.env` ensures all required variables (`CUBE_SANDBOX_DOMAIN`, `CUBE_API_PUBLIC_HOST`) are exported before services start
5. **Enable debug logging** by setting `CUBE_LOG_LEVEL=debug` before sandbox creation—note this does not affect already-running instances
6. **Inspect VM state** by checking Firecracker sockets and potential core files when crashes occur

## Code Examples for Programmatic Debugging

Detect sandbox creation failures using the Go SDK by checking state after initialization:

```go
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()

sb, err := client.NewSandbox(ctx, "tpl-my-image")
if err != nil {
    log.Fatalf("sandbox creation failed: %v", err) // see Cubelet logs for root cause
}
info, err := sb.GetInfo(ctx)
if err != nil {
    log.Fatalf("cannot retrieve sandbox info: %v", err)
}
if info.State != "running" {
    log.Fatalf("sandbox %s is not running (state=%s)", sb.SandboxID, info.State)
}

```

Enable debug logging across services before restarting:

```bash
export CUBE_LOG_LEVEL=debug
systemctl restart cube-sandbox-{cubelet,network-agent,cube-api}.service
journalctl -u cube-sandbox-cubelet.service -f

```

Handle VM crashes programmatically by checking for `ErrVMCrash`:

```go
err := sb.Wait(ctx) // blocks until sandbox exits
if err != nil && errors.Is(err, cubesandbox.ErrVMCrash) {
    // retrieve crash details from cubelet logs
    logPath := "/data/log/cubelet/" + sb.SandboxID + ".log"
    data, _ := os.ReadFile(logPath)
    fmt.Println(string(data))
}

```

## Summary

- **Bootstrap failures** appear in `journalctl -u cube-sandbox-cubelet.service` and `/data/log/cubelet/` when environment variables or binaries are missing
- **TAP pool issues** in [`network-agent/internal/service/tap_lifecycle.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/network-agent/internal/service/tap_lifecycle.go) cause networking stalls visible in `journalctl -u cube-sandbox-network-agent.service`
- **VM creation errors** return `VMCreateFailed` from [`cubelet/services/cubebox/update.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/cubelet/services/cubebox/update.go) when Firecracker cannot launch due to missing kernel files or CPU feature gaps
- **Envd health check timeouts** occur when port `49983` is unreachable, documented in [`docs/guide/troubleshooting/templates.md`](https://github.com/TencentCloud/CubeSandbox/blob/main/docs/guide/troubleshooting/templates.md)
- **Snapshot corruption** after crashes is reconciled by [`CubeMaster/pkg/templatecenter/snapshot_ops.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/templatecenter/snapshot_ops.go), with logs in the CubeMaster systemd service
- **Debug mode** requires setting `CUBE_LOG_LEVEL=debug` before sandbox creation, not during runtime

## Frequently Asked Questions

### How do I know if a sandbox failure is caused by configuration or VM creation?

Check the cubelet systemd journal first. If you see errors from `validate_cubelet_cubecow_deps` in [`deploy/one-click/scripts/common.sh`](https://github.com/TencentCloud/CubeSandbox/blob/main/deploy/one-click/scripts/common.sh) or messages about missing `CUBE_SANDBOX_DOMAIN`, the failure is bootstrap-related. If logs show "firecracker" or "VMCreateFailed" from [`cubelet/services/cubebox/update.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/cubelet/services/cubebox/update.go), the issue occurs during MicroVM initialization, typically due to missing kernel files or insufficient disk space.

### Where can I find logs for the envd data-plane when health checks fail?

Envd logs are available through the systemd journal by querying both services: `journalctl -u cube-sandbox-cubelet.service -u cube-sandbox-envd.service`. The readiness probe expects a response on `/:49983/health` as defined in [`docs/guide/troubleshooting/templates.md`](https://github.com/TencentCloud/CubeSandbox/blob/main/docs/guide/troubleshooting/templates.md). If envd is missing from the container image or fails to bind to this port, the logs will show connection refused or timeout errors.

### What causes VM crashes after successful sandbox startup?

Post-launch crashes typically result from resource exhaustion (OOM) or security policy violations (seccomp). The [`cubelet/services/cubebox/update.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/cubelet/services/cubebox/update.go) file monitors the VMM process and writes "VM crashed" entries to `/data/log/cubelet/` when the Firecracker process exits unexpectedly. Inspect `/run/firecracker/` for core dumps and check if automatic snapshot cleanup was triggered to handle residual state.

### How do I debug snapshot-related startup failures?

Snapshot corruption or leftover files from previous crashes are detected by [`CubeMaster/pkg/templatecenter/snapshot_ops.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/templatecenter/snapshot_ops.go), which logs "reconcileSnapshotDefinitionTimeouts" during reconciliation. Check `journalctl -u cube-sandbox-cubemaster.service` for these specific timeout messages, indicating that snapshot metadata needs manual cleanup or repair before new sandboxes can launch successfully.