Debugging Sandbox Startup Failures and VM Crashes in CubeSandbox: A Complete Guide

You can diagnose CubeSandbox startup failures by checking systemd journals for bootstrap errors, verifying TAP pool warmup in network-agent logs, and inspecting Firecracker VM creation logs in /data/log/cubelet/ to pinpoint exactly which lifecycle phase failed.

CubeSandbox orchestrates lightweight MicroVMs using Firecracker, attaching networking via network-agent and storage via cubelet, with the envd data-plane handling runtime operations. When a sandbox fails to start or crashes unexpectedly, the root cause could exist in any of these distinct layers—from missing configuration files to exhausted TAP device pools or corrupt snapshot metadata. Understanding the specific failure points in the CubeSandbox architecture allows operators to quickly identify whether the issue lies in bootstrap validation, VM creation, or post-startup health monitoring.

Understanding the Sandbox Startup Lifecycle

CubeSandbox initializes a sandbox through several distinct phases, each with specific failure modes and log locations. The process begins with bootstrap validation in deploy/one-click/scripts/common.sh, where the validate_cubelet_cubecow_deps function verifies required binaries and environment variables like CUBE_SANDBOX_DOMAIN.

Once bootstrap passes, the system warms up networking resources. The network-agent/internal/service/tap_lifecycle.go pre-creates TAP devices (default pool of 500) before Firecracker attempts VM creation. Finally, the envd binary must start and respond on port 49983 before the sandbox reaches a healthy state.

Common Failure Points and How to Diagnose Them

Bootstrap and Configuration Validation

Early failures occur when required environment variables are missing or configuration files are malformed. The cubelet service validates dependencies during initialization and aborts if config.toml is invalid or required binaries are missing.

Check for these issues by examining the systemd journal for the cubelet service:

journalctl -u cube-sandbox-cubelet.service -b

Also inspect the runtime log directory at /data/log/cubelet/ for bootstrap error messages. Common indicators include missing CUBE_API_PUBLIC_HOST or CUBE_PROXY_HOST_PORT settings in your .one-click.env file.

Network Agent TAP Pool Initialization

If the network-agent cannot warm up its TAP device pool, sandbox creation will stall indefinitely. The warmupTapPoolBackground routine in network-agent/internal/service/tap_lifecycle.go handles this pre-allocation, and failures appear as explicit errors in the network-agent logs.

Monitor these issues in real-time with:

journalctl -u cube-sandbox-network-agent.service -f

Races between network-agent and cubelet can cause intermittent startup failures if the TAP pool is not ready when VM creation begins.

VM Creation via Firecracker

Firecracker MicroVM creation fails when kernel files are missing, disks are full, or CPU features like XSAVE are unavailable. The cubelet/services/cubebox/update.go file handles these operations and returns VMCreateFailed errors when Firecracker cannot launch.

Locate these failures in /data/log/cubelet/ by searching for "firecracker" and "VM creation" keywords. The Cubelet/storage/local.go component also writes relevant error details during storage attachment failures.

Envd Data-Plane Startup

The envd data-plane must listen on port 49983 and respond to health checks before the sandbox becomes ready. If the binary is missing from the image or fails to start, the readiness probe defined in docs/guide/troubleshooting/templates.md will timeout.

Check both cubelet and envd service statuses simultaneously:

journalctl -u cube-sandbox-cubelet.service -u cube-sandbox-envd.service

Snapshot and Rollback State

After VM crashes, leftover snapshot metadata can prevent new sandbox creation. The CubeMaster/pkg/templatecenter/snapshot_ops.go reconciles snapshots on startup, logging "reconcileSnapshotDefinitionTimeouts" when it encounters corrupt state.

Inspect the CubeMaster logs to identify these issues:

journalctl -u cube-sandbox-cubemaster.service

Post-Startup VM Crashes

VMs may exit after successful launch due to OOM conditions or seccomp violations. The cubelet/services/cubebox/update.go watches VMM processes and writes "VM crashed" entries when it detects unexpected exits. Check for automatic snapshot cleanup triggers in the same logs.

If crashes persist, examine potential core dumps in /run/firecracker/ and verify the Firecracker PID status.

Step-by-Step Debugging Workflow

Follow this systematic approach to isolate startup failures:

1. Check systemd status across all components: systemctl status cube-sandbox-<component>.service
2. Read startup logs from the journal for bootstrap phase errors, then check /data/log/<component>/ for runtime details
3. Identify the failure point by searching logs for keywords: bootstrap, tap, firecracker, envd, or snapshot
4. Verify configuration in .one-click.env ensures all required variables (CUBE_SANDBOX_DOMAIN, CUBE_API_PUBLIC_HOST) are exported before services start
5. Enable debug logging by setting CUBE_LOG_LEVEL=debug before sandbox creation—note this does not affect already-running instances
6. Inspect VM state by checking Firecracker sockets and potential core files when crashes occur

Code Examples for Programmatic Debugging

Detect sandbox creation failures using the Go SDK by checking state after initialization:

ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()

sb, err := client.NewSandbox(ctx, "tpl-my-image")
if err != nil {
    log.Fatalf("sandbox creation failed: %v", err) // see Cubelet logs for root cause
}
info, err := sb.GetInfo(ctx)
if err != nil {
    log.Fatalf("cannot retrieve sandbox info: %v", err)
}
if info.State != "running" {
    log.Fatalf("sandbox %s is not running (state=%s)", sb.SandboxID, info.State)
}

Enable debug logging across services before restarting:

export CUBE_LOG_LEVEL=debug
systemctl restart cube-sandbox-{cubelet,network-agent,cube-api}.service
journalctl -u cube-sandbox-cubelet.service -f

Handle VM crashes programmatically by checking for ErrVMCrash:

err := sb.Wait(ctx) // blocks until sandbox exits
if err != nil && errors.Is(err, cubesandbox.ErrVMCrash) {
    // retrieve crash details from cubelet logs
    logPath := "/data/log/cubelet/" + sb.SandboxID + ".log"
    data, _ := os.ReadFile(logPath)
    fmt.Println(string(data))
}

Summary

• Bootstrap failures appear in journalctl -u cube-sandbox-cubelet.service and /data/log/cubelet/ when environment variables or binaries are missing
• TAP pool issues in network-agent/internal/service/tap_lifecycle.go cause networking stalls visible in journalctl -u cube-sandbox-network-agent.service
• VM creation errors return VMCreateFailed from cubelet/services/cubebox/update.go when Firecracker cannot launch due to missing kernel files or CPU feature gaps
• Envd health check timeouts occur when port 49983 is unreachable, documented in docs/guide/troubleshooting/templates.md
• Snapshot corruption after crashes is reconciled by CubeMaster/pkg/templatecenter/snapshot_ops.go, with logs in the CubeMaster systemd service
• Debug mode requires setting CUBE_LOG_LEVEL=debug before sandbox creation, not during runtime

Frequently Asked Questions

How do I know if a sandbox failure is caused by configuration or VM creation?

Check the cubelet systemd journal first. If you see errors from validate_cubelet_cubecow_deps in deploy/one-click/scripts/common.sh or messages about missing CUBE_SANDBOX_DOMAIN, the failure is bootstrap-related. If logs show "firecracker" or "VMCreateFailed" from cubelet/services/cubebox/update.go, the issue occurs during MicroVM initialization, typically due to missing kernel files or insufficient disk space.

Where can I find logs for the envd data-plane when health checks fail?

Envd logs are available through the systemd journal by querying both services: journalctl -u cube-sandbox-cubelet.service -u cube-sandbox-envd.service. The readiness probe expects a response on /:49983/health as defined in docs/guide/troubleshooting/templates.md. If envd is missing from the container image or fails to bind to this port, the logs will show connection refused or timeout errors.

What causes VM crashes after successful sandbox startup?

Post-launch crashes typically result from resource exhaustion (OOM) or security policy violations (seccomp). The cubelet/services/cubebox/update.go file monitors the VMM process and writes "VM crashed" entries to /data/log/cubelet/ when the Firecracker process exits unexpectedly. Inspect /run/firecracker/ for core dumps and check if automatic snapshot cleanup was triggered to handle residual state.

How do I debug snapshot-related startup failures?

Snapshot corruption or leftover files from previous crashes are detected by CubeMaster/pkg/templatecenter/snapshot_ops.go, which logs "reconcileSnapshotDefinitionTimeouts" during reconciliation. Check journalctl -u cube-sandbox-cubemaster.service for these specific timeout messages, indicating that snapshot metadata needs manual cleanup or repair before new sandboxes can launch successfully.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}