Troubleshooting Steps for Common CubeSandbox Creation and Startup Failures

Diagnose CubeSandbox creation failures by checking template validation in CubeMaster/pkg/service/sandbox/sandbox_create.go, network CIDR conflicts in hostdir_mount.go, and host-mount permissions in Cubelet/storage/files.go, then verify Cubelet connectivity via the CBR-I health endpoint.

When working with the CubeSandbox repository from TencentCloud, sandbox creation and startup failures typically occur during the orchestration between Cubemaster and Cubelet components. Understanding the specific troubleshooting steps for common sandbox creation and startup failures requires tracing the request lifecycle through template validation, network allocation, and root-fs preparation. This guide provides actionable diagnostics based on the actual source code implementation to resolve these failures quickly.

Common Failure Categories and Root Causes

Sandboxes fail when any step in the creation chain returns an error. The system categorizes these failures into six primary areas based on the code paths in sandbox_run.go and related modules.

Template Validation Errors

The Cubemaster receives the CreateSandbox request and immediately validates the templateID in CubeMaster/pkg/service/sandbox/sandbox_create.go. If the template lookup fails, the request aborts with ErrorCode_TemplateNotFound or cubesandbox.ErrTemplateNotFound. Verify template existence using the CLI before submission.

Network CIDR Conflicts

During host selection, CubeMaster/pkg/service/sandbox/hostdir_mount.go allocates a pod-wide CIDR. Duplicate subnet errors surface when this range overlaps with existing network interfaces, causing the creation to timeout. Check ip netns list on the host to identify conflicts.

Host-Mount Permission Denied

File operations fail with "Permission denied" when Cubelet/storage/files.go detects insufficient privileges. The sandbox user (typically UID sandbox) requires read/write access to the host directory. This check occurs during the CreateSandboxRootfsFromTemplate phase in Cubelet/storage/local.go.

Resource Quota Violations

Immediate stops marked as "OOMKilled" or "CPU throttling" originate from Cubelet/storage/pool.go and Cubelet/storage/pool_withreflink.go. These modules enforce size limits during root-fs creation, rejecting over-allocated requests before the sandbox process launches.

Cubelet Connectivity and RPC Timeouts

CubeMaster/pkg/service/sandbox/sandbox_timeout.go and sandbox_exec.go handle RPC retries to the Cubelet via the CBR-I (CubeBox Runtime Interface). Network partitions or misconfigured CUBE_SANDBOX_DOMAIN variables cause "cubesandbox: timeout" errors after the initial creation appears successful.

Authentication and Proxy Failures

The Go SDK in sdk/go/client.go injects Authorization headers for every request. Missing or expired tokens result in 401/403 responses from the API gateway before the request ever reaches the Cubemaster.

Step-by-Step Debugging Workflow

Follow this systematic approach to isolate the failure point:

1. Inspect the API response – Error messages wrap cubesandbox.Err* types. Check for cubesandbox.ErrSandboxNotFound or cubesandbox.ErrTemplateNotFound in the client output.

2. Analyze Cubemaster logs – Search for log.G(ctx) traces containing CreateSandbox, handleCubelet, or failover. Look for the RetCode field to identify specific error codes like ErrorCode_TemplateNotFound.

3. Validate template availability – Run cubemastercli template list to confirm the template exists in the template store (CubeMaster/pkg/service/templatecenter/store.go).

4. Check network allocations – Execute ip netns list on the host to verify no CIDR overlaps exist with the subnet allocated by network-agent/internal/service/netdevice.go.

5. Verify host-mount permissions – Ensure the sandbox user has write access to the host directory. Adjust permissions using chmod or chown as documented in the host-mount permissions guide.

6. Test Cubelet connectivity – Run curl http://$CUBE_SANDBOX_DOMAIN/healthz from the master node. If this fails, inspect DNS, firewall, or TLS configuration.

Code Examples for Error Handling

Implement defensive checks in your application code to catch failures at the SDK level:

// Example: Create a sandbox and handle common errors
ctx := context.Background()
client, _ := cubesandbox.NewClient(&cubesandbox.Config{
    SandboxDomain: "sandbox.internal", // must match the deployment
})

// Request a sandbox from a known template
sb, err := client.CreateSandbox(ctx, &cubesandbox.CreateSandboxRequest{
    TemplateID: "tpl-basic",
    CpuCount:   2,
    MemoryMB:   1024,
})
if err != nil {
    // Inspect the wrapped error
    if errors.Is(err, cubesandbox.ErrTemplateNotFound) {
        log.Fatalf("Template missing – run `cubemastercli template list`")
    }
    if errors.Is(err, cubesandbox.ErrSandboxNotFound) {
        log.Fatalf("Sandbox creation failed – check host‑mount permissions")
    }
    log.Fatalf("Unexpected error: %v", err)
}
log.Printf("Sandbox %s created, ID=%s", sb.TemplateID, sb.SandboxID)

// After creation, ensure the sandbox is running
info, _ := client.GetInfo(ctx, sb.SandboxID)
if info.State != "running" {
    log.Printf("Sandbox not started, state=%s – see Cubemaster logs for details", info.State)
}

Use these CLI commands for rapid host-side diagnostics:

# 1. Verify template existence

cubemastercli template list | grep tpl-basic

# 2. Inspect host CIDR allocations

ip netns exec $(cubemastercli host list | grep active | awk '{print $1}') ip -o -4 addr show

# 3. Test Cubelet health endpoint

curl -s http://sandbox.internal/healthz && echo "Cubelet reachable"

Key Source Files for Troubleshooting

Understanding these specific files accelerates root cause analysis:

• CubeMaster/pkg/service/sandbox/sandbox_run.go – Orchestrates sandbox creation, retry logic, and metric collection via createSandboxContext.

• CubeMaster/pkg/service/sandbox/sandbox_create.go – Contains the CreateSandbox function that validates templateID and initiates the context.

• CubeMaster/pkg/service/sandbox/hostdir_mount.go – Allocates pod-wide CIDRs; the source of network conflict errors.

• Cubelet/storage/local.go – Implements CreateSandboxRootfsFromTemplate to prepare the filesystem before process launch.

• Cubelet/storage/files.go – Performs file reads/writes inside sandboxes; surfaces permission errors to the user.

• Cubelet/storage/pool.go and pool_withreflink.go – Enforce resource quotas and size limits during storage allocation.

• CubeMaster/pkg/service/sandbox/sandbox_timeout.go – Manages RPC timeouts and retry logic for Cubelet communication.

• sdk/go/client.go – Provides the Go SDK interface and authentication header injection.

• network-agent/internal/service/netdevice.go – Manages pod CIDR allocation and conflict detection.

Summary

• Template validation occurs in sandbox_create.go and returns ErrorCode_TemplateNotFound when the templateID is invalid.
• Network CIDR conflicts arise from overlapping subnets in hostdir_mount.go and require checking ip netns allocations.
• Host-mount permissions are enforced in Cubelet/storage/files.go, requiring the sandbox user to have directory write access.
• Resource quotas are managed by pool.go and pool_withreflink.go, rejecting requests that exceed memory or CPU limits.
• Cubelet connectivity issues manifest as RPC timeouts in sandbox_timeout.go and require verifying CUBE_SANDBOX_DOMAIN and health endpoints.
• Authentication failures occur in sdk/go/client.go when Authorization headers are missing or expired.

Frequently Asked Questions

Why does my sandbox return "template not found" during creation?

This error originates in CubeMaster/pkg/service/sandbox/sandbox_create.go when the CreateSandbox function fails to locate the requested templateID in the template store. Verify the template exists by running cubemastercli template list and ensure the ID matches exactly, including case sensitivity.

How do I resolve network CIDR conflicts when creating sandboxes?

CIDR conflicts occur when CubeMaster/pkg/service/sandbox/hostdir_mount.go attempts to allocate a subnet that overlaps with existing host interfaces. Run ip netns list on the host to identify occupied ranges, then reconfigure the network pool or remove conflicting interfaces before retrying the creation request.

What causes "permission denied" errors in sandbox file operations?

The Cubelet/storage/files.go module returns this error when the sandbox user (typically UID sandbox) lacks read/write permissions on the mounted host directory. Check the directory ownership with ls -la and adjust permissions using chown or chmod to grant the sandbox user access, particularly for directories used during CreateSandboxRootfsFromTemplate.

Why is my sandbox stuck in a timeout or "not found" state after creation?

This indicates RPC communication failure between Cubemaster and Cubelet, handled in CubeMaster/pkg/service/sandbox/sandbox_timeout.go and sandbox_exec.go. Verify the CUBE_SANDBOX_DOMAIN environment variable is correctly set, test connectivity with curl http://$CUBE_SANDBOX_DOMAIN/healthz, and ensure no firewall rules or DNS misconfigurations block the CBR-I interface.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}