CubeSandbox Snapshot Health Check Notifications and App Snapshot Annotations: A Technical Guide

CubeSandbox uses HTTP notification endpoints in CubeMaster to track snapshot health status updates and relies on specific Kubernetes-style annotations in Cubelet to control application snapshot creation workflows.

CubeSandbox is an open-source sandbox platform developed by Tencent Cloud that manages containerized workloads with snapshot capabilities. Understanding how snapshot health check notifications propagate through the system and how app snapshot annotations configure creation workflows is essential for operators managing persistent container states. This guide examines the source code implementation in the TencentCloud/CubeSandbox repository to explain these mechanisms.

Snapshot Health Check Notifications in CubeMaster

The CubeMaster component exposes an HTTP notification service that receives health status updates from sandbox nodes. These notifications persist snapshot state in the database and trigger timeout handling when snapshot operations exceed expected durations.

Notification Endpoints and Payloads

The notification handlers are registered in CubeMaster/pkg/service/httpservice/notify/notify.go. The service exposes three distinct endpoints under the /notify path:

• /notify/health: A simple liveness probe that returns HTTP 200 with body "OK"
• /notify/snapnotify: Receives standard health status updates for snapshots
• /notify/snapnotifytimeout: Handles timeout notifications when snapshot creation exceeds time limits

The handler function HttpHandler routes requests based on the URL path and parses JSON payloads using util.DecodeJSONBody.

For standard health updates, the endpoint expects a JSON payload with snapshot_id and status fields:

var payload struct {
    SnapshotID string `json:"snapshot_id"`
    Status     string `json:"status"`
}

Upon successful decoding, the handler invokes base.UpdateSnapshotHealth(payload.SnapshotID, payload.Status) to persist the state.

For timeout scenarios, the payload structure includes a boolean timeout flag:

var payload struct {
    SnapshotID string `json:"snapshot_id"`
    Timeout    bool   `json:"timeout"`
}

This triggers base.UpdateSnapshotTimeout(payload.SnapshotID, payload.Timeout), marking the snapshot operation as timed out in the database.

Database Persistence and Health Status

The health status is stored in the snapshot_runtime_refs table, defined in CubeMaster/pkg/base/db/models/snapshot_runtime_ref.go. The SnapshotRuntimeRef struct includes a HealthStatus field that tracks the current state of the snapshot:

type SnapshotRuntimeRef struct {
    ID            string    `gorm:"primary_key;column:id" json:"id"`
    SnapshotID    string    `gorm:"column:snapshot_id" json:"snapshot_id"`
    NodeID        string    `gorm:"column:node_id" json:"node_id"`
    NodeIP        string    `gorm:"column:node_ip" json:"node_ip"`
    AttachedAt    time.Time `gorm:"column:attached_at" json:"attached_at"`
    HealthStatus  string    `gorm:"column:health_status" json:"health_status"`
}

The model provides an UpdateHealthStatus method that wraps the GORM Save operation, ensuring atomic updates to the health status column when notifications arrive from the Cubelet.

App Snapshot Annotations in Cubelet

Cubelet processes application snapshot creation requests by inspecting pod annotations. These annotations act as declarative configuration flags that determine whether to create a snapshot, which template to use, and which resources to include.

Required Annotations for Snapshot Creation

The CreateAppSnapshot function in Cubelet/services/cubebox/appsnapshot.go validates two mandatory annotations before proceeding with snapshot creation. These constants are defined in Cubelet/pkg/constants/const.go:

• cube.app.snapshot.create: Must be set to "true" (case-insensitive) to trigger snapshot creation
• cube.app.snapshot.template.id: Must contain a non-empty string identifying the snapshot template

The validation logic explicitly checks for these annotations:

createFlag, ok := annotations[constants.MasterAnnotationsAppSnapshotCreate]
if !ok || strings.ToLower(createFlag) != "true" {
    return nil, fmt.Errorf("annotation %s must be set to \"true\"", 
        constants.MasterAnnotationsAppSnapshotCreate)
}

templateID, ok := annotations[constants.MasterAnnotationAppSnapshotTemplateID]
if !ok || strings.TrimSpace(templateID) == "" {
    return nil, fmt.Errorf("annotation %s is required and must not be empty", 
        constants.MasterAnnotationAppSnapshotTemplateID)
}

Container Identification and Resource Annotations

Additional annotations control which container to snapshot and what resources to include:

• cube.app.snapshot.container.id: Specifies the target container ID. If omitted, the function snapshotContainerIDFromAnnotations falls back to the sandbox ID
• cube.vm.kernel.path: Defines the VM kernel image path for VM-based snapshots (AnnotationsVMKernelPath)
• cube.vmmres: Contains JSON-encoded VM resource specifications (AnnotationsVMSpecKey)
• cube.disk: Describes disk mount configurations (AnnotationsMountListKey)
• cube.pmem: Configures persistent memory settings (AnnotationsPmemKey and AnnotationPmem)

The snapshotContainerIDFromAnnotations helper function implements the fallback logic:

func snapshotContainerIDFromAnnotations(annotations map[string]string, sandboxID string) string {
    if len(annotations) == 0 {
        return ""
    }
    if value := strings.TrimSpace(annotations[constants.AnnotationAppSnapshotContainerID]); 
       value != "" {
        return value
    }
    return sandboxID
}

Notification and Annotation Integration Flow

When a sandbox initiates snapshot creation, Cubelet reads the pod annotations defined in const.go to configure the operation. Upon completion or failure, the sandbox sends an HTTP POST request to CubeMaster's /notify/snapnotify endpoint with the snapshot ID and status. If the operation times out, the sandbox sends a notification to /notify/snapnotifytimeout instead. CubeMaster persists these updates via UpdateSnapshotHealth or UpdateSnapshotTimeout, modifying the HealthStatus field in the SnapshotRuntimeRef database record.

This decoupled architecture ensures that snapshot creation logic remains stateless in Cubelet while CubeMaster maintains the authoritative state in snapshot_runtime_refs, enabling reliable health monitoring across distributed sandbox nodes.

Summary

• CubeMaster notification endpoints (/notify/snapnotify and /notify/snapnotifytimeout) receive HTTP POST requests containing snapshot_id and health status updates from sandbox nodes.
• Health status persistence occurs in the SnapshotRuntimeRef model via UpdateSnapshotHealth, storing state in the health_status column of the snapshot_runtime_refs table.
• Required annotations cube.app.snapshot.create and cube.app.snapshot.template.id must be present and valid for CreateAppSnapshot to proceed.
• Container identification uses the cube.app.snapshot.container.id annotation with automatic fallback to the sandbox ID when unspecified.
• Resource annotations (cube.vm.kernel.path, cube.vmmres, cube.disk, cube.pmem) provide VM and storage configuration for comprehensive snapshot capture.

Frequently Asked Questions

What triggers a snapshot health check notification in CubeSandbox?

A snapshot health check notification triggers when a sandbox completes, fails, or times out during snapshot creation. The sandbox process sends an HTTP POST request to CubeMaster's /notify/snapnotify endpoint with the snapshot_id and current status, or to /notify/snapnotifytimeout if the operation exceeded its time limit.

Which annotations are mandatory for creating an app snapshot?

CubeSandbox requires two specific annotations: cube.app.snapshot.create must be set to "true" (case-insensitive), and cube.app.snapshot.template.id must contain a non-empty template identifier string. The CreateAppSnapshot function in Cubelet/services/cubebox/appsnapshot.go validates both fields before initiating the snapshot process.

How does CubeSandbox handle timeout scenarios during snapshot creation?

When snapshot creation exceeds the expected duration, the sandbox sends a notification to the /notify/snapnotifytimeout endpoint with a JSON payload containing "snapshot_id" and "timeout": true. The CubeMaster handler calls base.UpdateSnapshotTimeout, which marks the snapshot as timed out in the SnapshotRuntimeRef database record.

Where is the snapshot health status stored in the CubeSandbox architecture?

The health status is stored in the snapshot_runtime_refs database table, managed through the SnapshotRuntimeRef model defined in CubeMaster/pkg/base/db/models/snapshot_runtime_ref.go. The HealthStatus column receives updates via the UpdateHealthStatus method when health check notifications arrive from Cubelet nodes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}