How CubeSandbox Handles Cross-Node Distribution: Template Replication Deep Dive

The CubeSandbox template system implements a robust three-stage pipeline—resolving target nodes, concurrently distributing rootfs artifacts via cubelet RPCs, and persisting placement metadata—to ensure reliable template replication and garbage collection across the cluster.

The TencentCloud CubeSandbox project uses a sophisticated cross-node distribution mechanism to ensure that sandbox templates are available on every physical node that hosts instances. This article examines the Go-based implementation in CubeMaster/pkg/templatecenter that drives the distribution of root filesystem artifacts from the CubeMaster to multiple cubelets, handling concurrency, fault tolerance, and eventual cleanup.

The Three-Stage Distribution Architecture

The cross-node distribution process follows a strict pipeline defined in CubeMaster/pkg/templatecenter/distribution.go and request_validation.go.

Stage 1: Resolving Target Nodes with resolveTemplateNodes

Distribution begins with node resolution. The resolveTemplateNodes function in CubeMaster/pkg/templatecenter/request_validation.go accepts an instance type (e.g., cubebox or cvm) and a distribution scope (IP strings or node names) from the CreateTemplateFromImageReq request.

It validates each target against the live node inventory and returns a slice of *node.Node objects representing the physical machines that must receive the artifact. This prevents distribution attempts to offline or mismatched nodes.

Stage 2: Parallel Artifact Creation on Each Node

Once targets are resolved, distributeRootfsArtifact orchestrates the actual transfer. The implementation uses a worker pool (defaultDistributionWorkers) to launch concurrent goroutines for each node, limiting resource exhaustion while maximizing throughput.

Each goroutine executes:

1. Replica initialization via buildReplicaForDistribution, creating a ReplicaStatus record with phase Distributing and initial status Failed.
2. RPC transmission by calling cubelet.CreateImage with the artifact metadata, download token, and ext4 checksum.
3. Success handling on RPC completion: updates the replica to Distributed, clears cleanup flags, and calls upsertArtifactNodePlacement to record the physical node-artifact mapping.
4. Failure handling on RPC error: records the error message, keeps the replica in Failed status, and bubbles the first error back to the caller.

The function returns aggregate counters (expected, ready, failed) alongside the list of successfully prepared nodes, enabling upstream callers like image_job_runner.go to trigger retry logic.

Stage 3: Persisting Placement and Cleanup Metadata

After successful distribution, upsertArtifactNodePlacement persists the physical placement data. This metadata is critical for the "last-owner-cleanup" mechanism: if the replica database row disappears, the system can still locate the artifact on the node for garbage collection.

Safe Cleanup and Idempotent Deletion

When a template is deleted, cleanupTemplateReplicasOnNodes and cleanupArtifactOnNodes invoke destroyArtifactOnNode for each placement record. This function sends a DestroyImage RPC to the cubelet with storage_media=ext4 and the appropriate instance-type annotation.

The deletion logic treats specific RPC responses as success states:

• NotFound – The artifact is already gone; the operation is idempotent.
• Conflict – The artifact is still referenced by a running sandbox; deletion is deferred to a later GC cycle.

Code Example: Distributing a Rootfs Artifact

Below is a minimal reproduction of the distribution flow found in the CubeSandbox source:

// 1. Prepare the request (normally received from the API layer)
req := &types.CreateTemplateFromImageReq{
    InstanceType:      "cubebox",
    DistributionScope: []string{"10.1.2.3", "node-b"},
    WritableLayerSize: "2Gi",
}

// 2. Resolve the nodes that should receive the artifact
targets, err := resolveTemplateNodes(req.InstanceType, req.DistributionScope)
if err != nil {
    log.Fatalf("cannot resolve nodes: %v", err)
}

// 3. Build the artifact metadata (normally retrieved from DB)
artifact := &models.RootfsArtifact{
    ArtifactID:            "rootfs-abc123",
    MasterNodeIP:          "10.1.2.3",
    DownloadToken:         "token-xyz",
    Ext4SHA256:            "deadbeef...",
    Ext4SizeBytes:         1073741824,
    TemplateSpecFingerprint: "fp-01",
}

// 4. Distribute the artifact to every target node
readyNodes, expected, ready, failed, err := distributeRootfsArtifact(
    context.Background(),
    req,
    nil,               // generatedReq – nil for this low-level demo
    artifact,
    "template-01",
    uuid.NewString(),
)

fmt.Printf("distribution: %d/%d nodes ready, %d failures (first error: %v)\n",
    ready, expected, failed, err)

Running this snippet triggers the same cross-node distribution flow: node resolution, parallel CreateImage RPCs, replica status updates, and placement recording.

Key Source Files for Cross-Node Distribution

Understanding the complete system requires examining these files in the CubeMaster/pkg/templatecenter package:

• distribution.go – Contains distributeRootfsArtifact, buildReplicaForDistribution, upsertArtifactNodePlacement, and destroyArtifactOnNode.
• request_validation.go – Implements resolveTemplateNodes for scope-to-node translation.
• image_job_runner.go – Orchestrates the distribution as part of the image-creation job pipeline.
• redo.go – Re-executes distribution when a previous attempt failed after partial cleanup.
• artifact_gc.go and artifact_cleanup.go – Provide garbage-collection pathways that rely on placement metadata.
• cache.go – Caches node-artifact placements to avoid repeated database lookups during GC.

Summary

• Three-stage pipeline: Node resolution, parallel artifact creation, and metadata persistence form the core cross-node distribution mechanism.
• Concurrency control: distributeRootfsArtifact uses defaultDistributionWorkers to limit goroutine explosion while distributing to many nodes.
• Fault isolation: Each node’s success or failure is tracked independently via ReplicaStatus, with the first error returned to the caller.
• Idempotent cleanup: destroyArtifactOnNode treats NotFound as success and Conflict as a deferral, ensuring safe, repeatable deletion.
• Placement tracking: upsertArtifactNodePlacement records physical locations, enabling garbage collection even if database replicas vanish.

Frequently Asked Questions

How does CubeSandbox determine which nodes receive a template?

The system calls resolveTemplateNodes in CubeMaster/pkg/templatecenter/request_validation.go, which filters the cluster’s node inventory by the requested instance type (e.g., cubebox) and matches against the distribution scope provided in the request (IP addresses or node names). It returns only validated, live nodes capable of hosting the sandbox.

What happens if a node fails during cross-node distribution?

If a cubelet RPC fails, distributeRootfsArtifact records the error in the replica’s status, keeps it in the Failed phase, and returns the error to the caller. The process continues for other nodes; partial success is recorded via the ready and failed counters. Upstream logic in redo.go can retry the distribution for failed nodes later.

How does CubeSandbox handle template deletion across the cluster?

Deletion invokes cleanupTemplateReplicasOnNodes, which gathers placement metadata and calls destroyArtifactOnNode for each location. This sends a DestroyImage RPC to the cubelet. If the artifact is still referenced by a running sandbox (Conflict), deletion is deferred; if already gone (NotFound), the operation succeeds idempotently.

Why is placement metadata stored separately from the replica record?

The upsertArtifactNodePlacement function records the physical node-artifact mapping so that garbage-collection routines in artifact_gc.go can locate and clean up ext4 artifacts even if the replica database row is deleted or corrupted. This separation ensures that temporary database inconsistencies do not leak storage on the cubelets.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}