# How CubeSandbox Handles Cross-Node Distribution: Template Replication Deep Dive

> Learn how CubeSandbox achieves cross node distribution with its template replication system. Discover the three stage pipeline for reliable artifact distribution and metadata persistence.

- Repository: [Tencent Cloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox)
- Tags: deep-dive
- Published: 2026-07-15

---

**The CubeSandbox template system implements a robust three-stage pipeline—resolving target nodes, concurrently distributing rootfs artifacts via cubelet RPCs, and persisting placement metadata—to ensure reliable template replication and garbage collection across the cluster.**

The TencentCloud CubeSandbox project uses a sophisticated cross-node distribution mechanism to ensure that sandbox templates are available on every physical node that hosts instances. This article examines the Go-based implementation in `CubeMaster/pkg/templatecenter` that drives the distribution of root filesystem artifacts from the CubeMaster to multiple cubelets, handling concurrency, fault tolerance, and eventual cleanup.

## The Three-Stage Distribution Architecture

The cross-node distribution process follows a strict pipeline defined in [`CubeMaster/pkg/templatecenter/distribution.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/templatecenter/distribution.go) and [`request_validation.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/request_validation.go).

### Stage 1: Resolving Target Nodes with resolveTemplateNodes

Distribution begins with node resolution. The `resolveTemplateNodes` function in [`CubeMaster/pkg/templatecenter/request_validation.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/templatecenter/request_validation.go) accepts an **instance type** (e.g., `cubebox` or `cvm`) and a **distribution scope** (IP strings or node names) from the `CreateTemplateFromImageReq` request.

It validates each target against the live node inventory and returns a slice of `*node.Node` objects representing the physical machines that must receive the artifact. This prevents distribution attempts to offline or mismatched nodes.

### Stage 2: Parallel Artifact Creation on Each Node

Once targets are resolved, `distributeRootfsArtifact` orchestrates the actual transfer. The implementation uses a worker pool (`defaultDistributionWorkers`) to launch concurrent goroutines for each node, limiting resource exhaustion while maximizing throughput.

Each goroutine executes:
1. **Replica initialization** via `buildReplicaForDistribution`, creating a `ReplicaStatus` record with phase `Distributing` and initial status `Failed`.
2. **RPC transmission** by calling `cubelet.CreateImage` with the artifact metadata, download token, and ext4 checksum.
3. **Success handling** on RPC completion: updates the replica to `Distributed`, clears cleanup flags, and calls `upsertArtifactNodePlacement` to record the physical node-artifact mapping.
4. **Failure handling** on RPC error: records the error message, keeps the replica in `Failed` status, and bubbles the first error back to the caller.

The function returns aggregate counters (`expected`, `ready`, `failed`) alongside the list of successfully prepared nodes, enabling upstream callers like [`image_job_runner.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/image_job_runner.go) to trigger retry logic.

### Stage 3: Persisting Placement and Cleanup Metadata

After successful distribution, `upsertArtifactNodePlacement` persists the physical placement data. This metadata is critical for the **"last-owner-cleanup"** mechanism: if the replica database row disappears, the system can still locate the artifact on the node for garbage collection.

## Safe Cleanup and Idempotent Deletion

When a template is deleted, `cleanupTemplateReplicasOnNodes` and `cleanupArtifactOnNodes` invoke `destroyArtifactOnNode` for each placement record. This function sends a `DestroyImage` RPC to the cubelet with `storage_media=ext4` and the appropriate instance-type annotation.

The deletion logic treats specific RPC responses as success states:
- **`NotFound`** – The artifact is already gone; the operation is idempotent.
- **`Conflict`** – The artifact is still referenced by a running sandbox; deletion is deferred to a later GC cycle.

## Code Example: Distributing a Rootfs Artifact

Below is a minimal reproduction of the distribution flow found in the CubeSandbox source:

```go
// 1. Prepare the request (normally received from the API layer)
req := &types.CreateTemplateFromImageReq{
    InstanceType:      "cubebox",
    DistributionScope: []string{"10.1.2.3", "node-b"},
    WritableLayerSize: "2Gi",
}

// 2. Resolve the nodes that should receive the artifact
targets, err := resolveTemplateNodes(req.InstanceType, req.DistributionScope)
if err != nil {
    log.Fatalf("cannot resolve nodes: %v", err)
}

// 3. Build the artifact metadata (normally retrieved from DB)
artifact := &models.RootfsArtifact{
    ArtifactID:            "rootfs-abc123",
    MasterNodeIP:          "10.1.2.3",
    DownloadToken:         "token-xyz",
    Ext4SHA256:            "deadbeef...",
    Ext4SizeBytes:         1073741824,
    TemplateSpecFingerprint: "fp-01",
}

// 4. Distribute the artifact to every target node
readyNodes, expected, ready, failed, err := distributeRootfsArtifact(
    context.Background(),
    req,
    nil,               // generatedReq – nil for this low-level demo
    artifact,
    "template-01",
    uuid.NewString(),
)

fmt.Printf("distribution: %d/%d nodes ready, %d failures (first error: %v)\n",
    ready, expected, failed, err)

```

Running this snippet triggers the same cross-node distribution flow: node resolution, parallel `CreateImage` RPCs, replica status updates, and placement recording.

## Key Source Files for Cross-Node Distribution

Understanding the complete system requires examining these files in the `CubeMaster/pkg/templatecenter` package:

- **[`distribution.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/distribution.go)** – Contains `distributeRootfsArtifact`, `buildReplicaForDistribution`, `upsertArtifactNodePlacement`, and `destroyArtifactOnNode`.
- **[`request_validation.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/request_validation.go)** – Implements `resolveTemplateNodes` for scope-to-node translation.
- **[`image_job_runner.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/image_job_runner.go)** – Orchestrates the distribution as part of the image-creation job pipeline.
- **[`redo.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/redo.go)** – Re-executes distribution when a previous attempt failed after partial cleanup.
- **[`artifact_gc.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/artifact_gc.go)** and **[`artifact_cleanup.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/artifact_cleanup.go)** – Provide garbage-collection pathways that rely on placement metadata.
- **[`cache.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/cache.go)** – Caches node-artifact placements to avoid repeated database lookups during GC.

## Summary

- **Three-stage pipeline**: Node resolution, parallel artifact creation, and metadata persistence form the core cross-node distribution mechanism.
- **Concurrency control**: `distributeRootfsArtifact` uses `defaultDistributionWorkers` to limit goroutine explosion while distributing to many nodes.
- **Fault isolation**: Each node’s success or failure is tracked independently via `ReplicaStatus`, with the first error returned to the caller.
- **Idempotent cleanup**: `destroyArtifactOnNode` treats `NotFound` as success and `Conflict` as a deferral, ensuring safe, repeatable deletion.
- **Placement tracking**: `upsertArtifactNodePlacement` records physical locations, enabling garbage collection even if database replicas vanish.

## Frequently Asked Questions

### How does CubeSandbox determine which nodes receive a template?

The system calls `resolveTemplateNodes` in [`CubeMaster/pkg/templatecenter/request_validation.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/templatecenter/request_validation.go), which filters the cluster’s node inventory by the requested **instance type** (e.g., `cubebox`) and matches against the **distribution scope** provided in the request (IP addresses or node names). It returns only validated, live nodes capable of hosting the sandbox.

### What happens if a node fails during cross-node distribution?

If a cubelet RPC fails, `distributeRootfsArtifact` records the error in the replica’s status, keeps it in the `Failed` phase, and returns the error to the caller. The process continues for other nodes; partial success is recorded via the `ready` and `failed` counters. Upstream logic in [`redo.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/redo.go) can retry the distribution for failed nodes later.

### How does CubeSandbox handle template deletion across the cluster?

Deletion invokes `cleanupTemplateReplicasOnNodes`, which gathers placement metadata and calls `destroyArtifactOnNode` for each location. This sends a `DestroyImage` RPC to the cubelet. If the artifact is still referenced by a running sandbox (Conflict), deletion is deferred; if already gone (NotFound), the operation succeeds idempotently.

### Why is placement metadata stored separately from the replica record?

The `upsertArtifactNodePlacement` function records the physical node-artifact mapping so that garbage-collection routines in [`artifact_gc.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/artifact_gc.go) can locate and clean up ext4 artifacts even if the replica database row is deleted or corrupted. This separation ensures that temporary database inconsistencies do not leak storage on the cubelets.