# How CubeSandbox Handles Multi-Node Cluster Scaling and State Distribution

> Discover how CubeSandbox handles multi-node cluster scaling and state distribution. Learn about its separation of control plane and agents with a distributed KV store for consistency.

- Repository: [Tencent Cloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox)
- Tags: architecture
- Published: 2026-07-10

---

**CubeSandbox achieves horizontal scaling by separating the CubeMaster control plane from distributed Cubelet agents, using an embedded etcd-like KV store as the single source of truth for consistent state distribution across the cluster.**

TencentCloud's CubeSandbox implements a distributed micro-VM orchestration system where multi-node cluster scaling and state distribution rely on a centralized control plane coordinating with node-local agents. The architecture ensures consistency through a replicated key-value store while enabling automatic workload rebalancing when nodes join or leave the cluster.

## Control Plane and Node Architecture

CubeSandbox employs a **control plane / worker** split with four core components:

- **CubeMaster** — The central scheduler that receives sandbox creation requests and assigns micro-VMs (MVMs) to nodes with sufficient resources. It maintains the canonical cluster state and reacts to topology changes.
- **Cubelet** — A node-local agent running on every compute node that launches and stops MVMs on demand. Each Cubelet registers itself with the master via gRPC.
- **CubeAPI** — A REST façade exposing the public HTTP API (`/cluster/*`) that aggregates data from the master and forwards actions to appropriate Cubelets.
- **Embedded KV Store** — An etcd-like key-value store that persists the cluster state (node list, resource capacities, sandbox placements) and replicates across high-availability master instances.

Configuration for the master resides in [`configs/single-node/cubemaster.yaml`](https://github.com/TencentCloud/CubeSandbox/blob/main/configs/single-node/cubemaster.yaml), while the core data structures are defined in [`CubeAPI/src/state.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeAPI/src/state.rs).

## Automatic Cluster Scaling

New nodes join the cluster through a **gRPC registration handshake** initiated by the Cubelet. Upon startup, the Cubelet sends a `RegisterNode` request containing resource metadata:

```json
{
  "node_id": "node-42",
  "total_cpu_milli": 8000,
  "total_memory_mb": 16384,
  "allocatable_cpu_milli": 8000,
  "allocatable_memory_mb": 16384,
  "version": "v0.5.2"
}

```

The master stores this data under the key `cluster/nodes/{node_id}` in its KV store. Continuous **heartbeats** from each Cubelet report real-time resource availability, allowing the master to maintain an accurate view of cluster capacity.

When the master detects that a node's allocatable resources drop below scheduling thresholds (e.g., after launching a heavy sandbox), it **defers** new placements to other nodes. Conversely, when a new node registers capacity, pending sandbox creation requests are immediately reassigned to utilize the fresh resources. This on-the-fly redistribution enables horizontal scaling without manual intervention.

If a node fails and stops sending heartbeats, the master marks it unhealthy, removes its capacity from aggregate totals, and optionally re-schedules affected sandboxes onto healthy nodes based on restart policies.

## Consistent State Distribution

CubeSandbox implements a **partition-free** consistency model where every master instance reads from the same KV store snapshot. The canonical state is defined in [`CubeAPI/src/state.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeAPI/src/state.rs) by the `ClusterState` struct:

```rust
pub struct ClusterState {
    pub nodes: HashMap<String, NodeInfo>,
    pub total_cpu_milli: u64,
    pub total_memory_mb: u64,
    // … other aggregate metrics
}

```

The `NodeInfo` structure tracks per-node health, capacity, and running sandbox IDs. The master recomputes cluster aggregates (`total_cpu_milli`, `total_memory_mb`) whenever node state changes, ensuring all scheduling decisions operate against a consistent view.

Clients obtain read-only snapshots via two primary REST endpoints:

- **`GET /cluster/overview`** — Returns aggregate CPU, memory, node count, and health status.
- **`GET /cluster/versions`** — Returns a version matrix describing the control-plane version on each node, enabling rolling upgrade coordination.

The web UI polls `GET /cluster/overview` every 10 seconds to display real-time metrics, as implemented in [`web/src/pages/Overview.tsx`](https://github.com/TencentCloud/CubeSandbox/blob/main/web/src/pages/Overview.tsx):

```tsx
const cluster = useQuery({
  queryKey: ['cluster'],
  queryFn: clusterApi.overview,
  refetchInterval: 10_000,
});

```

## Implementation Examples

### Querying Cluster Health

Access cluster utilization metrics programmatically using the TypeScript API client in [`web/src/api/client.ts`](https://github.com/TencentCloud/CubeSandbox/blob/main/web/src/api/client.ts):

```ts
import { clusterApi } from '@/api/client';

async function printClusterStatus() {
  const overview = await clusterApi.overview();
  console.log(`Healthy nodes: ${overview.healthyNodes}/${overview.nodeCount}`);
  console.log(`CPU usage: ${overview.totalCpuMilli - overview.allocatableCpuMilli} mCPU`);
  console.log(`Memory usage: ${overview.totalMemoryMB - overview.allocatableMemoryMB} MB`);
}
printClusterStatus();

```

### Node Registration Flow

The Cubelet implementation in [`Cubelet/storage/local.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/storage/local.go) handles the gRPC client initialization that transmits the registration payload. Once the master persists this data to `cluster/nodes/{node_id}`, the new capacity is immediately visible to the scheduling logic.

### Monitoring Version Compatibility

During rolling upgrades, retrieve the version matrix to identify nodes requiring updates:

```ts
const versions = await clusterApi.versions();
versions.forEach(v => {
  console.log(`${v.node_id}: ${v.version}`);
});

```

## Key Source Files

| Path | Purpose |
|------|---------|
| [`configs/single-node/cubemaster.yaml`](https://github.com/TencentCloud/CubeSandbox/blob/main/configs/single-node/cubemaster.yaml) | Default master configuration for networking and storage |
| [`CubeAPI/src/state.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeAPI/src/state.rs) | Core data structures (`ClusterState`, `NodeInfo`) and REST handlers |
| [`Cubelet/storage/local.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/storage/local.go) | Node-side registration and heartbeat logic |
| [`web/src/api/generated/schema.ts`](https://github.com/TencentCloud/CubeSandbox/blob/main/web/src/api/generated/schema.ts) | TypeScript definitions for Cluster API payloads (`ClusterOverviewDto`, `VersionMatrixDto`) |
| [`web/src/pages/Overview.tsx`](https://github.com/TencentCloud/CubeSandbox/blob/main/web/src/pages/Overview.tsx) | UI component displaying aggregated cluster state |
| [`web/src/api/client.ts`](https://github.com/TencentCloud/CubeSandbox/blob/main/web/src/api/client.ts) | HTTP client wrapper for Cluster API endpoints |

## Summary

- **CubeMaster** maintains cluster state in an embedded KV store, serving as the single source of truth for resource allocation and node health.
- **Cubelets** register via gRPC and transmit continuous heartbeats containing `totalCpuMilli`, `totalMemoryMB`, and allocatable resources.
- **Automatic rebalancing** occurs when nodes join or fail, with the master deferring scheduling to healthy nodes and redistributing workloads based on real-time capacity.
- **Consistent state distribution** relies on a partition-free architecture where all master instances read from the same replicated KV store snapshot.
- **REST endpoints** (`/cluster/overview`, `/cluster/versions`) provide real-time visibility into aggregate metrics and version compatibility.

## Frequently Asked Questions

### How does CubeSandbox detect when a new node is available?

When a new node boots, its Cubelet initiates a gRPC `RegisterNode` handshake with the CubeMaster, transmitting resource capacities and version metadata. The master stores this information in the KV store under `cluster/nodes/{node_id}`, immediately incorporating the new capacity into scheduling decisions and aggregate totals.

### What happens to workloads when a node fails?

If a Cubelet stops sending heartbeats, the master marks the node unhealthy and removes its allocatable resources from the cluster overview. Depending on the sandbox's restart policy, the master may automatically re-schedule affected micro-VMs onto other healthy nodes with sufficient capacity, maintaining service availability without manual intervention.

### How does CubeSandbox ensure state consistency during scaling events?

The system uses a **partition-free** architecture where every CubeMaster instance reads from the same embedded KV store snapshot. The `ClusterState` struct in [`CubeAPI/src/state.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeAPI/src/state.rs) maintains authoritative records of node health and resource allocation, ensuring that scaling decisions and API responses reflect a consistent view of the cluster topology.

### Can external tools query cluster state without using the web UI?

Yes. The CubeAPI exposes REST endpoints that return JSON payloads defined in [`web/src/api/generated/schema.ts`](https://github.com/TencentCloud/CubeSandbox/blob/main/web/src/api/generated/schema.ts). External scripts can call `GET /cluster/overview` for utilization metrics or `GET /cluster/versions` for software version matrices, using standard HTTP clients or the provided TypeScript wrapper in [`web/src/api/client.ts`](https://github.com/TencentCloud/CubeSandbox/blob/main/web/src/api/client.ts).