How CubeSandbox Handles Multi-Node Cluster Scaling and State Distribution

CubeSandbox achieves horizontal scaling by separating the CubeMaster control plane from distributed Cubelet agents, using an embedded etcd-like KV store as the single source of truth for consistent state distribution across the cluster.

TencentCloud's CubeSandbox implements a distributed micro-VM orchestration system where multi-node cluster scaling and state distribution rely on a centralized control plane coordinating with node-local agents. The architecture ensures consistency through a replicated key-value store while enabling automatic workload rebalancing when nodes join or leave the cluster.

Control Plane and Node Architecture

CubeSandbox employs a control plane / worker split with four core components:

• CubeMaster — The central scheduler that receives sandbox creation requests and assigns micro-VMs (MVMs) to nodes with sufficient resources. It maintains the canonical cluster state and reacts to topology changes.
• Cubelet — A node-local agent running on every compute node that launches and stops MVMs on demand. Each Cubelet registers itself with the master via gRPC.
• CubeAPI — A REST façade exposing the public HTTP API (/cluster/*) that aggregates data from the master and forwards actions to appropriate Cubelets.
• Embedded KV Store — An etcd-like key-value store that persists the cluster state (node list, resource capacities, sandbox placements) and replicates across high-availability master instances.

Configuration for the master resides in configs/single-node/cubemaster.yaml, while the core data structures are defined in CubeAPI/src/state.rs.

Automatic Cluster Scaling

New nodes join the cluster through a gRPC registration handshake initiated by the Cubelet. Upon startup, the Cubelet sends a RegisterNode request containing resource metadata:

{
  "node_id": "node-42",
  "total_cpu_milli": 8000,
  "total_memory_mb": 16384,
  "allocatable_cpu_milli": 8000,
  "allocatable_memory_mb": 16384,
  "version": "v0.5.2"
}

The master stores this data under the key cluster/nodes/{node_id} in its KV store. Continuous heartbeats from each Cubelet report real-time resource availability, allowing the master to maintain an accurate view of cluster capacity.

When the master detects that a node's allocatable resources drop below scheduling thresholds (e.g., after launching a heavy sandbox), it defers new placements to other nodes. Conversely, when a new node registers capacity, pending sandbox creation requests are immediately reassigned to utilize the fresh resources. This on-the-fly redistribution enables horizontal scaling without manual intervention.

If a node fails and stops sending heartbeats, the master marks it unhealthy, removes its capacity from aggregate totals, and optionally re-schedules affected sandboxes onto healthy nodes based on restart policies.

Consistent State Distribution

CubeSandbox implements a partition-free consistency model where every master instance reads from the same KV store snapshot. The canonical state is defined in CubeAPI/src/state.rs by the ClusterState struct:

pub struct ClusterState {
    pub nodes: HashMap<String, NodeInfo>,
    pub total_cpu_milli: u64,
    pub total_memory_mb: u64,
    // … other aggregate metrics
}

The NodeInfo structure tracks per-node health, capacity, and running sandbox IDs. The master recomputes cluster aggregates (total_cpu_milli, total_memory_mb) whenever node state changes, ensuring all scheduling decisions operate against a consistent view.

Clients obtain read-only snapshots via two primary REST endpoints:

• GET /cluster/overview — Returns aggregate CPU, memory, node count, and health status.
• GET /cluster/versions — Returns a version matrix describing the control-plane version on each node, enabling rolling upgrade coordination.

The web UI polls GET /cluster/overview every 10 seconds to display real-time metrics, as implemented in web/src/pages/Overview.tsx:

const cluster = useQuery({
  queryKey: ['cluster'],
  queryFn: clusterApi.overview,
  refetchInterval: 10_000,
});

Implementation Examples

Querying Cluster Health

Access cluster utilization metrics programmatically using the TypeScript API client in web/src/api/client.ts:

import { clusterApi } from '@/api/client';

async function printClusterStatus() {
  const overview = await clusterApi.overview();
  console.log(`Healthy nodes: ${overview.healthyNodes}/${overview.nodeCount}`);
  console.log(`CPU usage: ${overview.totalCpuMilli - overview.allocatableCpuMilli} mCPU`);
  console.log(`Memory usage: ${overview.totalMemoryMB - overview.allocatableMemoryMB} MB`);
}
printClusterStatus();

Node Registration Flow

The Cubelet implementation in Cubelet/storage/local.go handles the gRPC client initialization that transmits the registration payload. Once the master persists this data to cluster/nodes/{node_id}, the new capacity is immediately visible to the scheduling logic.

Monitoring Version Compatibility

During rolling upgrades, retrieve the version matrix to identify nodes requiring updates:

const versions = await clusterApi.versions();
versions.forEach(v => {
  console.log(`${v.node_id}: ${v.version}`);
});

Key Source Files

Path	Purpose
configs/single-node/cubemaster.yaml	Default master configuration for networking and storage
CubeAPI/src/state.rs	Core data structures (ClusterState, NodeInfo) and REST handlers
Cubelet/storage/local.go	Node-side registration and heartbeat logic
web/src/api/generated/schema.ts	TypeScript definitions for Cluster API payloads (ClusterOverviewDto, VersionMatrixDto)
web/src/pages/Overview.tsx	UI component displaying aggregated cluster state
web/src/api/client.ts	HTTP client wrapper for Cluster API endpoints

Summary

• CubeMaster maintains cluster state in an embedded KV store, serving as the single source of truth for resource allocation and node health.
• Cubelets register via gRPC and transmit continuous heartbeats containing totalCpuMilli, totalMemoryMB, and allocatable resources.
• Automatic rebalancing occurs when nodes join or fail, with the master deferring scheduling to healthy nodes and redistributing workloads based on real-time capacity.
• Consistent state distribution relies on a partition-free architecture where all master instances read from the same replicated KV store snapshot.
• REST endpoints (/cluster/overview, /cluster/versions) provide real-time visibility into aggregate metrics and version compatibility.

Frequently Asked Questions

How does CubeSandbox detect when a new node is available?

When a new node boots, its Cubelet initiates a gRPC RegisterNode handshake with the CubeMaster, transmitting resource capacities and version metadata. The master stores this information in the KV store under cluster/nodes/{node_id}, immediately incorporating the new capacity into scheduling decisions and aggregate totals.

What happens to workloads when a node fails?

If a Cubelet stops sending heartbeats, the master marks the node unhealthy and removes its allocatable resources from the cluster overview. Depending on the sandbox's restart policy, the master may automatically re-schedule affected micro-VMs onto other healthy nodes with sufficient capacity, maintaining service availability without manual intervention.

How does CubeSandbox ensure state consistency during scaling events?

The system uses a partition-free architecture where every CubeMaster instance reads from the same embedded KV store snapshot. The ClusterState struct in CubeAPI/src/state.rs maintains authoritative records of node health and resource allocation, ensuring that scaling decisions and API responses reflect a consistent view of the cluster topology.

Can external tools query cluster state without using the web UI?

Yes. The CubeAPI exposes REST endpoints that return JSON payloads defined in web/src/api/generated/schema.ts. External scripts can call GET /cluster/overview for utilization metrics or GET /cluster/versions for software version matrices, using standard HTTP clients or the provided TypeScript wrapper in web/src/api/client.ts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}