CubeSandbox Multi-Node Deployment Architecture and Requirements

CubeSandbox scales from a single-node installation to a distributed cluster by deploying a control node that hosts the full orchestration stack while lightweight compute nodes register via the internal meta API to handle sandbox execution.

CubeSandbox supports horizontal scaling through a multi-node architecture that separates control plane responsibilities from sandbox runtime execution. This guide explains the architecture, hardware requirements, and deployment steps necessary to expand a TencentCloud/CubeSandbox installation across multiple bare-metal servers.

Architecture Overview

The multi-node deployment model consists of two distinct roles: the control node and compute nodes. This separation allows you to dedicate specific machines to orchestration while distributing sandbox workloads across additional hardware.

Control Node Components

The control node hosts the complete management plane and persistent services. According to the source code in docs/guide/multi-node-deploy.md, this node runs:

• CubeMaster – The core orchestrator that schedules and manages sandbox lifecycles
• cube-api – The API gateway for external requests
• CubeProxy – Traffic routing component with CoreDNS for service discovery
• MySQL and Redis – Persistent storage for metadata and caching
• Cubelet and network-agent – Local sandbox runtime components (the control node also acts as a compute node)

Compute Node Components

Each compute node runs only the lightweight runtime stack required to execute sandboxes. As implemented in the install.sh script referenced by deploy/one-click/install-compute.sh, these nodes run:

• Cubelet – The node agent that manages local sandbox containers
• network-agent – Handles network configuration and connectivity
• CubeShim – The container runtime interface component

Compute nodes register themselves to the control plane through the /internal/meta API on port 8089 (configurable), then await scheduling requests from CubeMaster.

Prerequisites and Requirements

All nodes must satisfy specific hardware, software, and networking criteria before joining the cluster.

Hardware Requirements

• Physical machine or bare-metal server – Nested virtualization is explicitly not supported
• CPU Architecture – x86_64 or aarch64 (ARM64) with KVM enabled (verify with ls /dev/kvm)
• Resources – Sufficient CPU, memory, and storage to run the respective components (control node requires more resources for database and API services)

Software Requirements

• Operating System – Linux distribution capable of running Docker
• Docker – Must be installed and running on every node (control and compute) to manage containerized components
• Release Bundle – All nodes must use the same release bundle (the tarball generated by the one-click installer) to ensure version consistency across the cluster

Network Requirements

• Compute nodes must reach the control node's CubeMaster service on port 8089 by default
• Internal network connectivity must allow bidirectional communication between the control plane and compute nodes for heartbeat and scheduling operations

Step-by-Step Deployment Process

Deploying a multi-node cluster involves preparing the release bundle, configuring environment variables, and executing the installation scripts.

1. Prepare the Release Bundle

Extract the release tarball on the control node, then copy it to each compute node and extract it there:

tar -xzf cube-sandbox-one-click-<version>.tar.gz
cd cube-sandbox-one-click-<version>

2. Configure the Compute Node Environment

Create a .env file from the template provided in env.example. The deploy/one-click/install-compute.sh script automatically sets ONE_CLICK_DEPLOY_ROLE=compute, but you must specify the control plane address:

cp env.example .env

# Edit .env

ONE_CLICK_DEPLOY_ROLE=compute
CUBE_SANDBOX_NODE_IP=<node-ip>
ONE_CLICK_CONTROL_PLANE_IP=<control-node-ip>

# Optional: explicit CubeMaster address

ONE_CLICK_CONTROL_PLANE_CUBEMASTER_ADDR=<control-ip>:8089

3. Install Compute Services

Run the compute-specific installer, which delegates to the generic install.sh script after setting the appropriate role:

sudo ./install-compute.sh

This script extracts binaries, creates systemd services, and starts the Cubelet and network-agent components.

4. Verify Local Installation

Execute the health-check script to confirm local services are running and can reach the control plane:

sudo ./smoke.sh

Expected output confirms network-agent is operational, CubeMaster is reachable, and the node registration status is healthy.

5. Confirm Registration on the Control Node

From the control node, query the internal meta API to verify the compute node appears in the cluster:

curl http://127.0.0.1:8089/internal/meta/nodes

The response should list the new compute node with a healthy status, indicating successful registration.

Scheduler Configuration for Multi-Node

To achieve balanced sandbox placement across multiple compute nodes, modify the scheduler configuration in cubemaster.yaml (located in the conf/ directory of the release bundle):

scheduler:
  priority_select_num: 3            # >1 enables random selection among top-scored nodes

  score:
    enable_scorers:
      - real_time_weighted_average
    resource_weights:
      mvm_num: 2
      local_create_num: 3
      quota_cpu_usage: 1
      quota_mem_usage: 1
    plugin_conf:
      real_time_weighted_average:
        weight: 1.0
        enable_weight_factors:
          - mvm_num
          - local_create_num
          - quota_cpu_usage
          - quota_mem_usage

After editing cubemaster.yaml, restart the CubeMaster service to apply the new scoring algorithm. This configuration enables the scheduler to distribute workloads based on real-time resource utilization across the compute pool.

Summary

• CubeSandbox uses a split architecture where the control node runs the full orchestration stack (CubeMaster, databases, API gateway) and compute nodes run only the sandbox runtime (Cubelet, network-agent, CubeShim).
• Hardware requirements mandate bare-metal servers with KVM-enabled x86_64 or aarch64 CPUs; nested virtualization is not supported.
• Deployment requires the same release bundle on all nodes, Docker installed everywhere, and specific environment variables (ONE_CLICK_DEPLOY_ROLE=compute, ONE_CLICK_CONTROL_PLANE_IP) on compute nodes.
• Verification involves running smoke.sh on compute nodes and checking /internal/meta/nodes on the control node to confirm registration.
• Performance can be optimized by configuring the scheduler in cubemaster.yaml to use weighted scoring across multiple compute nodes.

Frequently Asked Questions

What hardware architecture does CubeSandbox support for multi-node deployment?

CubeSandbox supports x86_64 and aarch64 (ARM64) CPU architectures for both control and compute nodes. All machines must be physical or bare-metal servers with KVM enabled, as nested virtualization is explicitly not supported. Verify KVM availability by checking for /dev/kvm on each node before installation.

Can compute nodes register to the control plane if they are on different network segments?

Yes, provided the compute nodes can reach the control node's CubeMaster service on port 8089 (or the configured custom port). The ONE_CLICK_CONTROL_PLANE_IP and optional ONE_CLICK_CONTROL_PLANE_CUBEMASTER_ADDR environment variables allow you to specify the exact address and port for registration across network segments.

How do I verify that a compute node successfully registered with the control node?

Execute sudo ./smoke.sh on the compute node to verify local services and connectivity, then query the control node's internal meta API using curl http://127.0.0.1:8089/internal/meta/nodes. The JSON response should list the compute node's IP address and indicate a healthy registration status.

Does the control node also run sandbox workloads, or only the compute nodes?

The control node runs a full Cubelet instance in addition to the control plane services, meaning it can execute sandbox workloads alongside its orchestration duties. However, for production deployments, it is recommended to dedicate the control node primarily to management tasks and distribute user workloads across dedicated compute nodes to ensure cluster stability.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}