How to Perform Multi-Node Cluster Deployment with the CubeMaster Orchestrator in CubeSandbox

Deploy a multi-node CubeSandbox cluster by configuring compute nodes with ONE_CLICK_DEPLOY_ROLE=compute, pointing them to the CubeMaster control plane, and executing install-compute.sh to register via the /internal/meta API.

CubeSandbox separates the control plane (CubeMaster, CubeAPI, Redis, MySQL) from the data plane (Cubelet, CubeShim, CubeHypervisor) to enable horizontal scaling. This guide covers expanding a single control node into a production-ready multi-node cluster by adding compute-only nodes that automatically register with the CubeMaster orchestrator. All procedures reference the deployment automation in the TencentCloud/CubeSandbox repository.

Architecture Overview

CubeSandbox clusters consist of one control node and multiple compute nodes. The control node hosts CubeMaster, which exposes an internal registration API at /internal/meta on port 8089. Compute nodes run a minimal stack—Cubelet, network-agent, and CubeShim—and register themselves as available capacity to the orchestrator.

Control Node (CubeMaster:8089)
         │
         │ /internal/meta API
         ▼
    ┌────────┬────────┐
    ▼        ▼        ▼
Compute1  Compute2  ComputeN

The control node runs the full stack while also acting as a compute resource. Additional compute nodes run only the sandbox runtime components and report health status to CubeMaster.

Prerequisites

Before adding compute nodes, ensure your infrastructure meets these requirements:

• Hardware: Physical or bare-metal servers; nested virtualization is not supported
• CPU Architecture: x86_64 or aarch64 with KVM enabled (ls /dev/kvm must return the device)
• Operating System: Linux with Docker installed and running
• Network: Compute nodes must reach the control node's CubeMaster port 8089
• Control Node: Already deployed via the Self-Build Deployment guide with CubeMaster running

Step-by-Step Multi-Node Deployment

Prepare the Release Bundle

Build the one-click deployment bundle on your control node, then distribute it to each compute node:

# On control node

make one-click-bundle VERSION=0.4.0
scp cube-sandbox-one-click-0.4.0.tar.gz user@compute-node:/tmp/

# On compute node

tar -xzf cube-sandbox-one-click-0.4.0.tar.gz
cd cube-sandbox-one-click-0.4.0

Configure Environment Variables

Copy the example environment file and set three critical variables to define the node's role and control plane endpoint:

cp env.example .env

Edit .env with these values:

ONE_CLICK_DEPLOY_ROLE=compute
CUBE_SANDBOX_NODE_IP=<current-node-ip>
ONE_CLICK_CONTROL_PLANE_IP=<control-node-ip>

• ONE_CLICK_DEPLOY_ROLE: Must be set to compute to install only data-plane components
• CUBE_SANDBOX_NODE_IP: Primary NIC IP address of the current host (not 127.0.0.1)
• ONE_CLICK_CONTROL_PLANE_IP: IP address of the control node; the installer automatically appends port :8089

If CubeMaster uses a non-standard port, override with ONE_CLICK_CONTROL_PLANE_CUBEMASTER_ADDR=<ip>:<port> instead of the _IP variable.

Execute the Compute Node Installer

Run the compute-specific installer script located at deploy/one-click/install-compute.sh:

sudo ./install-compute.sh

This script performs the following actions:

1. Installs Cubelet, network-agent, CubeShim, CubeImage, and Cube-Kernel-SCF
2. Configures Cubelet's meta_server_endpoint to point to the control plane's CubeMaster address
3. Starts only the host processes network-agent and cubelet
4. Registers the node via the /internal/meta API and begins reporting health metrics

The script automatically stops any existing deployment if re-run, making it safe for updates.

Verify Node Registration

Validate the deployment from both perspectives:

On the compute node:

sudo ./smoke.sh

On the control node:

curl http://<control-node-ip>:8089/internal/meta/nodes

The API response should list the new compute node's IP address with a healthy status, confirming successful registration with the CubeMaster orchestrator.

Optimize CubeMaster Scheduling (Optional)

For clusters larger than a single node, enable weighted scoring in cubemaster.yaml to distribute workloads across the top-rated nodes rather than always selecting the first available:

scheduler:
  priority_select_num: 3
  score:
    enable_scorers:
      - real_time_weighted_average
    resource_weights:
      mvm_num: 2
      local_create_num: 3
      quota_cpu_usage: 1
      quota_mem_usage: 1
    plugin_conf:
      real_time_weighted_average:
        weight: 1.0
        enable_weight_factors:
          - mvm_num
          - local_create_num
          - quota_cpu_usage
          - quota_mem_usage

Restart CubeMaster after applying these changes to activate the multi-node scheduling logic.

Operational Commands and Log Locations

Manage compute nodes using these utilities:

• Stop services: sudo ./down.sh (stops only Cubelet and network-agent)
• Re-install: Re-run sudo ./install-compute.sh (auto-stops existing processes)

Log file locations for troubleshooting:

Component	Path
Cubelet	/data/log/Cubelet/
CubeShim	/data/log/CubeShim/
CubeVmm	/data/log/CubeVmm/
PID files	/var/run/cube-sandbox-one-click/
Stdout/Stderr	/var/log/cube-sandbox-one-click/

Troubleshooting Multi-Node Registration

Node fails to appear in CubeMaster:

Verify the compute node can reach the control plane on port 8089. Check firewall rules and confirm .env contains the correct ONE_CLICK_CONTROL_PLANE_IP or ONE_CLICK_CONTROL_PLANE_CUBEMASTER_ADDR.

Registration errors in logs:

Examine /data/log/Cubelet/ for connection failures. Ensure CUBE_SANDBOX_NODE_IP is set to a routable address, not localhost, and that the CubeMaster service is active on the control node.

Summary

• CubeSandbox clusters separate control plane services from data plane compute nodes, with CubeMaster orchestrating via the /internal/meta API on port 8089
• Compute nodes require ONE_CLICK_DEPLOY_ROLE=compute and proper ONE_CLICK_CONTROL_PLANE_IP configuration to register correctly
• The install-compute.sh script automates installation, configuration, and registration of compute nodes with the CubeMaster orchestrator
• Verify deployments using ./smoke.sh locally and curl /internal/meta/nodes from the control node
• Scale scheduling efficiently by enabling weighted scoring with priority_select_num in cubemaster.yaml

Frequently Asked Questions

What distinguishes a control node from a compute node in CubeSandbox?

A control node runs the complete stack including CubeMaster, CubeAPI, Redis, MySQL, and CoreDNS, while also functioning as a compute resource. A compute node runs only the sandbox runtime components—Cubelet, network-agent, and CubeShim—and registers its capacity with the control node's CubeMaster orchestrator.

Which network port must be accessible between compute nodes and the control plane?

Compute nodes must reach the control node's CubeMaster service on port 8089 to access the /internal/meta registration and health-checking API. Ensure firewalls allow outbound connections from compute nodes to this port on the control node.

How can I confirm that a compute node successfully registered with CubeMaster?

On the compute node, run sudo ./smoke.sh to verify local service health. On the control node, execute curl http://127.0.0.1:8089/internal/meta/nodes and verify the JSON response contains the compute node's IP address with a healthy status indicator.

Does CubeSandbox support nested virtualization for compute nodes?

No. CubeSandbox requires physical or bare-metal servers with KVM enabled. Nested virtualization environments will fail to provide the hardware acceleration required by the CubeHypervisor component.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}