# CubeSandbox Multi-Node Deployment Architecture and Requirements > Explore the CubeSandbox multi-node deployment architecture. Learn about control and compute node roles and essential requirements for a distributed cluster setup. - Repository: [Tencent Cloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox) - Tags: architecture - Published: 2026-07-12 --- **CubeSandbox scales from a single-node installation to a distributed cluster by deploying a control node that hosts the full orchestration stack while lightweight compute nodes register via the internal meta API to handle sandbox execution.** CubeSandbox supports horizontal scaling through a multi-node architecture that separates control plane responsibilities from sandbox runtime execution. This guide explains the architecture, hardware requirements, and deployment steps necessary to expand a TencentCloud/CubeSandbox installation across multiple bare-metal servers. ## Architecture Overview The multi-node deployment model consists of two distinct roles: the **control node** and **compute nodes**. This separation allows you to dedicate specific machines to orchestration while distributing sandbox workloads across additional hardware. ### Control Node Components The control node hosts the complete management plane and persistent services. According to the source code in [`docs/guide/multi-node-deploy.md`](https://github.com/TencentCloud/CubeSandbox/blob/main/docs/guide/multi-node-deploy.md), this node runs: - **CubeMaster** – The core orchestrator that schedules and manages sandbox lifecycles - **cube-api** – The API gateway for external requests - **CubeProxy** – Traffic routing component with CoreDNS for service discovery - **MySQL and Redis** – Persistent storage for metadata and caching - **Cubelet and network-agent** – Local sandbox runtime components (the control node also acts as a compute node) ### Compute Node Components Each compute node runs only the lightweight runtime stack required to execute sandboxes. As implemented in the [`install.sh`](https://github.com/TencentCloud/CubeSandbox/blob/main/install.sh) script referenced by [`deploy/one-click/install-compute.sh`](https://github.com/TencentCloud/CubeSandbox/blob/main/deploy/one-click/install-compute.sh), these nodes run: - **Cubelet** – The node agent that manages local sandbox containers - **network-agent** – Handles network configuration and connectivity - **CubeShim** – The container runtime interface component Compute nodes register themselves to the control plane through the `/internal/meta` API on port **8089** (configurable), then await scheduling requests from CubeMaster. ## Prerequisites and Requirements All nodes must satisfy specific hardware, software, and networking criteria before joining the cluster. ### Hardware Requirements - **Physical machine or bare-metal server** – Nested virtualization is explicitly **not supported** - **CPU Architecture** – `x86_64` or `aarch64` (ARM64) with KVM enabled (verify with `ls /dev/kvm`) - **Resources** – Sufficient CPU, memory, and storage to run the respective components (control node requires more resources for database and API services) ### Software Requirements - **Operating System** – Linux distribution capable of running Docker - **Docker** – Must be installed and running on every node (control and compute) to manage containerized components - **Release Bundle** – All nodes must use the **same release bundle** (the tarball generated by the one-click installer) to ensure version consistency across the cluster ### Network Requirements - Compute nodes must reach the control node's **CubeMaster** service on port **8089** by default - Internal network connectivity must allow bidirectional communication between the control plane and compute nodes for heartbeat and scheduling operations ## Step-by-Step Deployment Process Deploying a multi-node cluster involves preparing the release bundle, configuring environment variables, and executing the installation scripts. ### 1. Prepare the Release Bundle Extract the release tarball on the control node, then copy it to each compute node and extract it there: ```bash tar -xzf cube-sandbox-one-click-.tar.gz cd cube-sandbox-one-click- ``` ### 2. Configure the Compute Node Environment Create a `.env` file from the template provided in `env.example`. The [`deploy/one-click/install-compute.sh`](https://github.com/TencentCloud/CubeSandbox/blob/main/deploy/one-click/install-compute.sh) script automatically sets `ONE_CLICK_DEPLOY_ROLE=compute`, but you must specify the control plane address: ```bash cp env.example .env # Edit .env ONE_CLICK_DEPLOY_ROLE=compute CUBE_SANDBOX_NODE_IP= ONE_CLICK_CONTROL_PLANE_IP= # Optional: explicit CubeMaster address ONE_CLICK_CONTROL_PLANE_CUBEMASTER_ADDR=:8089 ``` ### 3. Install Compute Services Run the compute-specific installer, which delegates to the generic [`install.sh`](https://github.com/TencentCloud/CubeSandbox/blob/main/install.sh) script after setting the appropriate role: ```bash sudo ./install-compute.sh ``` This script extracts binaries, creates systemd services, and starts the `Cubelet` and `network-agent` components. ### 4. Verify Local Installation Execute the health-check script to confirm local services are running and can reach the control plane: ```bash sudo ./smoke.sh ``` Expected output confirms `network-agent` is operational, CubeMaster is reachable, and the node registration status is healthy. ### 5. Confirm Registration on the Control Node From the control node, query the internal meta API to verify the compute node appears in the cluster: ```bash curl http://127.0.0.1:8089/internal/meta/nodes ``` The response should list the new compute node with a healthy status, indicating successful registration. ## Scheduler Configuration for Multi-Node To achieve balanced sandbox placement across multiple compute nodes, modify the scheduler configuration in [`cubemaster.yaml`](https://github.com/TencentCloud/CubeSandbox/blob/main/cubemaster.yaml) (located in the `conf/` directory of the release bundle): ```yaml scheduler: priority_select_num: 3 # >1 enables random selection among top-scored nodes score: enable_scorers: - real_time_weighted_average resource_weights: mvm_num: 2 local_create_num: 3 quota_cpu_usage: 1 quota_mem_usage: 1 plugin_conf: real_time_weighted_average: weight: 1.0 enable_weight_factors: - mvm_num - local_create_num - quota_cpu_usage - quota_mem_usage ``` After editing [`cubemaster.yaml`](https://github.com/TencentCloud/CubeSandbox/blob/main/cubemaster.yaml), restart the CubeMaster service to apply the new scoring algorithm. This configuration enables the scheduler to distribute workloads based on real-time resource utilization across the compute pool. ## Summary - **CubeSandbox** uses a split architecture where the control node runs the full orchestration stack (CubeMaster, databases, API gateway) and compute nodes run only the sandbox runtime (Cubelet, network-agent, CubeShim). - **Hardware requirements** mandate bare-metal servers with KVM-enabled x86_64 or aarch64 CPUs; nested virtualization is not supported. - **Deployment** requires the same release bundle on all nodes, Docker installed everywhere, and specific environment variables (`ONE_CLICK_DEPLOY_ROLE=compute`, `ONE_CLICK_CONTROL_PLANE_IP`) on compute nodes. - **Verification** involves running [`smoke.sh`](https://github.com/TencentCloud/CubeSandbox/blob/main/smoke.sh) on compute nodes and checking `/internal/meta/nodes` on the control node to confirm registration. - **Performance** can be optimized by configuring the scheduler in [`cubemaster.yaml`](https://github.com/TencentCloud/CubeSandbox/blob/main/cubemaster.yaml) to use weighted scoring across multiple compute nodes. ## Frequently Asked Questions ### What hardware architecture does CubeSandbox support for multi-node deployment? CubeSandbox supports `x86_64` and `aarch64` (ARM64) CPU architectures for both control and compute nodes. All machines must be physical or bare-metal servers with KVM enabled, as nested virtualization is explicitly not supported. Verify KVM availability by checking for `/dev/kvm` on each node before installation. ### Can compute nodes register to the control plane if they are on different network segments? Yes, provided the compute nodes can reach the control node's CubeMaster service on port **8089** (or the configured custom port). The `ONE_CLICK_CONTROL_PLANE_IP` and optional `ONE_CLICK_CONTROL_PLANE_CUBEMASTER_ADDR` environment variables allow you to specify the exact address and port for registration across network segments. ### How do I verify that a compute node successfully registered with the control node? Execute `sudo ./smoke.sh` on the compute node to verify local services and connectivity, then query the control node's internal meta API using `curl http://127.0.0.1:8089/internal/meta/nodes`. The JSON response should list the compute node's IP address and indicate a healthy registration status. ### Does the control node also run sandbox workloads, or only the compute nodes? The control node runs a full Cubelet instance in addition to the control plane services, meaning it can execute sandbox workloads alongside its orchestration duties. However, for production deployments, it is recommended to dedicate the control node primarily to management tasks and distribute user workloads across dedicated compute nodes to ensure cluster stability.