# Multi-Node Resource Scheduling Mechanisms in CubeMaster: Filter, Score, and Select

> Discover CubeMaster's multi-node scheduling: filter healthy nodes, score resources via weighted averages, and select the best host with randomized selection for efficient orchestration.

- Repository: [Tencent Cloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox)
- Tags: deep-dive
- Published: 2026-07-16

---

**CubeMaster’s scheduler uses a three-stage pipeline that filters nodes by health and resource quotas, scores them using a real-time weighted average of CPU, memory, and sandbox density metrics, and selects the optimal host through configurable top-node randomization.**

CubeMaster serves as the control plane for TencentCloud’s CubeSandbox project, orchestrating sandbox workloads across distributed compute infrastructure. Its multi-node resource scheduling mechanisms employ a pluggable architecture that combines predicate-based filtering with priority scoring to ensure balanced workload distribution across heterogeneous clusters.

## The Three-Stage Scheduling Pipeline

CubeMaster’s scheduling logic follows a **filter → score → select** pattern implemented in `CubeMaster/pkg/scheduler` (imported in [`CubeMaster/pkg/service/sandbox/sandbox_run.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/service/sandbox/sandbox_run.go)).

### Stage 1: Node Filtering (Prefilter)

The scheduler first inspects all alive nodes and removes those that fail health or resource constraints. According to the source code in [`docs/changelog/v0.4.0.md`](https://github.com/TencentCloud/CubeSandbox/blob/main/docs/changelog/v0.4.0.md), the prefilter evaluates:

- **Health status**: Nodes must report fresh heartbeats to remain eligible ([`docs/changelog/v0.4.0.md`](https://github.com/TencentCloud/CubeSandbox/blob/main/docs/changelog/v0.4.0.md#L125-L128).
- **Over-commit ratios**: Configurable resource quotas prevent scheduling on overloaded nodes ([`docs/changelog/v0.4.0.md`](https://github.com/TencentCloud/CubeSandbox/blob/main/docs/changelog/v0.4.0.md#L87-L90).
- **Redis allocation bypass**: The optional `ignore_redis_allocation` flag treats Redis-recorded allocations as zero, effectively ignoring historical resource accounting during the filtering phase.

This produces a filtered pool of candidate nodes suitable for hosting the new sandbox.

### Stage 2: Node Scoring with real_time_weighted_average

Each remaining node receives a numerical score via the built-in `real_time_weighted_average` plugin. This default scorer calculates a composite value blending four key metrics:

- `mvm_num`: Number of running MVMs on the node
- `local_create_num`: Count of locally created sandboxes
- `cpu_usage`: Current CPU utilization
- `quota_mem_usage`: Memory quota consumption

The implementation resides in the scheduler package under `CubeMaster/pkg/scheduler`, where these real-time metrics determine the relative desirability of each node.

### Stage 3: Node Selection

The final selection phase uses two configurable strategies defined in [`docs/guide/multi-node-deploy.md`](https://github.com/TencentCloud/CubeSandbox/blob/main/docs/guide/multi-node-deploy.md#L148-L150):

- **Top-node sampling**: The `scheduler.priority_select_num` parameter controls how many highest-scored nodes are considered. The default value of `1` selects only the top-scored node, while values greater than `1` enable random selection among the best nodes to improve distribution in larger clusters.
- **Final selection strategy**: Within the chosen set, the `least_select_name` strategy (defaulting to `random`) makes the final host determination.

## Resource Reclamation for Paused Sandboxes

CubeMaster handles paused sandboxes through configurable resource release mechanisms. As documented in [`docs/changelog/v0.5.0.md`](https://github.com/TencentCloud/CubeSandbox/blob/main/docs/changelog/v0.5.0.md#L17-L22), the `host.quota.paused_resource_release_ratio` setting (ranging from 0 to 1) determines how much CPU and memory quota returns to the scheduler when a sandbox pauses:

- A ratio of `1.0` releases the full quota immediately, increasing node capacity for new sandboxes.
- A ratio of `0` keeps the quota reserved, guaranteeing resources remain available for successful resume operations.

## High Availability and Concurrency Control

In HA mode where `cubemaster_replicas` exceeds 1, the scheduler divides its concurrency budget across control-plane instances to prevent oversubscription. As noted in [`docs/changelog/v0.5.0.md`](https://github.com/TencentCloud/CubeSandbox/blob/main/docs/changelog/v0.5.0.md#L50-L55), this replica-aware budgeting ensures that scheduler decisions remain consistent across the clustered control plane without exceeding resource limits.

## Implementation Example

The scheduling workflow appears in [`CubeMaster/pkg/service/sandbox/sandbox_run.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/service/sandbox/sandbox_run.go#L157) and [`CubeMaster/pkg/service/sandbox/sandbox_migrate.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/service/sandbox/sandbox_migrate.go#L40). The following pattern demonstrates the API usage:

```go
// Build selection context (attributes such as required CPU, memory, instance type)
selctx := scheduler.NewSelCtx(req)

// Add the request to the scheduler buffer (asynchronous handling)
scheduler.AddBufferTask(selctx, req.InstanceType)

// Synchronously choose a host based on current filter+score state
host, err := scheduler.Select(selctx)
if err != nil {
    return nil, errorcode.Wrap(errorcode.ErrorCode_SelectNodesNoRes, err)
}

```

The `Select` function triggers the full filter-score-select pipeline, returning the chosen node or an error if no resources are available.

## Summary

- CubeMaster uses a **three-stage pipeline** (filter, score, select) to determine sandbox placement in multi-node clusters.
- The **prefilter** eliminates unhealthy nodes and enforces over-commit ratios and Redis allocation flags.
- **Scoring** relies on the `real_time_weighted_average` plugin combining CPU, memory, MVM count, and local creation metrics.
- **Node selection** uses `priority_select_num` for top-node sampling and `least_select_name` for final host determination.
- **Paused sandboxes** can release resources via `paused_resource_release_ratio` to optimize cluster utilization.
- **HA deployments** split scheduler concurrency budgets across replicas to maintain consistency.

## Frequently Asked Questions

### How does CubeMaster filter nodes during the scheduling process?

CubeMaster’s prefilter evaluates node health through heartbeat freshness, applies configurable over-commit ratios and resource quotas, and optionally ignores Redis-recorded allocations when the `ignore_redis_allocation` flag is set. This filtration occurs before scoring to ensure only viable candidates enter the selection pool, as implemented in the scheduler package referenced by [`CubeMaster/pkg/service/sandbox/sandbox_run.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/service/sandbox/sandbox_run.go).

### What metrics does the real_time_weighted_average scoring plugin use?

The default scoring plugin calculates node desirability using four metrics: the number of running MVMs (`mvm_num`), locally created sandboxes (`local_create_num`), current CPU utilization (`cpu_usage`), and memory quota usage (`quota_mem_usage`). These values blend into a composite score that guides placement decisions according to the implementation in `CubeMaster/pkg/scheduler`.

### How can I configure CubeMaster to distribute sandboxes across multiple top-scored nodes?

Set `scheduler.priority_select_num` to a value greater than `1` in your configuration. While the default of `1` always selects the highest-scored node, increasing this parameter enables random selection among the top-scored nodes, which improves load distribution across larger clusters according to [`docs/guide/multi-node-deploy.md`](https://github.com/TencentCloud/CubeSandbox/blob/main/docs/guide/multi-node-deploy.md).

### What happens to resource quotas when a sandbox is paused?

The `host.quota.paused_resource_release_ratio` setting controls whether paused sandboxes release their CPU and memory quotas back to the scheduler. A value of `1.0` releases the full quota for new workloads, while `0` reserves all resources to ensure the paused sandbox can resume successfully, as documented in [`docs/changelog/v0.5.0.md`](https://github.com/TencentCloud/CubeSandbox/blob/main/docs/changelog/v0.5.0.md).