Multi-Node Resource Scheduling Mechanisms in CubeMaster: Filter, Score, and Select

CubeMaster’s scheduler uses a three-stage pipeline that filters nodes by health and resource quotas, scores them using a real-time weighted average of CPU, memory, and sandbox density metrics, and selects the optimal host through configurable top-node randomization.

CubeMaster serves as the control plane for TencentCloud’s CubeSandbox project, orchestrating sandbox workloads across distributed compute infrastructure. Its multi-node resource scheduling mechanisms employ a pluggable architecture that combines predicate-based filtering with priority scoring to ensure balanced workload distribution across heterogeneous clusters.

The Three-Stage Scheduling Pipeline

CubeMaster’s scheduling logic follows a filter → score → select pattern implemented in CubeMaster/pkg/scheduler (imported in CubeMaster/pkg/service/sandbox/sandbox_run.go).

Stage 1: Node Filtering (Prefilter)

The scheduler first inspects all alive nodes and removes those that fail health or resource constraints. According to the source code in docs/changelog/v0.4.0.md, the prefilter evaluates:

• Health status: Nodes must report fresh heartbeats to remain eligible (docs/changelog/v0.4.0.md.
• Over-commit ratios: Configurable resource quotas prevent scheduling on overloaded nodes (docs/changelog/v0.4.0.md.
• Redis allocation bypass: The optional ignore_redis_allocation flag treats Redis-recorded allocations as zero, effectively ignoring historical resource accounting during the filtering phase.

This produces a filtered pool of candidate nodes suitable for hosting the new sandbox.

Stage 2: Node Scoring with real_time_weighted_average

Each remaining node receives a numerical score via the built-in real_time_weighted_average plugin. This default scorer calculates a composite value blending four key metrics:

• mvm_num: Number of running MVMs on the node
• local_create_num: Count of locally created sandboxes
• cpu_usage: Current CPU utilization
• quota_mem_usage: Memory quota consumption

The implementation resides in the scheduler package under CubeMaster/pkg/scheduler, where these real-time metrics determine the relative desirability of each node.

Stage 3: Node Selection

The final selection phase uses two configurable strategies defined in docs/guide/multi-node-deploy.md:

• Top-node sampling: The scheduler.priority_select_num parameter controls how many highest-scored nodes are considered. The default value of 1 selects only the top-scored node, while values greater than 1 enable random selection among the best nodes to improve distribution in larger clusters.
• Final selection strategy: Within the chosen set, the least_select_name strategy (defaulting to random) makes the final host determination.

Resource Reclamation for Paused Sandboxes

CubeMaster handles paused sandboxes through configurable resource release mechanisms. As documented in docs/changelog/v0.5.0.md, the host.quota.paused_resource_release_ratio setting (ranging from 0 to 1) determines how much CPU and memory quota returns to the scheduler when a sandbox pauses:

• A ratio of 1.0 releases the full quota immediately, increasing node capacity for new sandboxes.
• A ratio of 0 keeps the quota reserved, guaranteeing resources remain available for successful resume operations.

High Availability and Concurrency Control

In HA mode where cubemaster_replicas exceeds 1, the scheduler divides its concurrency budget across control-plane instances to prevent oversubscription. As noted in docs/changelog/v0.5.0.md, this replica-aware budgeting ensures that scheduler decisions remain consistent across the clustered control plane without exceeding resource limits.

Implementation Example

The scheduling workflow appears in CubeMaster/pkg/service/sandbox/sandbox_run.go and CubeMaster/pkg/service/sandbox/sandbox_migrate.go. The following pattern demonstrates the API usage:

// Build selection context (attributes such as required CPU, memory, instance type)
selctx := scheduler.NewSelCtx(req)

// Add the request to the scheduler buffer (asynchronous handling)
scheduler.AddBufferTask(selctx, req.InstanceType)

// Synchronously choose a host based on current filter+score state
host, err := scheduler.Select(selctx)
if err != nil {
    return nil, errorcode.Wrap(errorcode.ErrorCode_SelectNodesNoRes, err)
}

The Select function triggers the full filter-score-select pipeline, returning the chosen node or an error if no resources are available.

Summary

• CubeMaster uses a three-stage pipeline (filter, score, select) to determine sandbox placement in multi-node clusters.
• The prefilter eliminates unhealthy nodes and enforces over-commit ratios and Redis allocation flags.
• Scoring relies on the real_time_weighted_average plugin combining CPU, memory, MVM count, and local creation metrics.
• Node selection uses priority_select_num for top-node sampling and least_select_name for final host determination.
• Paused sandboxes can release resources via paused_resource_release_ratio to optimize cluster utilization.
• HA deployments split scheduler concurrency budgets across replicas to maintain consistency.

Frequently Asked Questions

How does CubeMaster filter nodes during the scheduling process?

CubeMaster’s prefilter evaluates node health through heartbeat freshness, applies configurable over-commit ratios and resource quotas, and optionally ignores Redis-recorded allocations when the ignore_redis_allocation flag is set. This filtration occurs before scoring to ensure only viable candidates enter the selection pool, as implemented in the scheduler package referenced by CubeMaster/pkg/service/sandbox/sandbox_run.go.

What metrics does the real_time_weighted_average scoring plugin use?

The default scoring plugin calculates node desirability using four metrics: the number of running MVMs (mvm_num), locally created sandboxes (local_create_num), current CPU utilization (cpu_usage), and memory quota usage (quota_mem_usage). These values blend into a composite score that guides placement decisions according to the implementation in CubeMaster/pkg/scheduler.

How can I configure CubeMaster to distribute sandboxes across multiple top-scored nodes?

Set scheduler.priority_select_num to a value greater than 1 in your configuration. While the default of 1 always selects the highest-scored node, increasing this parameter enables random selection among the top-scored nodes, which improves load distribution across larger clusters according to docs/guide/multi-node-deploy.md.

What happens to resource quotas when a sandbox is paused?

The host.quota.paused_resource_release_ratio setting controls whether paused sandboxes release their CPU and memory quotas back to the scheduler. A value of 1.0 releases the full quota for new workloads, while 0 reserves all resources to ensure the paused sandbox can resume successfully, as documented in docs/changelog/v0.5.0.md.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}