How CubeMaster Performs Resource-Aware Scheduling Across Nodes in CubeSandbox

CubeMaster selects optimal nodes for new sandboxes by executing a multi-stage pipeline that filters candidates based on CPU, memory, disk, and NIC availability, then scores remaining nodes using weighted plugins to identify the least-loaded host.

CubeMaster serves as the control plane for TencentCloud's CubeSandbox project, managing the placement of sandbox workloads across distributed compute nodes. The scheduler implements a resource-aware architecture that evaluates real-time node capacity through pluggable filters and scoring functions. This article examines the source code implementation to reveal how the system makes intelligent placement decisions.

The Scheduling Pipeline Architecture

The core selection logic resides in CubeMaster/pkg/scheduler/schedule.go, where the Select method orchestrates a five-stage pipeline. Each stage progressively narrows the candidate node pool until a final host is chosen.

Pre-Filter and Backoff Selection

The scheduler first executes runPreFilter to eliminate nodes that cannot possibly satisfy the request. This stage discards dead nodes or those already over-committed according to the global scheduler configuration. If the pre-filter returns no candidates, the system invokes BackoffSelect, which triggers runBackoffFilter to attempt a narrower subset of nodes before failing with ErrNoRes.

Parallel Filter Execution

After pre-filtering, parallelRunFilters executes all registered filter plugins concurrently. Each plugin implements the filter.Selector interface, specifically the Select(*selctx.SelectorCtx) (node.NodeList, error) method. The scheduler computes the intersection of all returned node lists, keeping only nodes that satisfy every resource constraint simultaneously.

Scoring and Final Selection

Once filtered, viable nodes pass through runScoreFilter, where each scoring plugin assigns a weighted score based on utilization metrics. The scheduler normalizes scores by totalPluginWeight and sorts the results using result.AllSortByScore(). Finally, LeastRandomSelect chooses among the top candidates according to PrioritySelectNum, favoring the least-loaded nodes while maintaining randomization to prevent thundering herds.

Resource-Aware Filter Plugins

Filter plugins enforce hard constraints by examining specific resource dimensions. The most critical implementations reside in the CubeMaster/pkg/selector/filter/ directory.

CPU and Memory Constraints

The CPU filter in CubeMaster/pkg/selector/filter/cpufilter.go calculates available capacity as QuotaCpu - AllocatedCpu. It returns only nodes where quotaCpuFree exceeds the requested millicores and CpuUtil remains below NodeMaxCpuUtil. Similarly, CubeMaster/pkg/selector/filter/memfilter.go evaluates QuotaMem - AllocatedMem against the requested memory count.

Disk and NIC Utilization

The disk filter (CubeMaster/pkg/selector/filter/diskfilter.go) monitors DiskUsageMaxPercent, rejecting nodes that exceed the configured threshold. For network resources, the NIC queue filter checks MaxNICQueue to prevent selection of nodes that have exhausted their hardware queue resources.

Template Locality Awareness

CubeMaster/pkg/selector/filter/template_locality.go implements the shouldSkipBackoffForTemplate logic, ensuring that template-specific node affinity preferences are respected. This filter can override the standard back-off behavior when a template requires specific locality constraints.

Scoring Plugin Architecture

After filtering eliminates unsuitable nodes, scoring plugins rank the remaining candidates according to policy-driven optimization goals.

Weighted Scoring Calculation

Each scoring plugin returns a node.NodeScore slice containing raw scores. The scheduler multiplies these values by the plugin's Weight(), configurable via SchedulerScoreConf, then normalizes by dividing by totalPluginWeight. The aggregated results are stored in the selector context via selCtx.SetNodeScoreList before final sorting.

Available Scoring Strategies

The CubeMaster/pkg/selector/score/ directory contains implementations for:

• CPU-utilisation score – preferring nodes with lower active CPU usage
• Memory-pressure score – favoring nodes with greater free memory capacity
• Affinity score – respecting user-specified node affinity selectors

Scheduling Workflow Integration

Scheduling requests originate in CubeMaster/pkg/service/sandbox/sandbox_run.go when processing sandbox creation requests.

1. The system constructs a createSandboxContext containing requested resources.
2. scheduler.AddBufferTask enqueues the scheduling task.
3. The scheduler later invokes scheduler.Select(selCtx), where selCtx provides resource amounts via selCtx.GetResCpuFromCtx() and selCtx.GetResMemFromCtx().
4. The pipeline returns a node.Node containing the host IP and metadata, or an error if no suitable node exists.

Configuration and Extensibility

Resource limits and scheduling behavior are controlled through CubeMaster/pkg/base/config/config.go. The global scheduler configuration exposes several critical parameters:

• Quota calculations – EffectiveQuotaCpu and EffectiveAllocated account for system overhead and per-instance-type reservations.
• Back-off mode – InBackoffMode toggles automatic retry with reduced node sets.
• Circuit filter – DisableCircuitFilter disables the circuit-breaker that removes nodes with repeated failures.

All configuration values support hot-reloading via the hotswap package, allowing runtime adjustments to NodeMaxCpuUtil, DiskUsageMaxPercent, and plugin weights without restarting the scheduler.

Practical Example

The following code demonstrates how to build a selector context and request node placement:

// Build a selector context containing the request resources.
sel := selctx.NewSelectorCtx().
    WithCpu(requestCPU).          // requestCPU is a *resource.Quantity
    WithMem(requestMem).          // requestMem is a *resource.Quantity
    WithNodeAffinity(selAffinity) // optional affinity constraints

// Ask CubeMaster to pick a node.
node, err := scheduler.Select(sel)
if err != nil {
    // Handle ErrNoRes, ErrSelectFailed, etc.
    log.Fatalf("no suitable node: %v", err)
}

// `node` now contains the host IP, instance type, and other metadata.
fmt.Printf("Selected node %s (type %s)\n", node.HostIP(), node.InstanceType)

Summary

• CubeMaster implements a multi-stage pipeline comprising pre-filter, parallel resource filters, and weighted scoring phases.
• Resource-aware filters in cpufilter.go, memfilter.go, and diskfilter.go enforce hard constraints on CPU, memory, disk, and NIC availability.
• The pluggable scoring system aggregates weighted scores from CPU, memory, and affinity plugins to rank viable nodes.
• Hot-reloadable configuration allows runtime adjustment of resource thresholds and scheduling policies via the hotswap package.

Frequently Asked Questions

What happens if no nodes satisfy the resource requirements?

If the pre-filter and main filter stages return no candidates, the scheduler either enters back-off mode via BackoffSelect to retry with a narrower node subset, or returns ErrNoRes to indicate insufficient resources.

Which interface must filter plugins implement?

All filter plugins must implement the filter.Selector interface, specifically the Select(*selctx.SelectorCtx) (node.NodeList, error) method signature, enabling the scheduler to invoke them uniformly during parallelRunFilters.

How does CubeMaster calculate effective CPU and memory quotas?

The scheduler computes effective quotas using EffectiveQuotaCpu and EffectiveAllocated functions in the configuration module, which subtract system overhead and reservations from raw node capacity before comparing against sandbox requests.

Can scheduling policies be updated without restarting CubeMaster?

Yes, the scheduler supports hot-reloading of configuration parameters including NodeMaxCpuUtil, DiskUsageMaxPercent, and plugin weights through the hotswap package, allowing policy adjustments without service interruption.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}