# How CubeMaster Performs Resource-Aware Scheduling Across Nodes in CubeSandbox

> CubeMaster optimizes sandbox placement with resource-aware scheduling. Learn how it filters and scores nodes by CPU, memory, disk, and NIC for efficient resource utilization in CubeSandbox.

- Repository: [Tencent Cloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox)
- Tags: internals
- Published: 2026-07-12

---

**CubeMaster selects optimal nodes for new sandboxes by executing a multi-stage pipeline that filters candidates based on CPU, memory, disk, and NIC availability, then scores remaining nodes using weighted plugins to identify the least-loaded host.**

CubeMaster serves as the control plane for TencentCloud's CubeSandbox project, managing the placement of sandbox workloads across distributed compute nodes. The scheduler implements a **resource-aware** architecture that evaluates real-time node capacity through pluggable filters and scoring functions. This article examines the source code implementation to reveal how the system makes intelligent placement decisions.

## The Scheduling Pipeline Architecture

The core selection logic resides in [`CubeMaster/pkg/scheduler/schedule.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/scheduler/schedule.go), where the `Select` method orchestrates a five-stage pipeline. Each stage progressively narrows the candidate node pool until a final host is chosen.

### Pre-Filter and Backoff Selection

The scheduler first executes `runPreFilter` to eliminate nodes that cannot possibly satisfy the request. This stage discards dead nodes or those already over-committed according to the global scheduler configuration. If the pre-filter returns no candidates, the system invokes `BackoffSelect`, which triggers `runBackoffFilter` to attempt a narrower subset of nodes before failing with `ErrNoRes`.

### Parallel Filter Execution

After pre-filtering, `parallelRunFilters` executes all registered filter plugins concurrently. Each plugin implements the `filter.Selector` interface, specifically the `Select(*selctx.SelectorCtx) (node.NodeList, error)` method. The scheduler computes the intersection of all returned node lists, keeping only nodes that satisfy every resource constraint simultaneously.

### Scoring and Final Selection

Once filtered, viable nodes pass through `runScoreFilter`, where each **scoring plugin** assigns a weighted score based on utilization metrics. The scheduler normalizes scores by `totalPluginWeight` and sorts the results using `result.AllSortByScore()`. Finally, `LeastRandomSelect` chooses among the top candidates according to `PrioritySelectNum`, favoring the least-loaded nodes while maintaining randomization to prevent thundering herds.

## Resource-Aware Filter Plugins

Filter plugins enforce hard constraints by examining specific resource dimensions. The most critical implementations reside in the `CubeMaster/pkg/selector/filter/` directory.

### CPU and Memory Constraints

The CPU filter in [`CubeMaster/pkg/selector/filter/cpufilter.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/selector/filter/cpufilter.go) calculates available capacity as `QuotaCpu - AllocatedCpu`. It returns only nodes where `quotaCpuFree` exceeds the requested millicores and `CpuUtil` remains below `NodeMaxCpuUtil`. Similarly, [`CubeMaster/pkg/selector/filter/memfilter.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/selector/filter/memfilter.go) evaluates `QuotaMem - AllocatedMem` against the requested memory count.

### Disk and NIC Utilization

The disk filter ([`CubeMaster/pkg/selector/filter/diskfilter.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/selector/filter/diskfilter.go)) monitors `DiskUsageMaxPercent`, rejecting nodes that exceed the configured threshold. For network resources, the NIC queue filter checks `MaxNICQueue` to prevent selection of nodes that have exhausted their hardware queue resources.

### Template Locality Awareness

[`CubeMaster/pkg/selector/filter/template_locality.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/selector/filter/template_locality.go) implements the `shouldSkipBackoffForTemplate` logic, ensuring that template-specific node affinity preferences are respected. This filter can override the standard back-off behavior when a template requires specific locality constraints.

## Scoring Plugin Architecture

After filtering eliminates unsuitable nodes, scoring plugins rank the remaining candidates according to policy-driven optimization goals.

### Weighted Scoring Calculation

Each scoring plugin returns a `node.NodeScore` slice containing raw scores. The scheduler multiplies these values by the plugin's `Weight()`, configurable via `SchedulerScoreConf`, then normalizes by dividing by `totalPluginWeight`. The aggregated results are stored in the selector context via `selCtx.SetNodeScoreList` before final sorting.

### Available Scoring Strategies

The `CubeMaster/pkg/selector/score/` directory contains implementations for:
- **CPU-utilisation score** – preferring nodes with lower active CPU usage
- **Memory-pressure score** – favoring nodes with greater free memory capacity
- **Affinity score** – respecting user-specified node affinity selectors

## Scheduling Workflow Integration

Scheduling requests originate in [`CubeMaster/pkg/service/sandbox/sandbox_run.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/service/sandbox/sandbox_run.go) when processing sandbox creation requests.

1. The system constructs a `createSandboxContext` containing requested resources.
2. `scheduler.AddBufferTask` enqueues the scheduling task.
3. The scheduler later invokes `scheduler.Select(selCtx)`, where `selCtx` provides resource amounts via `selCtx.GetResCpuFromCtx()` and `selCtx.GetResMemFromCtx()`.
4. The pipeline returns a `node.Node` containing the host IP and metadata, or an error if no suitable node exists.

## Configuration and Extensibility

Resource limits and scheduling behavior are controlled through [`CubeMaster/pkg/base/config/config.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/base/config/config.go). The **global scheduler configuration** exposes several critical parameters:

- **Quota calculations** – `EffectiveQuotaCpu` and `EffectiveAllocated` account for system overhead and per-instance-type reservations.
- **Back-off mode** – `InBackoffMode` toggles automatic retry with reduced node sets.
- **Circuit filter** – `DisableCircuitFilter` disables the circuit-breaker that removes nodes with repeated failures.

All configuration values support hot-reloading via the `hotswap` package, allowing runtime adjustments to `NodeMaxCpuUtil`, `DiskUsageMaxPercent`, and plugin weights without restarting the scheduler.

## Practical Example

The following code demonstrates how to build a selector context and request node placement:

```go
// Build a selector context containing the request resources.
sel := selctx.NewSelectorCtx().
    WithCpu(requestCPU).          // requestCPU is a *resource.Quantity
    WithMem(requestMem).          // requestMem is a *resource.Quantity
    WithNodeAffinity(selAffinity) // optional affinity constraints

// Ask CubeMaster to pick a node.
node, err := scheduler.Select(sel)
if err != nil {
    // Handle ErrNoRes, ErrSelectFailed, etc.
    log.Fatalf("no suitable node: %v", err)
}

// `node` now contains the host IP, instance type, and other metadata.
fmt.Printf("Selected node %s (type %s)\n", node.HostIP(), node.InstanceType)

```

## Summary

- CubeMaster implements a **multi-stage pipeline** comprising pre-filter, parallel resource filters, and weighted scoring phases.
- **Resource-aware filters** in [`cpufilter.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/cpufilter.go), [`memfilter.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/memfilter.go), and [`diskfilter.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/diskfilter.go) enforce hard constraints on CPU, memory, disk, and NIC availability.
- The **pluggable scoring system** aggregates weighted scores from CPU, memory, and affinity plugins to rank viable nodes.
- **Hot-reloadable configuration** allows runtime adjustment of resource thresholds and scheduling policies via the `hotswap` package.

## Frequently Asked Questions

### What happens if no nodes satisfy the resource requirements?

If the pre-filter and main filter stages return no candidates, the scheduler either enters **back-off mode** via `BackoffSelect` to retry with a narrower node subset, or returns `ErrNoRes` to indicate insufficient resources.

### Which interface must filter plugins implement?

All filter plugins must implement the `filter.Selector` interface, specifically the `Select(*selctx.SelectorCtx) (node.NodeList, error)` method signature, enabling the scheduler to invoke them uniformly during `parallelRunFilters`.

### How does CubeMaster calculate effective CPU and memory quotas?

The scheduler computes **effective quotas** using `EffectiveQuotaCpu` and `EffectiveAllocated` functions in the configuration module, which subtract system overhead and reservations from raw node capacity before comparing against sandbox requests.

### Can scheduling policies be updated without restarting CubeMaster?

Yes, the scheduler supports **hot-reloading** of configuration parameters including `NodeMaxCpuUtil`, `DiskUsageMaxPercent`, and plugin weights through the `hotswap` package, allowing policy adjustments without service interruption.