# CubeMaster Cluster Orchestration and Resource Scheduling Algorithms: A Deep Dive into the Multi-Stage Pipeline

> Explore CubeMaster's multi-stage scheduling pipeline: pre-filter, predicate, and scoring. Learn how this cluster orchestration and resource scheduling system efficiently allocates resources.

- Repository: [Tencent Cloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox)
- Tags: deep-dive
- Published: 2026-07-10

---

**CubeMaster implements a plugin-based, multi-stage scheduling pipeline that filters nodes through pre-filter, predicate, and scoring phases before making a final randomized selection from the highest-ranked candidates.**

CubeMaster, the control plane of TencentCloud's CubeSandbox container platform, employs sophisticated cluster orchestration and resource scheduling algorithms inspired by the Kubernetes scheduler design. The system uses a multi-stage pipeline that evaluates node suitability through hard constraints and soft scoring policies, enabling fine-grained placement of sandbox workloads across the cluster.

## The Scheduling Pipeline Architecture

The scheduling engine operates as a singleton initialized by `InitScheduler` in [`CubeMaster/pkg/scheduler/init.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/scheduler/init.go). The core entry point, `scheduler.Select`, defined in [`CubeMaster/pkg/scheduler/schedule.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/scheduler/schedule.go), orchestrates a six-phase pipeline that progressively narrows the candidate node pool.

### Pre-Filter Phase

The first phase quickly discards nodes that cannot satisfy basic requirements. Implemented in [`CubeMaster/pkg/selector/prefilter/prefilter.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/selector/prefilter/prefilter.go), the `prefilter.NewPreFilter()` function checks node health, readiness, and whether the node can possibly host the sandbox. This phase acts as a fast path to eliminate obviously unsuitable candidates before expensive computations occur.

### Back-off Filter

When a previous scheduling attempt has failed, the back-off filter handles retry logic with relaxed constraints. Located in [`CubeMaster/pkg/selector/backofffilter/backofffilter.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/selector/backofffilter/backofffilter.go), `backofffilter.NewBackoffFilter()` removes temporary restrictions—such as affinity rules—to allow the scheduler to retry placement on nodes that previously failed predicate checks.

### Filter (Predicate) Chain

The filter phase executes a series of selector plugins that enforce hard constraints. The `filter.NewSelector()` function in [`CubeMaster/pkg/selector/filter/selector.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/selector/filter/selector.go) initializes a chain where **all** selectors must pass for a node to survive. Individual filters include:

- **CPU Filter** ([`cpufilter.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/cpufilter.go)): Validates available CPU capacity against requests
- **Memory Filter** ([`memfilter.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/memfilter.go)): Checks memory availability and limits
- **Disk Filter** ([`diskfilter.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/diskfilter.go)): Verifies storage resources
- **Template Locality Filter** ([`template_locality.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/template_locality.go)): Ensures templates prefer nodes already holding related artifacts to reduce cross-node traffic
- **Third-Party Filter** ([`thirtpartyfilter.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/thirtpartyfilter.go)): Enables external plugins to inject custom predicates via the third-party interface
- **Realtime Create-Limit Filter** ([`realtimecreatelimit.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/realtimecreatelimit.go)): Caps concurrent sandbox creations per node to prevent spikes

Only nodes satisfying every filter constraint proceed to the scoring phase.

### Score (Priority) Chain

Surviving nodes receive numeric scores weighted by plugin-defined factors. The `score.NewSelector(ctx)` function in [`CubeMaster/pkg/selector/score/init.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/selector/score/init.go) manages scoring plugins including:

- **Realtime Score** ([`realtimescore.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/realtimescore.go)): Ranks nodes using live CPU, memory, and disk metrics from Redis
- **Affinity Score** ([`affinityscore.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/affinityscore.go)): Boosts nodes matching user-provided node-affinity selectors
- **Image Score** ([`imagescore.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/imagescore.go)): Prefers nodes caching required container image layers to reduce pull latency
- **Async Score** ([`asyncscore.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/asyncscore.go)): Allows asynchronous plugins to contribute additional weight for custom business logic

Scores are normalized and aggregated to produce a ranked node list.

### Post-Score and Final Selection

Optional post-processing occurs in `postscore.NewSelector()` within [`CubeMaster/pkg/selector/postscore/init.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/selector/postscore/init.go), enforcing whitelist constraints via [`whilelistscore.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/whilelistscore.go). Finally, the scheduler selects from the top-ranked nodes using `selCtx.LeastRandomSelect(config.GetConfig().Scheduler.PrioritySelectNum)` in [`schedule.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/schedule.go). This picks a random node from the highest-priority subset to avoid hotspotting.

## Plugin Architecture and Extensibility

All filters and scores implement the `filter.Selector` or `score.Selector` interfaces, making the cluster orchestration and resource scheduling algorithms fully extensible. Developers register custom plugins during initialization:

```go
type myFilter struct{}

func (f *myFilter) ID() string { return "myfilter" }

func (f *myFilter) Select(ctx *selctx.SelectorCtx) (node.NodeList, error) {
    // Custom predicate logic
    return nodeList, nil
}

func init() {
    // Register during scheduler init
    filter.Register("myfilter", func() filter.Selector { return &myFilter{} })
}

```

The scheduler loads registered plugins through `InitScheduler`, creating a deterministic yet customizable pipeline.

## Configuration and Tunables

Active plugins, their weights, and thresholds are controlled via the `SchedulerConf` section in [`CubeMaster/pkg/base/config/config.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/base/config/config.go). This configuration schema defines which algorithms participate in the scheduling decision and their relative importance, allowing operators to tune the cluster orchestration behavior without recompiling.

## Code Examples

Initialize the scheduler at master startup:

```go
// In CubeMaster main()
ctx := context.Background()
scheduler.InitScheduler(ctx)   // Loads all pre-filter, filter, score, post-score plugins

```

Select a node for a sandbox request:

```go
selCtx := selctx.NewSelectorCtx(ctx, req) // req contains CPU, memory, template ID, etc.
node, err := scheduler.Select(selCtx)      // Runs the entire pipeline
if err != nil {
    // Handle ErrNoRes, ErrPreSelect, etc.
}

```

## Summary

- CubeMaster uses a **six-phase scheduling pipeline**: pre-filter, back-off filter, predicate filter, score, post-score, and final selection.
- **Hard constraints** are enforced in the filter chain via plugins for CPU, memory, disk, template locality, and third-party predicates.
- **Soft scoring** ranks nodes using real-time metrics, affinity rules, image locality, and asynchronous weights.
- The architecture is **fully plugin-based**, allowing custom selectors by implementing `filter.Selector` or `score.Selector` interfaces.
- Final node selection uses **randomized picking** from the highest-scored subset to prevent hotspots.
- Configuration is driven by `SchedulerConf` in [`CubeMaster/pkg/base/config/config.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/base/config/config.go).

## Frequently Asked Questions

### How does CubeMaster handle scheduling failures and retries?

When a sandbox fails to schedule, the back-off filter in [`CubeMaster/pkg/selector/backofffilter/backofffilter.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/selector/backofffilter/backofffilter.go) relaxes constraints such as affinity rules and allows the scheduler to retry placement. This prevents permanent scheduling failures for pods that initially fail due to temporary resource constraints.

### Can I add custom scheduling logic to CubeMaster without modifying core code?

Yes. CubeMaster supports third-party plugins through the [`thirtpartyfilter.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/thirtpartyfilter.go) interface for predicates and the [`asyncscore.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/asyncscore.go) mechanism for scoring. Implement the `filter.Selector` or `score.Selector` interfaces, register your plugin in an `init()` function, and include it in the `SchedulerConf` configuration.

### What is the difference between the filter chain and the score chain in CubeMaster?

The **filter chain** (implemented in [`CubeMaster/pkg/selector/filter/selector.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/selector/filter/selector.go)) enforces hard constraints using predicates—nodes must satisfy **all** filters to proceed. The **score chain** (in [`CubeMaster/pkg/selector/score/init.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/selector/score/init.go)) assigns weighted numeric values to surviving nodes, allowing the scheduler to pick the best-fit candidates rather than simply the first valid one.

### How does CubeMaster prevent scheduling hotspots on highly-scored nodes?

After scoring, CubeMaster uses `LeastRandomSelect` with a configurable `PrioritySelectNum` parameter defined in [`CubeMaster/pkg/scheduler/schedule.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/scheduler/schedule.go). This selects a random node from the top N highest-scoring candidates rather than always picking the absolute highest score, distributing load across similarly suitable nodes.