# CubeMaster Cluster Orchestration and Resource Scheduling Architecture: A Multi-Stage Pipeline Deep Dive

> Explore CubeMaster's multi-stage pipeline architecture for Kubernetes-like cluster orchestration and resource scheduling, optimized for sandbox workloads. Learn about its six distinct scheduling phases.

- Repository: [Tencent Cloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox)
- Tags: architecture
- Published: 2026-07-11

---

**CubeMaster implements a pluggable, multi-stage scheduling pipeline that processes cluster placement decisions through six distinct phases—from pre-filtering to randomized final selection—mirroring the extensible design of Kubernetes' scheduler but optimized for sandbox workloads.**

The CubeMaster component within the TencentCloud/CubeSandbox repository provides the control plane for cluster orchestration and resource scheduling, offering a deterministic, metrics-driven approach to placing sandbox workloads across node clusters. According to the CubeSandbox source code, the architecture follows a **plugin-based pipeline pattern** that separates hard constraint validation from priority scoring, enabling operators to customize scheduling behavior without modifying core logic.

## Multi-Stage Scheduling Pipeline

The CubeMaster scheduling engine processes every placement request through a rigid sequence of filtering and scoring phases. Each phase is implemented as a discrete plugin, allowing the system to rapidly eliminate unsuitable nodes before applying expensive scoring calculations.

### Pre-Filter Phase

The **Pre-Filter** phase performs rapid disqualification of nodes that cannot possibly satisfy a request. Implemented in [`CubeMaster/pkg/selector/prefilter/prefilter.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/selector/prefilter/prefilter.go), this stage checks basic health, node readiness, and whether the node can host the requested sandbox type. Nodes failing this phase are immediately excluded from subsequent processing.

### Back-Off Filter

When a previous scheduling attempt has failed, the **Back-Off Filter**—defined in [`CubeMaster/pkg/selector/backofffilter/backofffilter.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/selector/backofffilter/backofffilter.go)—relaxes constraints temporarily to enable retry logic. This component prevents the scheduler from repeatedly attempting impossible placements while allowing eventual placement when cluster conditions change.

### Filter (Predicate) Chain

The **Filter Chain** enforces hard constraints through a series of selector plugins that must **all** return true for a node to survive. Located in [`CubeMaster/pkg/selector/filter/selector.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/selector/filter/selector.go) and individual selector files, this phase includes:
- **CPU Filter** ([`cpufilter.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/cpufilter.go)): Validates available CPU capacity
- **Memory Filter** ([`memfilter.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/memfilter.go)): Checks RAM availability
- **Disk Filter** ([`diskfilter.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/diskfilter.go)): Verifies storage resources
- **Template Locality Filter** ([`template_locality.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/template_locality.go)): Prefers nodes with existing artifacts to reduce cross-node traffic
- **Third-Party Filter** ([`thirtpartyfilter.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/thirtpartyfilter.go)): Executes external predicates via plugin interface
- **Realtime Create-Limit Filter** ([`realtimecreatelimit.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/realtimecreatelimit.go)): Caps concurrent sandbox creations per node

### Score (Priority) Chain

Surviving nodes enter the **Score Chain**, where each node receives a numeric score weighted by plugin-defined factors. Implemented in [`CubeMaster/pkg/selector/score/init.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/selector/score/init.go) and concrete score files, this phase includes:
- **Realtime Score** ([`realtimescore.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/realtimescore.go)): Ranks nodes using live CPU, memory, and disk metrics from Redis
- **Affinity Score** ([`affinityscore.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/affinityscore.go)): Boosts nodes matching user-defined affinity selectors
- **Image Score** ([`imagescore.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/imagescore.go)): Prefers nodes with cached container image layers
- **Async Score** ([`asyncscore.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/asyncscore.go)): Incorporates asynchronous business logic weights

Scores are normalized and aggregated, with the highest-scoring nodes advancing to final selection.

### Post-Score Processing

The **Post-Score** phase applies optional final constraints, such as whitelist enforcement ([`CubeMaster/pkg/selector/postscore/whilelistscore.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/selector/postscore/whilelistscore.go)), after all scoring calculations complete.

### Final Selection

Rather than always selecting the highest-scoring node—which could cause hotspotting—the scheduler uses `LeastRandomSelect` from [`CubeMaster/pkg/scheduler/schedule.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/scheduler/schedule.go). This method selects a small priority subset (configured via `PrioritySelectNum`) and randomly picks the final node, distributing load across top candidates.

## Core Scheduling Components

The pipeline is orchestrated by a scheduler singleton initialized via `InitScheduler` in [`CubeMaster/pkg/scheduler/init.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/scheduler/init.go). The system entry point is `scheduler.Select`, which executes the complete pipeline against a selector context.

**Key implementation files:**
- [`CubeMaster/pkg/scheduler/schedule.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/scheduler/schedule.go) — Contains `scheduler.Select` and back-off handling logic
- [`CubeMaster/pkg/scheduler/init.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/scheduler/init.go) — Constructs the scheduler singleton and wires plugins
- [`CubeMaster/pkg/base/config/config.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/base/config/config.go) — Defines `SchedulerConf` for plugin configuration

## Pluggable Extension Architecture

CubeMaster's resource scheduling architecture supports dynamic plugin registration. Developers implement the `filter.Selector` or `score.Selector` interfaces to inject custom logic without recompiling the master binary.

**Registering a custom filter:**

```go
type myFilter struct{}

func (f *myFilter) ID() string { return "myfilter" }

func (f *myFilter) Select(ctx *selctx.SelectorCtx) (node.NodeList, error) {
    // Custom predicate logic
    return nodeList, nil
}

func init() {
    filter.Register("myfilter", func() filter.Selector { return &myFilter{} })
}

```

Plugins are activated via the `SchedulerConf` section in the master configuration, where operators specify active selectors, weights, and thresholds.

## Initialization and Configuration

The scheduling system initializes at master startup through `InitScheduler`, which loads all pre-filter, filter, score, and post-score plugins according to `SchedulerConf`.

**Initializing the scheduler:**

```go
ctx := context.Background()
scheduler.InitScheduler(ctx)   // Loads all pipeline plugins

```

**Selecting a node for a sandbox:**

```go
selCtx := selctx.NewSelectorCtx(ctx, req) // req contains CPU, memory, template ID
node, err := scheduler.Select(selCtx)      // Executes full pipeline
if err != nil {
    // Handle ErrNoRes, ErrPreSelect, etc.
}

```

Configuration parameters—including `PrioritySelectNum` for final selection randomization—are defined in [`CubeMaster/pkg/base/config/config.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/base/config/config.go) within the `SchedulerConf` struct (lines 67-82).

## Summary

- CubeMaster uses a **six-phase scheduling pipeline** (Pre-filter, Back-off, Filter, Score, Post-score, Final Selection) for cluster orchestration.
- **Hard constraints** are enforced in the Filter Chain through individual selector plugins for CPU, memory, disk, and template locality.
- **Priority scoring** combines real-time metrics, affinity rules, and image locality to rank viable nodes.
- The **final selection** randomly chooses from a priority subset to prevent hotspotting.
- All filters and scores are **pluggable** via the `filter.Selector` and `score.Selector` interfaces, registered during `InitScheduler`.
- Configuration is controlled through `SchedulerConf` in [`CubeMaster/pkg/base/config/config.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/base/config/config.go).

## Frequently Asked Questions

### How does CubeMaster prevent scheduling hotspots on high-scoring nodes?

Instead of deterministically selecting the highest-scoring node, CubeMaster uses `LeastRandomSelect` configured with `PrioritySelectNum`. This method identifies the top *N* scoring nodes and randomly selects the final placement target among them, distributing load across the best candidates while avoiding concentration on a single node.

### Can I add custom scheduling logic without modifying CubeMaster's core code?

Yes. CubeMaster supports plugin-based extension through the `filter.Selector` and `score.Selector` interfaces. You implement the interface methods, register your plugin in an `init()` function, and configure it via `SchedulerConf`. The scheduler loads custom plugins during `InitScheduler` initialization without requiring core modifications.

### What is the difference between the Filter Chain and Score Chain in CubeMaster?

The **Filter Chain** (Predicate) enforces hard constraints—nodes must satisfy all active filters (CPU, memory, disk, etc.) to proceed. The **Score Chain** (Priority) assigns numeric weights to surviving nodes based on soft preferences like real-time metrics, affinity, and image locality. Filters eliminate nodes; scores rank them.

### Where does CubeMaster store real-time metrics for scheduling decisions?

CubeMaster retrieves live metrics (CPU, memory, disk usage) from Redis during the Score Chain phase, specifically within [`realtimescore.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/realtimescore.go). This allows the scheduler to make placement decisions based on current cluster state rather than static allocation tables.