CubeMaster Cluster Orchestration and Resource Scheduling Architecture: A Multi-Stage Pipeline Deep Dive

CubeMaster implements a pluggable, multi-stage scheduling pipeline that processes cluster placement decisions through six distinct phases—from pre-filtering to randomized final selection—mirroring the extensible design of Kubernetes' scheduler but optimized for sandbox workloads.

The CubeMaster component within the TencentCloud/CubeSandbox repository provides the control plane for cluster orchestration and resource scheduling, offering a deterministic, metrics-driven approach to placing sandbox workloads across node clusters. According to the CubeSandbox source code, the architecture follows a plugin-based pipeline pattern that separates hard constraint validation from priority scoring, enabling operators to customize scheduling behavior without modifying core logic.

Multi-Stage Scheduling Pipeline

The CubeMaster scheduling engine processes every placement request through a rigid sequence of filtering and scoring phases. Each phase is implemented as a discrete plugin, allowing the system to rapidly eliminate unsuitable nodes before applying expensive scoring calculations.

Pre-Filter Phase

The Pre-Filter phase performs rapid disqualification of nodes that cannot possibly satisfy a request. Implemented in CubeMaster/pkg/selector/prefilter/prefilter.go, this stage checks basic health, node readiness, and whether the node can host the requested sandbox type. Nodes failing this phase are immediately excluded from subsequent processing.

Back-Off Filter

When a previous scheduling attempt has failed, the Back-Off Filter—defined in CubeMaster/pkg/selector/backofffilter/backofffilter.go—relaxes constraints temporarily to enable retry logic. This component prevents the scheduler from repeatedly attempting impossible placements while allowing eventual placement when cluster conditions change.

Filter (Predicate) Chain

The Filter Chain enforces hard constraints through a series of selector plugins that must all return true for a node to survive. Located in CubeMaster/pkg/selector/filter/selector.go and individual selector files, this phase includes:

• CPU Filter (cpufilter.go): Validates available CPU capacity
• Memory Filter (memfilter.go): Checks RAM availability
• Disk Filter (diskfilter.go): Verifies storage resources
• Template Locality Filter (template_locality.go): Prefers nodes with existing artifacts to reduce cross-node traffic
• Third-Party Filter (thirtpartyfilter.go): Executes external predicates via plugin interface
• Realtime Create-Limit Filter (realtimecreatelimit.go): Caps concurrent sandbox creations per node

Score (Priority) Chain

Surviving nodes enter the Score Chain, where each node receives a numeric score weighted by plugin-defined factors. Implemented in CubeMaster/pkg/selector/score/init.go and concrete score files, this phase includes:

• Realtime Score (realtimescore.go): Ranks nodes using live CPU, memory, and disk metrics from Redis
• Affinity Score (affinityscore.go): Boosts nodes matching user-defined affinity selectors
• Image Score (imagescore.go): Prefers nodes with cached container image layers
• Async Score (asyncscore.go): Incorporates asynchronous business logic weights

Scores are normalized and aggregated, with the highest-scoring nodes advancing to final selection.

Post-Score Processing

The Post-Score phase applies optional final constraints, such as whitelist enforcement (CubeMaster/pkg/selector/postscore/whilelistscore.go), after all scoring calculations complete.

Final Selection

Rather than always selecting the highest-scoring node—which could cause hotspotting—the scheduler uses LeastRandomSelect from CubeMaster/pkg/scheduler/schedule.go. This method selects a small priority subset (configured via PrioritySelectNum) and randomly picks the final node, distributing load across top candidates.

Core Scheduling Components

The pipeline is orchestrated by a scheduler singleton initialized via InitScheduler in CubeMaster/pkg/scheduler/init.go. The system entry point is scheduler.Select, which executes the complete pipeline against a selector context.

Key implementation files:

• CubeMaster/pkg/scheduler/schedule.go — Contains scheduler.Select and back-off handling logic
• CubeMaster/pkg/scheduler/init.go — Constructs the scheduler singleton and wires plugins
• CubeMaster/pkg/base/config/config.go — Defines SchedulerConf for plugin configuration

Pluggable Extension Architecture

CubeMaster's resource scheduling architecture supports dynamic plugin registration. Developers implement the filter.Selector or score.Selector interfaces to inject custom logic without recompiling the master binary.

Registering a custom filter:

type myFilter struct{}

func (f *myFilter) ID() string { return "myfilter" }

func (f *myFilter) Select(ctx *selctx.SelectorCtx) (node.NodeList, error) {
    // Custom predicate logic
    return nodeList, nil
}

func init() {
    filter.Register("myfilter", func() filter.Selector { return &myFilter{} })
}

Plugins are activated via the SchedulerConf section in the master configuration, where operators specify active selectors, weights, and thresholds.

Initialization and Configuration

The scheduling system initializes at master startup through InitScheduler, which loads all pre-filter, filter, score, and post-score plugins according to SchedulerConf.

Initializing the scheduler:

ctx := context.Background()
scheduler.InitScheduler(ctx)   // Loads all pipeline plugins

Selecting a node for a sandbox:

selCtx := selctx.NewSelectorCtx(ctx, req) // req contains CPU, memory, template ID
node, err := scheduler.Select(selCtx)      // Executes full pipeline
if err != nil {
    // Handle ErrNoRes, ErrPreSelect, etc.
}

Configuration parameters—including PrioritySelectNum for final selection randomization—are defined in CubeMaster/pkg/base/config/config.go within the SchedulerConf struct (lines 67-82).

Summary

• CubeMaster uses a six-phase scheduling pipeline (Pre-filter, Back-off, Filter, Score, Post-score, Final Selection) for cluster orchestration.
• Hard constraints are enforced in the Filter Chain through individual selector plugins for CPU, memory, disk, and template locality.
• Priority scoring combines real-time metrics, affinity rules, and image locality to rank viable nodes.
• The final selection randomly chooses from a priority subset to prevent hotspotting.
• All filters and scores are pluggable via the filter.Selector and score.Selector interfaces, registered during InitScheduler.
• Configuration is controlled through SchedulerConf in CubeMaster/pkg/base/config/config.go.

Frequently Asked Questions

How does CubeMaster prevent scheduling hotspots on high-scoring nodes?

Instead of deterministically selecting the highest-scoring node, CubeMaster uses LeastRandomSelect configured with PrioritySelectNum. This method identifies the top N scoring nodes and randomly selects the final placement target among them, distributing load across the best candidates while avoiding concentration on a single node.

Can I add custom scheduling logic without modifying CubeMaster's core code?

Yes. CubeMaster supports plugin-based extension through the filter.Selector and score.Selector interfaces. You implement the interface methods, register your plugin in an init() function, and configure it via SchedulerConf. The scheduler loads custom plugins during InitScheduler initialization without requiring core modifications.

What is the difference between the Filter Chain and Score Chain in CubeMaster?

The Filter Chain (Predicate) enforces hard constraints—nodes must satisfy all active filters (CPU, memory, disk, etc.) to proceed. The Score Chain (Priority) assigns numeric weights to surviving nodes based on soft preferences like real-time metrics, affinity, and image locality. Filters eliminate nodes; scores rank them.

Where does CubeMaster store real-time metrics for scheduling decisions?

CubeMaster retrieves live metrics (CPU, memory, disk usage) from Redis during the Score Chain phase, specifically within realtimescore.go. This allows the scheduler to make placement decisions based on current cluster state rather than static allocation tables.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}