CubeMaster Cluster Orchestration and Resource Scheduling Algorithms: A Deep Dive into the Multi-Stage Pipeline

CubeMaster implements a plugin-based, multi-stage scheduling pipeline that filters nodes through pre-filter, predicate, and scoring phases before making a final randomized selection from the highest-ranked candidates.

CubeMaster, the control plane of TencentCloud's CubeSandbox container platform, employs sophisticated cluster orchestration and resource scheduling algorithms inspired by the Kubernetes scheduler design. The system uses a multi-stage pipeline that evaluates node suitability through hard constraints and soft scoring policies, enabling fine-grained placement of sandbox workloads across the cluster.

The Scheduling Pipeline Architecture

The scheduling engine operates as a singleton initialized by InitScheduler in CubeMaster/pkg/scheduler/init.go. The core entry point, scheduler.Select, defined in CubeMaster/pkg/scheduler/schedule.go, orchestrates a six-phase pipeline that progressively narrows the candidate node pool.

Pre-Filter Phase

The first phase quickly discards nodes that cannot satisfy basic requirements. Implemented in CubeMaster/pkg/selector/prefilter/prefilter.go, the prefilter.NewPreFilter() function checks node health, readiness, and whether the node can possibly host the sandbox. This phase acts as a fast path to eliminate obviously unsuitable candidates before expensive computations occur.

Back-off Filter

When a previous scheduling attempt has failed, the back-off filter handles retry logic with relaxed constraints. Located in CubeMaster/pkg/selector/backofffilter/backofffilter.go, backofffilter.NewBackoffFilter() removes temporary restrictions—such as affinity rules—to allow the scheduler to retry placement on nodes that previously failed predicate checks.

Filter (Predicate) Chain

The filter phase executes a series of selector plugins that enforce hard constraints. The filter.NewSelector() function in CubeMaster/pkg/selector/filter/selector.go initializes a chain where all selectors must pass for a node to survive. Individual filters include:

• CPU Filter (cpufilter.go): Validates available CPU capacity against requests
• Memory Filter (memfilter.go): Checks memory availability and limits
• Disk Filter (diskfilter.go): Verifies storage resources
• Template Locality Filter (template_locality.go): Ensures templates prefer nodes already holding related artifacts to reduce cross-node traffic
• Third-Party Filter (thirtpartyfilter.go): Enables external plugins to inject custom predicates via the third-party interface
• Realtime Create-Limit Filter (realtimecreatelimit.go): Caps concurrent sandbox creations per node to prevent spikes

Only nodes satisfying every filter constraint proceed to the scoring phase.

Score (Priority) Chain

Surviving nodes receive numeric scores weighted by plugin-defined factors. The score.NewSelector(ctx) function in CubeMaster/pkg/selector/score/init.go manages scoring plugins including:

• Realtime Score (realtimescore.go): Ranks nodes using live CPU, memory, and disk metrics from Redis
• Affinity Score (affinityscore.go): Boosts nodes matching user-provided node-affinity selectors
• Image Score (imagescore.go): Prefers nodes caching required container image layers to reduce pull latency
• Async Score (asyncscore.go): Allows asynchronous plugins to contribute additional weight for custom business logic

Scores are normalized and aggregated to produce a ranked node list.

Post-Score and Final Selection

Optional post-processing occurs in postscore.NewSelector() within CubeMaster/pkg/selector/postscore/init.go, enforcing whitelist constraints via whilelistscore.go. Finally, the scheduler selects from the top-ranked nodes using selCtx.LeastRandomSelect(config.GetConfig().Scheduler.PrioritySelectNum) in schedule.go. This picks a random node from the highest-priority subset to avoid hotspotting.

Plugin Architecture and Extensibility

All filters and scores implement the filter.Selector or score.Selector interfaces, making the cluster orchestration and resource scheduling algorithms fully extensible. Developers register custom plugins during initialization:

type myFilter struct{}

func (f *myFilter) ID() string { return "myfilter" }

func (f *myFilter) Select(ctx *selctx.SelectorCtx) (node.NodeList, error) {
    // Custom predicate logic
    return nodeList, nil
}

func init() {
    // Register during scheduler init
    filter.Register("myfilter", func() filter.Selector { return &myFilter{} })
}

The scheduler loads registered plugins through InitScheduler, creating a deterministic yet customizable pipeline.

Configuration and Tunables

Active plugins, their weights, and thresholds are controlled via the SchedulerConf section in CubeMaster/pkg/base/config/config.go. This configuration schema defines which algorithms participate in the scheduling decision and their relative importance, allowing operators to tune the cluster orchestration behavior without recompiling.

Code Examples

Initialize the scheduler at master startup:

// In CubeMaster main()
ctx := context.Background()
scheduler.InitScheduler(ctx)   // Loads all pre-filter, filter, score, post-score plugins

Select a node for a sandbox request:

selCtx := selctx.NewSelectorCtx(ctx, req) // req contains CPU, memory, template ID, etc.
node, err := scheduler.Select(selCtx)      // Runs the entire pipeline
if err != nil {
    // Handle ErrNoRes, ErrPreSelect, etc.
}

Summary

• CubeMaster uses a six-phase scheduling pipeline: pre-filter, back-off filter, predicate filter, score, post-score, and final selection.
• Hard constraints are enforced in the filter chain via plugins for CPU, memory, disk, template locality, and third-party predicates.
• Soft scoring ranks nodes using real-time metrics, affinity rules, image locality, and asynchronous weights.
• The architecture is fully plugin-based, allowing custom selectors by implementing filter.Selector or score.Selector interfaces.
• Final node selection uses randomized picking from the highest-scored subset to prevent hotspots.
• Configuration is driven by SchedulerConf in CubeMaster/pkg/base/config/config.go.

Frequently Asked Questions

How does CubeMaster handle scheduling failures and retries?

When a sandbox fails to schedule, the back-off filter in CubeMaster/pkg/selector/backofffilter/backofffilter.go relaxes constraints such as affinity rules and allows the scheduler to retry placement. This prevents permanent scheduling failures for pods that initially fail due to temporary resource constraints.

Can I add custom scheduling logic to CubeMaster without modifying core code?

Yes. CubeMaster supports third-party plugins through the thirtpartyfilter.go interface for predicates and the asyncscore.go mechanism for scoring. Implement the filter.Selector or score.Selector interfaces, register your plugin in an init() function, and include it in the SchedulerConf configuration.

What is the difference between the filter chain and the score chain in CubeMaster?

The filter chain (implemented in CubeMaster/pkg/selector/filter/selector.go) enforces hard constraints using predicates—nodes must satisfy all filters to proceed. The score chain (in CubeMaster/pkg/selector/score/init.go) assigns weighted numeric values to surviving nodes, allowing the scheduler to pick the best-fit candidates rather than simply the first valid one.

How does CubeMaster prevent scheduling hotspots on highly-scored nodes?

After scoring, CubeMaster uses LeastRandomSelect with a configurable PrioritySelectNum parameter defined in CubeMaster/pkg/scheduler/schedule.go. This selects a random node from the top N highest-scoring candidates rather than always picking the absolute highest score, distributing load across similarly suitable nodes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}