# How CubeMaster Orchestrates Multi-Node Cluster Scheduling: Plugin Pipeline Deep Dive

> Learn how CubeMaster orchestrates multi-node cluster scheduling using its five-stage plugin pipeline for optimized sandbox placement across heterogeneous clusters.

- Repository: [Tencent Cloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox)
- Tags: deep-dive
- Published: 2026-07-09

---

**CubeMaster orchestrates multi-node cluster scheduling through a five-stage plugin pipeline that filters nodes by health and resources, scores them by weighted metrics, and selects the least-loaded candidate to optimize sandbox placement across heterogeneous clusters.**

The `CubeMaster` component in the TencentCloud/CubeSandbox repository serves as the control plane for distributed sandbox workloads. Understanding how it handles multi-node cluster scheduling requires examining its modular scheduler package, which implements a Kubernetes-inspired plugin architecture to determine optimal node placement for each sandbox or template request.

## The Five-Stage Scheduling Pipeline

The scheduler operates as a sequential pipeline defined in [`CubeMaster/pkg/scheduler/schedule.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/scheduler/schedule.go). Each stage is implemented as a pluggable interface, allowing operators to customize behavior through configuration without modifying core logic.

### Stage 1: Pre-Filter Sanity Checks

The pipeline begins with `runPreFilter`, implemented in [`CubeMaster/pkg/selector/prefilter/pre_filter.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/selector/prefilter/pre_filter.go). This stage performs rapid disqualification of unhealthy nodes by verifying basic resource availability, node health status, and template-specific constraints. If a node fails these checks, it is immediately excluded from consideration.

### Stage 2: Back-Off Filter Fallback

When the pre-filter rejects all candidate nodes—typically during temporary cluster overloads—the scheduler invokes `runBackoffFilter` from [`CubeMaster/pkg/selector/backofffilter/backoff_filter.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/selector/backofffilter/backoff_filter.go). This softer selector provides a degradation path for high-availability scenarios, allowing scheduling to proceed with relaxed constraints for non-template workloads.

### Stage 3: Parallel Filter Stage

The `runFilter` method executes independent selector plugins concurrently. Each selector implements the `Select(*selctx.SelectorCtx) (node.NodeList, error)` interface. Key filters include:

- **Node Affinity** ([`selector/filter/affinityfilter.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/selector/filter/affinityfilter.go)): Enforces label and taint matching requirements.
- **Resource Filter** ([`selector/filter/resfilter.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/selector/filter/resfilter.go)): Validates CPU, memory, and disk quotas against node capacity.
- **Topology Filters**: Ensure geographic or rack-aware distribution constraints.

Only nodes satisfying **all** parallel filters advance to the scoring stage.

### Stage 4: Weighted Score Stage

Surviving nodes enter `runScoreFilter`, where multiple scoring plugins assign numeric rankings. Located in `CubeMaster/pkg/selector/score/*.go`, these plugins include:

- **Real-Time Score** ([`realtimescore.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/realtimescore.go)): Incorporates current node load metrics.
- **Multi-Factor Score** ([`multifactorscore.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/multifactorscore.go)): Balances historical performance data.
- **Image Score** ([`imagescore.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/imagescore.go)): Prioritizes nodes with cached container images.

Weights are configured via `Scheduler.Score.ScorePluginConf` in the global config, allowing fine-tuned balancing between latency, resource efficiency, and cache locality.

### Stage 5: Final Selection with Load Balancing

The `LeastRandomSelect` method (called from [`schedule.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/schedule.go) line 56) completes the pipeline. It identifies the `PrioritySelectNum` least-loaded nodes from the scored list, then randomly selects one finalist to prevent hot-spotting. This probabilistic approach ensures load distribution remains uniform across the cluster over time.

## Scheduler Initialization and Context Setup

### Bootstrap in init.go

During startup, `InitScheduler` in [`CubeMaster/pkg/scheduler/init.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/scheduler/init.go) registers all pipeline components. It instantiates the pre-selector, back-off selector, filter collection, and score collection. The function also launches `initTask` to warm up node caches before accepting scheduling requests.

### Building the Selector Context

Each scheduling request creates a `SelectorCtx` via [`CubeMaster/pkg/scheduler/selctx/selectcontext.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/scheduler/selctx/selectcontext.go). This context encapsulates:

- Request resources (CPU, memory, template ID)
- The `LeastSelectName` configuration parameter (`config.GetConfig().Scheduler.LeastSelectName`)
- Intermediate node lists passed between pipeline stages

The context pattern ensures thread-safe data sharing across the parallel filter stage.

## Entry Point and Sandbox Execution Flow

Sandbox creation triggers the scheduler from [`CubeMaster/pkg/service/sandbox/sandbox_run.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/service/sandbox/sandbox_run.go). The `schedule()` method constructs the selector context and invokes the pipeline:

```go
// CubeMaster/pkg/service/sandbox/sandbox_run.go (excerpt)
func (c *createSandboxContext) schedule() (err error) {
    // Build selector context with the request's resources
    c.selctx = selctx.New(config.GetConfig().Scheduler.LeastSelectName)
    c.selctx.ReqRes = &selctx.RequestResources{
        TemplateID:  c.req.TemplateID,
        CPU:         c.req.Cpu,
        Memory:      c.req.Memory,
        // … other fields …
    }

    // Invoke the scheduler pipeline
    c.selectHost, err = scheduler.Select(c.selctx)
    if err != nil {
        return ret.Err(errorcode.ErrorCode_SelectNodesNoRes, scheduler.ErrNoRes.Error())
    }
    return nil
}

```

The returned `selectHost` identifies the target node IP, which the Cubelet agent uses to instantiate the sandbox via RPC.

## Extending the Scheduler

The plugin architecture enables customization by implementing the `filter.Selector` or `score.Selector` interfaces. New selectors register alongside existing ones in `InitScheduler`, while scoring weights adjust through YAML configuration. This extensibility allows CubeMaster to adapt to diverse hardware profiles and workload characteristics without core code changes.

## Summary

- **CubeMaster** implements multi-node cluster scheduling via a five-stage plugin pipeline defined in [`CubeMaster/pkg/scheduler/schedule.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/scheduler/schedule.go).
- **Pre-filter** and **back-off filter** stages handle health checks and degradation scenarios before parallel filtering.
- **Filter plugins** run concurrently to enforce affinity, resource, and topology constraints.
- **Weighted scorers** rank viable nodes using real-time metrics, configurable via `ScorePluginConf`.
- **LeastRandomSelect** chooses among the top `PrioritySelectNum` nodes to balance load distribution.
- The entry point in [`sandbox_run.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/sandbox_run.go) line 420 triggers the entire pipeline via `scheduler.Select(c.selctx)`.

## Frequently Asked Questions

### How does CubeMaster handle scheduling when all nodes are overloaded?

When the pre-filter rejects all nodes, the scheduler activates the back-off filter defined in [`CubeMaster/pkg/selector/backofffilter/backoff_filter.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/selector/backofffilter/backoff_filter.go). This fallback mechanism relaxes constraints for non-template workloads, allowing critical sandboxes to schedule on nodes that would otherwise be excluded due to temporary resource pressure.

### Can operators customize the scoring weights for node selection?

Yes. The scheduler reads scoring weights from `config.GetConfig().Scheduler.Score.ScorePluginConf` during initialization. Operators can adjust the relative importance of real-time metrics, image caching, and multi-factor scores without recompiling the binary, enabling fine-tuning for specific workload patterns.

### What prevents the scheduler from always choosing the same optimal node?

The `LeastRandomSelect` algorithm in [`CubeMaster/pkg/scheduler/schedule.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/scheduler/schedule.go) implements a two-tier selection: it first identifies the `PrioritySelectNum` least-loaded candidates, then randomly selects one from that subset. This randomization prevents hot-spotting and ensures traffic distributes evenly across the cluster over time.

### Where does the scheduling context originate during sandbox creation?

The `createSandboxContext` in [`CubeMaster/pkg/service/sandbox/sandbox_run.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/service/sandbox/sandbox_run.go) constructs the `SelectorCtx` at line 420. It populates the context with the request's CPU, memory, and template ID requirements, then passes it to `scheduler.Select()` to execute the multi-stage filtering and scoring pipeline.