How CubeMaster Orchestrates Multi-Node Cluster Scheduling: Plugin Pipeline Deep Dive

CubeMaster orchestrates multi-node cluster scheduling through a five-stage plugin pipeline that filters nodes by health and resources, scores them by weighted metrics, and selects the least-loaded candidate to optimize sandbox placement across heterogeneous clusters.

The CubeMaster component in the TencentCloud/CubeSandbox repository serves as the control plane for distributed sandbox workloads. Understanding how it handles multi-node cluster scheduling requires examining its modular scheduler package, which implements a Kubernetes-inspired plugin architecture to determine optimal node placement for each sandbox or template request.

The Five-Stage Scheduling Pipeline

The scheduler operates as a sequential pipeline defined in CubeMaster/pkg/scheduler/schedule.go. Each stage is implemented as a pluggable interface, allowing operators to customize behavior through configuration without modifying core logic.

Stage 1: Pre-Filter Sanity Checks

The pipeline begins with runPreFilter, implemented in CubeMaster/pkg/selector/prefilter/pre_filter.go. This stage performs rapid disqualification of unhealthy nodes by verifying basic resource availability, node health status, and template-specific constraints. If a node fails these checks, it is immediately excluded from consideration.

Stage 2: Back-Off Filter Fallback

When the pre-filter rejects all candidate nodes—typically during temporary cluster overloads—the scheduler invokes runBackoffFilter from CubeMaster/pkg/selector/backofffilter/backoff_filter.go. This softer selector provides a degradation path for high-availability scenarios, allowing scheduling to proceed with relaxed constraints for non-template workloads.

Stage 3: Parallel Filter Stage

The runFilter method executes independent selector plugins concurrently. Each selector implements the Select(*selctx.SelectorCtx) (node.NodeList, error) interface. Key filters include:

• Node Affinity (selector/filter/affinityfilter.go): Enforces label and taint matching requirements.
• Resource Filter (selector/filter/resfilter.go): Validates CPU, memory, and disk quotas against node capacity.
• Topology Filters: Ensure geographic or rack-aware distribution constraints.

Only nodes satisfying all parallel filters advance to the scoring stage.

Stage 4: Weighted Score Stage

Surviving nodes enter runScoreFilter, where multiple scoring plugins assign numeric rankings. Located in CubeMaster/pkg/selector/score/*.go, these plugins include:

• Real-Time Score (realtimescore.go): Incorporates current node load metrics.
• Multi-Factor Score (multifactorscore.go): Balances historical performance data.
• Image Score (imagescore.go): Prioritizes nodes with cached container images.

Weights are configured via Scheduler.Score.ScorePluginConf in the global config, allowing fine-tuned balancing between latency, resource efficiency, and cache locality.

Stage 5: Final Selection with Load Balancing

The LeastRandomSelect method (called from schedule.go line 56) completes the pipeline. It identifies the PrioritySelectNum least-loaded nodes from the scored list, then randomly selects one finalist to prevent hot-spotting. This probabilistic approach ensures load distribution remains uniform across the cluster over time.

Scheduler Initialization and Context Setup

Bootstrap in init.go

During startup, InitScheduler in CubeMaster/pkg/scheduler/init.go registers all pipeline components. It instantiates the pre-selector, back-off selector, filter collection, and score collection. The function also launches initTask to warm up node caches before accepting scheduling requests.

Building the Selector Context

Each scheduling request creates a SelectorCtx via CubeMaster/pkg/scheduler/selctx/selectcontext.go. This context encapsulates:

• Request resources (CPU, memory, template ID)
• The LeastSelectName configuration parameter (config.GetConfig().Scheduler.LeastSelectName)
• Intermediate node lists passed between pipeline stages

The context pattern ensures thread-safe data sharing across the parallel filter stage.

Entry Point and Sandbox Execution Flow

Sandbox creation triggers the scheduler from CubeMaster/pkg/service/sandbox/sandbox_run.go. The schedule() method constructs the selector context and invokes the pipeline:

// CubeMaster/pkg/service/sandbox/sandbox_run.go (excerpt)
func (c *createSandboxContext) schedule() (err error) {
    // Build selector context with the request's resources
    c.selctx = selctx.New(config.GetConfig().Scheduler.LeastSelectName)
    c.selctx.ReqRes = &selctx.RequestResources{
        TemplateID:  c.req.TemplateID,
        CPU:         c.req.Cpu,
        Memory:      c.req.Memory,
        // … other fields …
    }

    // Invoke the scheduler pipeline
    c.selectHost, err = scheduler.Select(c.selctx)
    if err != nil {
        return ret.Err(errorcode.ErrorCode_SelectNodesNoRes, scheduler.ErrNoRes.Error())
    }
    return nil
}

The returned selectHost identifies the target node IP, which the Cubelet agent uses to instantiate the sandbox via RPC.

Extending the Scheduler

The plugin architecture enables customization by implementing the filter.Selector or score.Selector interfaces. New selectors register alongside existing ones in InitScheduler, while scoring weights adjust through YAML configuration. This extensibility allows CubeMaster to adapt to diverse hardware profiles and workload characteristics without core code changes.

Summary

• CubeMaster implements multi-node cluster scheduling via a five-stage plugin pipeline defined in CubeMaster/pkg/scheduler/schedule.go.
• Pre-filter and back-off filter stages handle health checks and degradation scenarios before parallel filtering.
• Filter plugins run concurrently to enforce affinity, resource, and topology constraints.
• Weighted scorers rank viable nodes using real-time metrics, configurable via ScorePluginConf.
• LeastRandomSelect chooses among the top PrioritySelectNum nodes to balance load distribution.
• The entry point in sandbox_run.go line 420 triggers the entire pipeline via scheduler.Select(c.selctx).

Frequently Asked Questions

How does CubeMaster handle scheduling when all nodes are overloaded?

When the pre-filter rejects all nodes, the scheduler activates the back-off filter defined in CubeMaster/pkg/selector/backofffilter/backoff_filter.go. This fallback mechanism relaxes constraints for non-template workloads, allowing critical sandboxes to schedule on nodes that would otherwise be excluded due to temporary resource pressure.

Can operators customize the scoring weights for node selection?

Yes. The scheduler reads scoring weights from config.GetConfig().Scheduler.Score.ScorePluginConf during initialization. Operators can adjust the relative importance of real-time metrics, image caching, and multi-factor scores without recompiling the binary, enabling fine-tuning for specific workload patterns.

What prevents the scheduler from always choosing the same optimal node?

The LeastRandomSelect algorithm in CubeMaster/pkg/scheduler/schedule.go implements a two-tier selection: it first identifies the PrioritySelectNum least-loaded candidates, then randomly selects one from that subset. This randomization prevents hot-spotting and ensures traffic distributes evenly across the cluster over time.

Where does the scheduling context originate during sandbox creation?

The createSandboxContext in CubeMaster/pkg/service/sandbox/sandbox_run.go constructs the SelectorCtx at line 420. It populates the context with the request's CPU, memory, and template ID requirements, then passes it to scheduler.Select() to execute the multi-stage filtering and scoring pipeline.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}