How Auto-Pause and Auto-Resume Work in CubeSandbox: Architecture and Implementation

CubeSandbox automatically pauses idle sandboxes to conserve resources and resumes them on demand through a coordinated system involving a Sweeper component for idle detection and a CubeMaster component for handling resurrection requests.

CubeSandbox, developed by TencentCloud, implements an intelligent resource management system that automatically pauses inactive sandboxes and restores them when needed. This auto-pause and auto-resume functionality ensures optimal resource utilization while maintaining seamless access for end users. The implementation relies on two primary components working in concert: the Sweeper, which monitors activity and initiates pauses, and the CubeMaster, which handles on-demand resumes.

The Sweeper: Auto-Pause Implementation

Located in cube-lifecycle-manager/internal/sweeper/sweeper.go, the Sweeper component runs a continuous loop that monitors sandbox activity and triggers pauses when idle thresholds are exceeded.

Idle Detection Logic

The Sweeper operates on a ticker interval, invoking sweepOnce on each iteration. Before evaluating idle states, the system implements a bootstrap-warmup guard (lines 98-106) that prevents newly observed entries during startup from being paused before the first last_active poll can populate activity timestamps.

For each sandbox, the Sweeper calculates a baseline timestamp as the most recent of LastActiveMs and CreatedAt (lines 119-126). The idle duration is computed as nowMs - baseline. If this duration exceeds the sandbox's configured TimeoutSeconds (or the default idle timeout) and AutoPause is enabled, the system proceeds to pause the sandbox.

The Pause Execution Flow

When the idle threshold is exceeded, the Sweeper invokes tryPause. This function acquires a distributed lock, calls the CubeMaster RPC to pause the sandbox, and pushes a state-change notification via ProxyPush. The implementation (lines 94-124) includes error handling metrics:

if e.Meta.AutoPause {
    s.o.Log.Info("idle threshold exceeded; pausing", …)
    if err := s.tryPause(ctx, e); err != nil {
        s.pauseFailed.Add(1)
        s.o.Log.Warn("auto‑pause failed", …)
    }
}

The Sweeper intentionally skips sandboxes in terminal states including paused, pausing, killing, and killed to avoid redundant RPC traffic (lines 156-165).

CubeMaster: Auto-Resume Implementation

The CubeMaster component handles the resurrection of paused sandboxes when traffic arrives.

Resume Triggers and Metadata Channels

The sidecar registers an auto-pause metadata channel in CubeMaster/pkg/lifecycle/init.go (lines 22-49) that receives resume events from the proxy. When an HTTP request arrives for a paused sandbox, the HTTP server entry point in CubeMaster/cmd/cubemaster/app/main.go calls the ResumeSandbox RPC.

This RPC clears the pause flag, updates the state in Redis, and notifies the proxy so traffic can flow again. The resume path is guarded by the same distributed lock used for pausing, ensuring only one sidecar performs the transition.

Activity Tracking and Proxy Integration

The proxy sidecar reports activity timestamps (last_active) to a Redis stream that the Sweeper reads, enabling the idle-threshold calculation. This connection is defined in CubeMaster/pkg/service/sandbox/types/types.go (line 73). When the proxy sees a resumed sandbox, it clears its local pause flag.

Implementation Examples

Configuring Auto-Pause via Go SDK

import "github.com/tencentcloud/CubeSandbox/sdk/go"

func createSandbox() {
    client := sdk.NewClient(...)
    // Enable auto-pause and set a 5-minute idle timeout
    sandbox := &sdk.Sandbox{
        Name:          "demo",
        AutoPause:     true,
        TimeoutSeconds: sdk.IntPtr(300), // 5 minutes
    }
    _, _ = client.CreateSandbox(context.Background(), sandbox)
}

Resuming a Paused Sandbox

func resumeSandbox(sandboxID string) error {
    client := sdk.NewClient(...)
    // This RPC un-pauses the sandbox; it will be called automatically
    // when a normal request reaches the HTTP server.
    return client.ResumeSandbox(context.Background(), sandboxID)
}

Testing Auto-Pause Behavior

func TestAutoPause(t *testing.T) {
    // Set a very short idle timeout for the test
    opts := sweeper.Options{
        Registry:           fakeRegistry,
        Redis:              fakeRedis,
        CubeMaster:         fakeCubeMaster,
        DefaultIdleTimeout: 2 * time.Second,
        Interval:           1 * time.Second,
        Log:                zap.NewExample(),
    }
    s := sweeper.New(opts)
    go s.Run(context.Background())
    // ...simulate no traffic, then verify s.pauseTriggered incremented
}

Key Source Files

• cube-lifecycle-manager/internal/sweeper/sweeper.go: Implements the periodic auto-pause sweep loop, including the sweepOnce and tryPause functions.
• CubeMaster/pkg/lifecycle/init.go: Configures the auto-pause metadata channel for resume notifications (lines 22-49).
• CubeMaster/cmd/cubemaster/app/main.go: HTTP entry point that triggers sandbox resume on demand.
• CubeMaster/pkg/service/sandbox/types/types.go: Connects the proxy sidecar to the auto-pause workflow (line 73).
• sdk/go/client.go: Exposes the ResumeSandbox method for application integration.

Summary

• The Sweeper component in cube-lifecycle-manager periodically scans for idle sandboxes and executes pauses via tryPause when idle thresholds are exceeded.
• Bootstrap-warmup guards prevent premature pausing of newly observed sandboxes during system startup.
• The CubeMaster handles auto-resume through HTTP endpoints that trigger the ResumeSandbox RPC, clearing pause states in Redis and notifying proxies.
• Distributed locks ensure that pause and resume operations are atomic and race-condition free.
• The proxy sidecar maintains activity timestamps in Redis, enabling accurate idle detection by the Sweeper.

Frequently Asked Questions

How does CubeSandbox determine when to auto-pause a sandbox?

The Sweeper calculates the idle duration by comparing the current time against the most recent of LastActiveMs and CreatedAt timestamps (lines 119-126 in sweeper.go). If this duration exceeds the configured TimeoutSeconds and AutoPause is enabled, the sandbox becomes eligible for pausing.

What prevents a sandbox from being paused immediately after creation?

A bootstrap-warmup guard (lines 98-106) ensures that newly observed entries during Sweeper startup are not evaluated for pausing until the first last_active poll can populate their activity timestamps. This prevents premature pausing of brand new sandboxes.

How does the auto-resume mechanism know when to wake a paused sandbox?

The CubeMaster HTTP server listens for incoming requests targeting paused sandboxes. When a request arrives at the entry point in CubeMaster/cmd/cubemaster/app/main.go, it invokes the ResumeSandbox RPC, which clears the pause flag, updates Redis, and notifies the proxy to resume traffic flow.

Can sandbox states other than "running" be auto-paused?

No. The Sweeper explicitly skips sandboxes in terminal states including paused, pausing, killing, and killed (lines 156-165). This optimization avoids unnecessary RPC calls to sandboxes that are already inactive or being terminated.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}