How the CubeSandbox Lifecycle Manager Coordinates Auto‑Pause and Auto‑Resume Across Nodes

The CubeSandbox Lifecycle Manager coordinates auto‑pause and auto‑resume across nodes by publishing state‑change commands to a Redis‑based event stream, ensuring every node in the cluster maintains a consistent view of sandbox states and can act on behalf of the entire system.

The TencentCloud CubeSandbox project provides a distributed sandbox environment where workloads may run on any node in a cluster. To optimize resource utilization, the Lifecycle Manager automatically pauses idle sandboxes and resumes them when activity returns, requiring seamless coordination across all nodes to maintain consistent state and prevent resource leaks.

The Core Coordination Architecture

The Lifecycle Manager relies on four tightly‑coupled components to maintain cluster‑wide consistency. Each component is implemented as a distinct internal package within the cube-lifecycle-manager directory.

Sweeper

Located in cube-lifecycle-manager/internal/sweeper/sweeper.go, the Sweeper periodically scans sandbox metadata to detect idle sandboxes. It enumerates sandboxes stored in the registry (registry/registry.go) and compares the last‑activity timestamp against the configured AutoPauseTimeout. When a sandbox exceeds the idle threshold and lifecycle.autoPause is enabled, the Sweeper builds a pause request and publishes it to the Redis stream.

Resumer

Implemented in cube-lifecycle-manager/internal/resumer/resumer.go, the Resumer monitors inbound connections via the HTTP API server (httpapi/server.go). When it detects traffic targeting a paused sandbox that has autoResume: true configured, it generates a resume request and publishes it to the distributed event bus.

Redis Stream and Discovery Layer

The cube-lifecycle-manager/internal/redisstream/stream.go file implements the event bus that all nodes subscribe to. Additionally, cube-lifecycle-manager/internal/discovery/redis.go handles node registration and health tracking. This combined layer ensures that pause and resume commands are reliably delivered to every healthy node in the cluster, regardless of which node originated the request.

ProxyPush Client

Found in cube-lifecycle-manager/internal/proxypush/client.go, this component executes the actual state transition by sending pause and resume RPCs to the sandbox process via the CubeMaster API. It acts as the final execution layer after a node consumes a command from the Redis stream.

Auto‑Pause Workflow

The auto‑pause mechanism follows a strict publish‑subscribe pattern to ensure fault tolerance:

1. Discovery – Each node registers itself in Redis via discovery/redis.go, and the list of healthy nodes is constantly refreshed.

2. Sweeper Tick – Every config.SweeperInterval seconds, the Sweeper enumerates sandboxes from the registry.

3. Idle Detection – For each sandbox, the Sweeper checks the last‑activity timestamp. If the sandbox has been idle longer than config.AutoPauseTimeout, the Sweeper builds a pause request containing the sandbox ID, action type (pause), and source node ID.

4. Publish – The request is serialized and published to the dedicated Redis stream (redisstream/stream.go).

5. Consume – All nodes consume the same stream. When a node sees a pause message for a sandbox it owns, it forwards the request via the ProxyPush client to the local sandbox process, which transitions to the paused state.

Because the event persists in Redis, even if the node that originally decided to pause crashes, any other node will still process the message and enforce the pause, guaranteeing cluster‑wide consistency.

Auto‑Resume Workflow

The auto‑resume mechanism ensures that any node can trigger a resume without requiring centralized coordination:

1. Connection Detection – The Resumer watches inbound traffic through the HTTP API server. When a client opens a websocket, TCP connection, or API call targeting a sandbox, the interceptor captures the event.

2. State Check – If the target sandbox is currently in the paused state (recorded in the registry) and its lifecycle configuration has autoResume: true, the Resumer creates a resume request.

3. Publish – The resume request is sent through the same Redis stream, labeled with action resume.

4. Consume – Nodes that own the paused sandbox receive the resume message, invoke the ProxyPush client (client.go), and issue a resume RPC to the sandbox process. The sandbox restores its runtime, re‑establishes network endpoints, and updates its last‑activity timestamp.

By using a single stream for both pause and resume actions, the system guarantees that any node can trigger a resume, while the pause decision remains centrally coordinated by the Sweeper.

Fault Tolerance and Edge Cases

The architecture handles several failure modes without manual intervention:

• Node failure after publishing – Messages remain in Redis until all consumers acknowledge them. A restarted node will re‑process pending pause commands, ensuring the sandbox ends up paused regardless of the originator’s availability.

• Concurrent pause and resume requests – Messages are processed in FIFO order. If a resume arrives before the pause has been applied, the Resumer notices the sandbox is still running and ignores the redundant resume. The ProxyPush client also checks current state, making duplicate pause RPCs no‑ops.

• Network partition – Nodes that lose connectivity stop consuming the stream. Once connectivity restores, they replay missed messages from the Redis backlog and bring their local sandbox state back in sync with the cluster.

Configuration

Auto‑pause and auto‑resume are configured via the lifecycle object when creating a sandbox. The following Python SDK example demonstrates the available options:

sandbox = client.sandbox.create(
    name="my-sandbox",
    lifecycle={
        "auto_pause": True,          # pause after configured idle timeout

        "auto_resume": True,         # auto-resume when a client connects

        "on_timeout": "pause",       # pause when an operation times out

    },
)

The same fields exist in the Node SDK (sdk/node/src/sandbox.ts) and the Go SDK (sdk/go/sandbox.go), ensuring consistent behavior across all client implementations.

Summary

• The Lifecycle Manager uses a Redis stream (redisstream/stream.go) as a distributed event bus to synchronize pause and resume commands across all nodes.
• The Sweeper (sweeper/sweeper.go) centrally detects idle sandboxes and publishes pause events, while the Resumer (resumer/resumer.go) listens for inbound connections to trigger resumes.
• The ProxyPush client (proxypush/client.go) executes the actual RPC calls to sandbox processes, ensuring commands are applied only on the node hosting the target sandbox.
• Fault tolerance is achieved through Redis persistence, FIFO message ordering, and automatic replay after network partitions, guaranteeing that state changes are eventually applied cluster‑wide.

Frequently Asked Questions

How does the Lifecycle Manager ensure a pause command is executed even if the originating node crashes?

The pause command is published to a Redis stream that persists until acknowledged by all consumers. If the node that detected the idle sandbox crashes immediately after publishing, the message remains in the stream. When any node—including one that just restarted—consumes the stream, it sees the pending pause command and executes it via the ProxyPush client, ensuring the sandbox transitions to the paused state regardless of the original node’s availability.

Can any node in the cluster trigger an auto‑resume, or must the request go through a central coordinator?

Any node can trigger an auto‑resume. When the Resumer component on any node detects inbound traffic targeting a paused sandbox, it publishes a resume message to the shared Redis stream. The node that actually owns the sandbox consumes this message and executes the resume RPC. This design allows for decentralized resume triggers while maintaining centralized idle detection for pauses.

What happens if a sandbox receives duplicate pause or resume commands due to network retries?

The system is idempotent. The ProxyPush client checks the current state of the sandbox before sending the RPC. If a sandbox is already paused and receives a second pause command, the client recognizes the state match and treats the duplicate as a no‑op. Similarly, resume requests for running sandboxes are ignored. This prevents state thrashing during network retries or delayed message delivery.

How does the system handle clock skew when determining if a sandbox has been idle long enough to auto‑pause?

The Sweeper relies on the last‑activity timestamp stored in the centralized registry (registry/registry.go), which is updated consistently across nodes. Because the timeout comparison uses this shared registry value rather than local node clocks, clock skew between individual nodes does not affect the idle detection logic. The Sweeper evaluates the difference between the current time and the stored timestamp using the node’s local clock, but since the registry provides the single source of truth for activity, the cluster maintains a consistent view of idle duration.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}