How AutoResume Works for Idle Sandbox Optimization in CubeSandbox

AutoResume wakes paused sandboxes on-demand by intercepting incoming requests in CubeProxy, acquiring a distributed Redis lock to serialize resume operations, and restoring the VM via CubeMaster RPC while coalescing concurrent requests to prevent duplicate work.

The CubeSandbox platform from TencentCloud implements an intelligent resource management system that automatically pauses idle sandboxes and rapidly resumes them when new traffic arrives. Understanding how AutoResume works for idle sandbox optimization reveals a sophisticated distributed coordination mechanism that balances millisecond-scale wake-up times with strict consistency guarantees across multiple side-car replicas.

The AutoResume Trigger Flow

When a sandbox exceeds its timeout_seconds threshold, the Sweeper component marks it as paused in Redis by setting cube:v1:shared:sandbox:lifecycle:state:<sandbox-id> to "paused". At this point, the side-car proxy (CubeProxy) stops forwarding traffic. The resume sequence initiates when a new dataplane request targets a paused sandbox.

CubeProxy Interception and Internal Routing

The request first hits the Lua gate (sandbox_state.lua) within CubeProxy. Instead of forwarding to the paused sandbox, the gate returns HTTP 503 and simultaneously dispatches an internal request to POST /internal/resume to trigger the wake-up sequence. This logic is implemented in the router at network-agent/internal/service/cube_router.go.

Side-car HTTP Handler

The side-car's HTTP server receives the resume request at cube-lifecycle-manager/internal/httpapi/server.go. The handler extracts the sandbox ID and delegates to the Resumer package:

func (s *Server) resumeHandler(w http.ResponseWriter, r *http.Request) {
    sandboxID := chi.URLParam(r, "sandbox_id")
    err := s.resumer.Resume(r.Context(), sandboxID)
    if err != nil {
        http.Error(w, err.Error(), http.StatusServiceUnavailable)
        return
    }
    w.WriteHeader(http.StatusOK)
}

Distributed State Coordination

The Resumer must coordinate across multiple side-car replicas to ensure only one attempts the expensive VM restoration. This uses a Redis-backed state machine with four possible values: "paused", "running", "pausing", and "resuming".

The Redis Lock Mechanism

In cube-lifecycle-manager/internal/resumer/resumer.go, the acquireResumeOwnership function implements a GET-then-SETNX pattern:

func (r *Resumer) acquireResumeOwnership(ctx context.Context, sandboxID string) error {
    cur, ok, err := r.o.Redis.GetState(ctx, sandboxID)
    if err != nil { return err }

    switch {
    case !ok || cur == "paused":
        return r.o.Redis.SetState(ctx, sandboxID, "resuming", r.o.StateLockTTL)
    case cur == "running":
        return errAlreadyRunning
    case cur == "pausing" || cur == "resuming":
        if err := r.waitForRunning(ctx, sandboxID); err != nil { return err }
        return errAlreadyRunning
    default:
        if err := r.waitForRunning(ctx, sandboxID); err != nil { return err }
        return errAlreadyRunning
    }
}

Key behaviors:

• If the state is "paused", the caller sets it to "resuming" and owns the resume operation.
• If "running", the operation short-circuits as already complete.
• If "pausing" or "resuming", the caller waits for the peer to finish using waitForRunning.

Concurrent Request Handling

Each side-car replica handles request coalescing internally to prevent duplicate RPCs when multiple simultaneous requests arrive for the same sandbox.

Single-Flight Implementation

The Resumer maintains an in-process map[string]*call (lines 40-44 in resumer.go) that merges concurrent Go goroutines:

func (r *Resumer) Resume(ctx context.Context, sandboxID string) error {
    r.mu.Lock()
    if c, ok := r.calls[sandboxID]; ok {
        r.mu.Unlock()
        select {
        case <-c.done: return c.err
        case <-ctx.Done(): return ctx.Err()
        }
    }
    c := &call{done: make(chan struct{})}
    r.calls[sandboxID] = c
    r.mu.Unlock()
    defer func() {
        r.mu.Lock(); delete(r.calls, sandboxID); r.mu.Unlock()
        close(c.done)
    }()
    c.err = r.doResume(ctx, sandboxID)
    return c.err
}

This ensures that even under high concurrency, only one resume RPC executes per replica, while others block until completion.

VM Restoration and State Propagation

Once the Resumer acquires ownership, it invokes CubeMaster.Resume via the RPC client in cube-lifecycle-manager/internal/cubemasterclient/client.go. This triggers the VM-restore path through CubeMaster → Cubelet → containerd, reconstructing the sandbox from its pause snapshot.

Success Bookkeeping

After successful restoration (or detecting an already-running state), the Resumer performs three critical updates:

1. Redis State: Writes "running" back to the lifecycle state key with SetState
2. Proxy Synchronization: Pushes the new state to all CubeProxy replicas via ProxyPush.SetState
3. Activity Tracking: Updates registry.LastActiveMs to prevent immediate re-pausing by the Sweeper

func (r *Resumer) doResume(ctx context.Context, sandboxID string) error {
    // ... after successful RPC or already-running case ...
    _ = r.o.Redis.SetState(ctx, sandboxID, "running", r.o.StateLockTTL)
    _ = r.o.ProxyPush.SetState(ctx, sandboxID, "running")
    r.o.Registry.MergeLastActive(sandboxID, time.Now().UnixMilli())
    r.o.Log.Info("auto-resumed sandbox", zap.String("sandbox_id", sandboxID))
    return nil
}

AutoResume Configuration and Safety

The AutoResume feature is controlled per-sandbox via the AutoResume flag in SandboxLifecycleMeta. If disabled, the Resumer returns auto_resume not enabled for sandbox (lines 99-101 in resumer.go) and the request fails fast.

The system handles edge cases gracefully:

• Lock expiration: If the Redis lock expires without reaching a terminal state, waitForRunning returns an error, the caller returns 503, and the next request retries the sequence.
• Idempotent RPCs: The callCubeMasterResume function treats "already-in-state" responses as success, making retries safe.

Summary

• AutoResume triggers when CubeProxy intercepts traffic to a paused sandbox and invokes the side-car's /internal/resume endpoint.
• Distributed coordination uses a Redis state machine (paused, pausing, resuming, running) with atomic GET-SETNX operations to serialize resume attempts across replicas.
• Request coalescing via an in-process calls map ensures single-flight execution within each side-car instance, preventing duplicate CubeMaster RPCs.
• State propagation updates Redis, pushes changes to all CubeProxy instances, and refreshes the activity timestamp to prevent immediate re-pausing.
• Safety mechanisms include per-sandbox feature flags, idempotent RPC handling, and graceful 503 fallbacks when locks expire.

Frequently Asked Questions

How does AutoResume prevent multiple side-car replicas from duplicating resume work?

The system uses Redis as a distributed lock with a state machine. When a replica calls acquireResumeOwnership, it attempts to atomically transition the state from "paused" to "resuming". Only the replica that succeeds owns the resume operation. Others wait via waitForRunning or short-circuit if the state is already "running". This is implemented in cube-lifecycle-manager/internal/resumer/resumer.go.

What happens if a resume request fails or the Redis lock expires?

If the lock expires before completion, waitForRunning returns an error, and the Resumer returns HTTP 503 to the caller. The original dataplane request (blocked in Nginx with proxy_read_timeout) may retry, or the next incoming request will restart the resume sequence. This makes the system self-healing without manual intervention.

Where is the AutoResume feature flag configured?

The AutoResume boolean resides in SandboxLifecycleMeta (defined in cube-lifecycle-manager/internal/lifecycle/schema.go). When the Resumer receives a resume request, it checks entry.Meta.AutoResume at lines 99-101 of resumer.go. If false, it immediately returns an error, preventing automatic wake-up for sandboxes that require manual management.

How does the system handle concurrent requests to the same paused sandbox?

Within each side-car replica, the Resumer maintains a calls map that implements a single-flight pattern. The first goroutine for a sandbox ID executes the actual resume logic, while subsequent concurrent callers block on a channel until the operation completes. This prevents duplicate RPCs to CubeMaster while allowing distributed coordination via Redis across different replicas.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}