How to Achieve High-Density Deployment with CubeSandbox: Running Thousands of Sandboxes Per Node

You can run thousands of CubeSandbox instances per node by leveraging reflink-based storage pools, aggressive resource quotas, automated snapshot pruning, and kernel-level tuning to minimize overhead per sandbox.

CubeSandbox is TencentCloud's container-like sandbox runtime designed for running isolated workloads at massive scale. When targeting high-density deployment scenarios with thousands of sandboxes per node, you must optimize storage, networking, and resource allocation to prevent node exhaustion. The following practices are derived directly from the TencentCloud/CubeSandbox source code.

Optimize Storage with Reflink Pools

Storage efficiency determines how many sandboxes you can pack onto a single node. Use the reflink-based pool implemented in Cubelet/storage/pool_withreflink.go to create copy-on-write snapshots that share underlying blocks without duplicating data.

Enable reflink support on your host filesystem (XFS or Btrfs) and initialize the pool during Cubelet startup:

// File: Cubelet/storage/pool_withreflink.go
pool, err := cubecow.NewPoolWithReflink("/var/lib/cubesandbox/pool")
if err != nil { 
    log.Fatal(err) 
}

This approach dramatically cuts the amount of data written per sandbox, allowing you to spin up thousands of instances from the same base image without exhausting disk space.

Centralize Image Management

Network I/O becomes a bottleneck when thousands of sandboxes require the same template. Pull templates once per node using the template-center in CubeMaster/pkg/templatecenter/template_image.go, and rely on the image cache (cache.go) to avoid redundant downloads.

When a request arrives, the controller checks the cache before pulling:

// File: CubeMaster/pkg/templatecenter/template_image.go
img, err := tmplCenter.GetOrPullImage(ctx, "nodejs-14")
if err != nil { 
    return err 
}

This reduces startup latency and prevents overwhelming your registry when scaling rapidly.

Automate Snapshot Lifecycle Management

Unbounded snapshot growth consumes memory and inodes, eventually crashing the node. Configure the snapshot reconciler in CubeMaster/pkg/templatecenter/snapshot_reconciler.go to periodically prune expired snapshots.

Set a sensible Time-To-Live (TTL) in CubeMaster/pkg/templatecenter/snapshot_metrics.go and run the cleanup loop every few minutes:

// File: CubeMaster/pkg/templatecenter/snapshot_reconciler.go
go func() {
    ticker := time.NewTicker(10 * time.Minute)
    for range ticker.C {
        _ = reconciler.PruneExpiredSnapshots(ctx)
    }
}()

This prevents metadata bloat and ensures long-running nodes remain stable.

Enforce Strict Resource Limits

Without limits, a single misbehaving sandbox can starve thousands of others. Configure conservative cgroups through the SDK by setting cpuCount and memoryMB in sdk/go/models.go.

A typical high-density configuration allocates fractional resources:

// File: sdk/go/sandbox.go
sb, err := client.NewSandbox(ctx, cubesandbox.SandboxOptions{
    TemplateID:  "nodejs-14",
    CPUCount:    0.1,           // 10% of a core
    MemoryMB:    64,            // 64 MiB
    IdleTimeout: 300 * time.Second,
})

These limits are enforced by the Cubelet runtime, guaranteeing node stability under heavy load.

Configure Timeouts and Connection Pooling

Idle sandboxes waste resources that active workloads need. Set a short idle timeout (e.g., 300 seconds) via sandbox.SetTimeout in sdk/go/sandbox.go to automatically terminate inactive instances.

For control plane efficiency, reuse HTTP connections across SDK operations. The Go SDK handles this in sdk/go/transport.go through persistent keep-alive and HTTP/2 support:

// File: sdk/go/transport.go
// The SDK reuses the HTTP client across operations to reduce 
// per-request overhead and TLS handshake latency when managing 
// thousands of sandbox endpoints.

This reduces connection churn when the control plane communicates with thousands of sandbox endpoints simultaneously.

Tune the Host Environment and Filesystem

Kernel defaults are too low for thousands of concurrent sandboxes. Boot the node with Linux kernel ≥ 5.10 and enable CONFIG_FS_POSIX_ACL. Tune system limits to prevent "too many files" errors:

• Increase fs.inotify.max_user_watches
• Raise fs.file-max
• Set ulimit -n to at least 200,000

Mount a dedicated tmpfs for each sandbox's writable layer using the logic in Cubelet/storage/hostdir.go. This keeps I/O fast and avoids disk hotspots for short-lived processes:

// File: Cubelet/storage/hostdir.go
// Mounts tmpfs for the sandbox's writable layer to provide 
// low-latency writes without backing disk contention.

Implement Health Monitoring and Auto-Recovery

Individual sandboxes will crash under memory pressure or application errors. The sandbox health monitor in Cubelet/services/server/server.go detects failed instances and automatically recreates them from the template pool.

This guarantees high availability without manual intervention during mass events like OOM kills or panic storms.

Export Metrics for Capacity Planning

Visibility prevents saturation. Expose per-sandbox metrics via the Prometheus exporter in CubeMaster/pkg/templatecenter/snapshot_metrics.go:

// File: CubeMaster/pkg/templatecenter/snapshot_metrics.go
prometheus.MustRegister(snapshotMetrics.ActiveSnapshots)
prometheus.MustRegister(snapshotMetrics.TotalBytes)

Aggregate these metrics at the node level to trigger horizontal scaling before resource exhaustion occurs.

Summary

• Use reflink pools (pool_withreflink.go) on XFS/Btrfs to eliminate duplicate storage across sandboxes.
• Cache templates in the template-center (template_image.go) to avoid redundant network pulls.
• Prune snapshots automatically via the reconciler (snapshot_reconciler.go) to control metadata growth.
• Set tight resource limits (0.1 CPU, 64 MiB) through the SDK to maximize node utilization safely.
• Configure short idle timeouts (sandbox.go) and reuse HTTP connections (transport.go) to reduce overhead.
• Tune kernel limits and use tmpfs (hostdir.go) for writable layers to handle thousands of open files.
• Enable health monitoring (server.go) for automatic recovery from crashed sandboxes.

Frequently Asked Questions

What is the maximum number of sandboxes per node in CubeSandbox?

There is no hardcoded limit in the CubeSandbox source code, but practical density depends on available CPU, memory, and file descriptor limits. Production deployments regularly achieve thousands of sandboxes per node when using reflink storage, conservative resource quotas (0.1 CPU / 64 MiB per sandbox), and proper kernel tuning (ulimit -n ≥ 200,000).

How does CubeSandbox minimize disk usage for thousands of sandboxes?

CubeSandbox uses reflink-based copy-on-write snapshots via Cubelet/storage/pool_withreflink.go. When you create a sandbox from a template, the filesystem creates a lightweight reference to existing blocks rather than copying the entire image. This allows thousands of sandboxes to share the same base data while writing only their unique changes, drastically reducing per-sandbox disk overhead.

What happens when a CubeSandbox instance crashes?

The sandbox health monitor in Cubelet/services/server/server.go continuously checks sandbox status. If a process exits unexpectedly or is killed by the OOM killer, the monitor automatically recreates the sandbox from the cached template pool. This ensures high availability without requiring manual intervention when individual sandboxes fail under high-density load.

Which filesystem should I use for high-density CubeSandbox deployment?

Use XFS or Btrfs with reflink support enabled. The pool_withreflink.go implementation relies on filesystem-level copy-on-write capabilities to create efficient snapshots. Ensure your kernel is version 5.10 or later with CONFIG_FS_POSIX_ACL enabled, and mount the storage pool with appropriate mount options to enable reflink functionality.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}