BatchSandbox Runtime Architecture in Kubernetes: A Deep Dive into Alibaba OpenSandbox

The BatchSandbox runtime implements a three-layer Kubernetes-native architecture comprising a CRD for declarative state, a controller for reconciliation and lifecycle management, and a provider SDK for manifest generation, supporting both direct pod creation and pool-based warm allocation modes.

The BatchSandbox runtime is a core component of Alibaba's OpenSandbox project that enables secure, scalable sandbox execution within Kubernetes clusters. This architectural design centers on a custom resource definition (CRD) that bridges high-level user requests with low-level pod orchestration through a specialized controller and provider layer. Understanding this architecture is essential for operators deploying OpenSandbox in production Kubernetes environments.

Three-Layer Architecture Overview

The BatchSandbox runtime architecture is divided into three tightly coupled layers that handle distinct responsibilities within the Kubernetes control plane.

CRD and API Layer

The foundation resides in kubernetes/apis/sandbox/v1alpha1/batchsandbox_types.go, which defines the BatchSandbox custom resource. This layer declares the desired state through BatchSandboxSpec (lines 23-71) and reports observed state through BatchSandboxStatus (lines 82-103). The spec captures critical parameters including replicas, poolRef for warm pool references, template for PodTemplateSpec injection, expireTime for automatic deletion deadlines, and taskTemplate for post-startup task scheduling.

Controller Layer

The reconciliation logic lives in kubernetes/internal/controller/batchsandbox_controller.go. The BatchSandboxReconciler (specifically the Reconcile method at lines 82-128) watches BatchSandbox objects and manages the full lifecycle: creating and updating underlying Pods, handling expiration logic, managing pool allocations, and driving the task-scheduling subsystem. This layer ensures optimistic concurrency through client.RawPatch and Status().Update operations (see updateStatus at lines 401-409).

Provider Layer

The server-side SDK resides in server/src/services/k8s/batchsandbox_provider.py. The BatchSandboxProvider class offers a high-level Python API that translates user requests into valid BatchSandbox CRs. It constructs the pod specification including init containers for the execd installer, wraps user entrypoints with bootstrap.sh, injects optional egress sidecars, and merges user-supplied templates via BatchSandboxTemplateManager (lines 24-31 in batchsandbox_template.py).

CRD Specification and Status Contract

The BatchSandboxSpec struct defines the declarative contract between users and the OpenSandbox system. Key fields include:

• replicas: The desired number of sandbox pods to maintain
• poolRef: Optional reference to a pre-warmed Pool resource; when present, the controller operates in pool mode
• template: A standard corev1.PodTemplateSpec that the controller expands into real Pods with volumes and container specifications
• expireTime: Absolute deletion deadline for automatic cleanup
• taskTemplate: A Task spec automatically scheduled after pod creation, processed by the controller's task-scheduler

The corresponding BatchSandboxStatus (lines 82-103) tracks the observed replica count, allocation and ready counts, and per-task counters including TaskRunning, TaskSucceed, and failure states. This status sub-resource enables the controller to report accurate state without modifying the spec.

Provider Manifest Generation Process

When the OpenSandbox service initiates a sandbox, it invokes BatchSandboxProvider.create_workload(...) (lines 13-62). The provider executes a deterministic build sequence:

1. RuntimeClass Selection: If secure runtime isolation is configured, the provider invokes SecureRuntimeResolver to select an appropriate RuntimeClass (lines 84-88) and injects runtimeClassName into the pod spec (lines 107-110).

2. Template Loading: The provider loads user-supplied templates via BatchSandboxTemplateManager (lines 24-31), extracting read-only volumes and volumeMounts through _extract_template_pod_extras (lines 176-190).

3. Init Container Construction: The _build_execd_init_container method (lines 50-84) creates an init container that copies the execd binary and bootstrap.sh into a shared emptyDir volume named opensandbox-bin.

4. Main Container Wrapping: The _build_main_container method (lines 96-133) wraps the user entrypoint with bootstrap.sh inside the main container, ensuring the execd agent is available before user processes start.

5. Network Policy Application: When egress control is required, apply_egress_to_spec (lines 112-117) injects an egress sidecar container routing outbound traffic through a configurable image. Concurrently, build_security_context_for_sandbox_container sets runAsUser, allowPrivilegeEscalation, and other security contexts (lines 136-144).

6. CR Creation: Finally, the provider calls the Kubernetes CustomObjects API to create the BatchSandbox resource, merging runtime-generated fields with user templates via self.template_manager.merge_with_runtime_values.

Controller Reconciliation Logic

The BatchSandboxReconciler processes every BatchSandbox change through its Reconcile method (lines 82-128), implementing the following operational steps:

• Expiration Handling: Lines 100-115 check expireTime and delete the CR when the deadline passes.

• Pool Detection: Line 122 constructs a PoolStrategy to determine whether the CR operates in pooled mode based on spec.poolRef presence.

• Pod Discovery: The listPods method (lines 174-210) returns either Pods owned by the CR in direct mode, or the allocated pod set recorded in annotations (sandbox.opensandbox.io/alloc) when in pool mode.

• Scaling Operations: scaleBatchSandbox (lines 260-322) creates missing Pods according to spec.replicas, applying per-shard patches from spec.shardPatches. It generates Pods from templates using utils.GetPodFromTemplate and establishes owner references via metav1.NewControllerRef.

• Status Aggregation: Lines 131-165 compute replicas, allocated, and ready counts, updating the status sub-resource to reflect current cluster state.

• Task Scheduling: When taskTemplate is present, the controller instantiates a TaskScheduler (lines 188-207) and invokes scheduleTasks (lines 203-229) to create Task objects and update status.task* counters.

• Finalizer Management: Lines 226-258 ensure task resources release before final CR deletion, parsing allocation and release annotations via parseSandboxAllocation and parseSandboxReleased (lines 520-560).

Deployment Modes: Direct vs Pool

The BatchSandbox runtime supports two distinct operational modes:

Direct mode (default) requires the provider to supply a complete pod specification including image, resources, and environment variables. The controller creates one pod per replica and tracks them individually through standard Kubernetes owner references.

Pool mode activates when spec.poolRef points to a pre-created Pool object containing warm pods. In this mode, the provider only specifies taskTemplate (entrypoint and environment), while the controller allocates existing pods from the warm pool. Allocation and release states persist in CR annotations (sandbox.opensandbox.io/alloc and sandbox.opensandbox.io/release), with the controller reconciling only the active subset of pods.

Security and Networking Implementation

The architecture implements defense in depth through multiple isolation mechanisms:

• Egress Sidecar: The apply_egress_to_spec function injects a dedicated sidecar container that routes outbound traffic through a configurable egress_image, enabling fine-grained network policy enforcement.

• Secure RuntimeClass: Via SecureRuntimeResolver, the provider detects secure runtime configurations and injects runtimeClassName (e.g., Kata Containers) into the pod spec, providing kernel-level isolation.

• Security Contexts: When network policies are attached, the provider invokes build_security_context_for_sandbox_container to configure runAsUser, disable privilege escalation, and apply additional hardening parameters.

End-to-End Execution Flow

A complete sandbox lifecycle follows this sequence:

1. The OpenSandbox service receives an API call and invokes BatchSandboxProvider.create_workload.
2. The provider constructs a BatchSandbox CR with init containers, main containers, optional egress sidecars, and RuntimeClass specifications.
3. The controller detects the new CR and either creates required Pods or selects them from a warm pool.
4. Pods initialize, executing the init container to install execd into opensandbox-bin, then launch bootstrap.sh to start the user process.
5. If a TaskTemplate is defined, the controller's task-scheduler creates Task objects, monitors their lifecycle, and updates status counters.
6. Upon expireTime arrival or explicit deletion requests, the controller deletes Pods, cleans up pool allocation annotations, and removes the CR.

Key Implementation Files

The following source files constitute the complete BatchSandbox runtime implementation:

• kubernetes/apis/sandbox/v1alpha1/batchsandbox_types.go — CRD definition for spec and status schemas
• kubernetes/internal/controller/batchsandbox_controller.go — Core reconciliation, scaling, and lifecycle logic
• server/src/services/k8s/batchsandbox_provider.py — High-level Python API for CR generation
• server/src/services/k8s/batchsandbox_template.py — Template loading and merging utilities
• kubernetes/internal/controller/strategy/pool_strategy.go — Pool allocation policies
• kubernetes/internal/controller/strategy/task_scheduling_strategy.go — Task scheduling implementation
• kubernetes/pkg/client/listers/sandbox/v1alpha1/batchsandbox.go — Generated client listers for cache access

Code Examples

Python: Creating a Sandbox in Direct Mode

from src.services.k8s.batchsandbox_provider import BatchSandboxProvider
from src.api.schema import ImageSpec
from datetime import datetime, timedelta

provider = BatchSandboxProvider(k8s_client, template_file_path="batch.yaml")

provider.create_workload(
    sandbox_id="demo-sbx",
    namespace="default",
    image_spec=ImageSpec(uri="python:3.11-slim"),
    entrypoint=["python", "app.py"],
    env={"ENV": "prod"},
    resource_limits={"cpu": "2", "memory": "4Gi"},
    labels={"app": "demo"},
    expires_at=datetime.utcnow() + timedelta(hours=6),
    execd_image="alibaba/execd:latest",
)

Relevant source: BatchSandboxProvider.create_workload — lines 13-62.

YAML: BatchSandbox CR in Pool Mode

apiVersion: sandbox.opensandbox.io/v1alpha1
kind: BatchSandbox
metadata:
  name: pooled-sbx
  namespace: default
spec:
  replicas: 1
  poolRef: demo-pool
  expireTime: "2026-04-01T00:00:00Z"
  taskTemplate:
    spec:
      process:
        command: ["/bin/sh", "-c", "/opt/opensandbox/bin/bootstrap.sh python app.py"]
        env:
        - name: ENV
          value: prod

CRD schema reference: BatchSandboxSpec — lines 23-71.

Go: Pod Scaling Logic

// Inside scaleBatchSandbox (lines 260-322)
for i := 0; i < int(*batchSandbox.Spec.Replicas); i++ {
    if _, ok := indexedPodMap[i]; !ok {
        needCreateIndex = append(needCreateIndex, i)
    }
}
for _, idx := range needCreateIndex {
    pod, err := utils.GetPodFromTemplate(
        podTemplateSpec, 
        batchSandbox,
        metav1.NewControllerRef(batchSandbox, sandboxv1alpha1.SchemeBuilder.GroupVersion.WithKind("BatchSandbox")),
    )
    if err != nil {
        return err
    }
    r.Create(ctx, pod)
}

Relevant source: scaleBatchSandbox — lines 260-322.

Summary

• The BatchSandbox runtime implements a three-layer architecture separating concerns between API definition (CRD), operational logic (Controller), and user interface (Provider).
• The controller manages complex lifecycle operations including expiration, pool-based allocation, task scheduling, and optimistic concurrency control.
• Two deployment modes exist: Direct mode for full pod specification control, and Pool mode for warm-start optimization using pre-allocated pods.
• Security implementations include RuntimeClass injection for kernel isolation, egress sidecars for network policy enforcement, and automated security context configuration.
• All operations are traceable through specific source files in the alibaba/OpenSandbox repository, with reconciliation logic concentrated in the Go controller and manifest generation handled by the Python provider.

Frequently Asked Questions

What is the role of the BatchSandboxProvider in the OpenSandbox architecture?

The BatchSandboxProvider serves as the bridge between the OpenSandbox service API and the Kubernetes control plane. Implemented in server/src/services/k8s/batchsandbox_provider.py, it transforms high-level user requests into valid BatchSandbox CRs by constructing pod specifications, selecting RuntimeClasses, merging templates, and invoking the Kubernetes CustomObjects API. It handles the complexity of init container injection, bootstrap script wrapping, and security context configuration.

How does the BatchSandbox controller handle pod scaling and lifecycle management?

The controller uses the BatchSandboxReconciler to watch CR changes and execute reconciliation loops. For scaling, the scaleBatchSandbox method (lines 260-322) compares desired replicas against existing pods, creating missing instances through utils.GetPodFromTemplate. For lifecycle management, it handles expiration via expireTime checks (lines 100-115), manages finalizers for resource cleanup (lines 226-258), and aggregates status counters through updateStatus (lines 401-409) using optimistic concurrency controls.

What is the difference between Direct mode and Pool mode in BatchSandbox?

Direct mode requires complete pod specifications including images and resources, with the controller creating fresh pods for each replica. Pool mode leverages pre-warmed Pool resources referenced via spec.poolRef, allowing the controller to allocate existing warm pods and inject only entrypoints and environment variables. Pool mode uses annotations (sandbox.opensandbox.io/alloc) to track allocations, significantly reducing cold-start latency for short-lived workloads.

How does BatchSandbox ensure security isolation in Kubernetes?

The architecture implements multiple isolation layers: RuntimeClass injection (via SecureRuntimeResolver) enables kernel-level isolation through technologies like Kata Containers; egress sidecars enforce network policies by routing traffic through controlled proxy containers; and security contexts automatically configure runAsUser, allowPrivilegeEscalation, and other hardening parameters when network policies are detected. These mechanisms combine to provide defense in depth for sandboxed workloads.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}