# BatchSandbox Runtime Architecture in Kubernetes: A Deep Dive into Alibaba OpenSandbox

> Explore the BatchSandbox runtime architecture in Kubernetes. Discover its three-layer design with CRD controller and provider SDK for efficient pod management.

- Repository: [Alibaba/OpenSandbox](https://github.com/alibaba/OpenSandbox)
- Tags: architecture
- Published: 2026-03-08

---

**The BatchSandbox runtime implements a three-layer Kubernetes-native architecture comprising a CRD for declarative state, a controller for reconciliation and lifecycle management, and a provider SDK for manifest generation, supporting both direct pod creation and pool-based warm allocation modes.**

The **BatchSandbox** runtime is a core component of Alibaba's OpenSandbox project that enables secure, scalable sandbox execution within Kubernetes clusters. This architectural design centers on a custom resource definition (CRD) that bridges high-level user requests with low-level pod orchestration through a specialized controller and provider layer. Understanding this architecture is essential for operators deploying OpenSandbox in production Kubernetes environments.

## Three-Layer Architecture Overview

The BatchSandbox runtime architecture is divided into three tightly coupled layers that handle distinct responsibilities within the Kubernetes control plane.

### CRD and API Layer

The foundation resides in [`kubernetes/apis/sandbox/v1alpha1/batchsandbox_types.go`](https://github.com/alibaba/OpenSandbox/blob/main/kubernetes/apis/sandbox/v1alpha1/batchsandbox_types.go), which defines the **BatchSandbox** custom resource. This layer declares the desired state through `BatchSandboxSpec` (lines 23-71) and reports observed state through `BatchSandboxStatus` (lines 82-103). The spec captures critical parameters including **replicas**, **poolRef** for warm pool references, **template** for PodTemplateSpec injection, **expireTime** for automatic deletion deadlines, and **taskTemplate** for post-startup task scheduling.

### Controller Layer

The reconciliation logic lives in [`kubernetes/internal/controller/batchsandbox_controller.go`](https://github.com/alibaba/OpenSandbox/blob/main/kubernetes/internal/controller/batchsandbox_controller.go). The **BatchSandboxReconciler** (specifically the `Reconcile` method at lines 82-128) watches BatchSandbox objects and manages the full lifecycle: creating and updating underlying Pods, handling expiration logic, managing pool allocations, and driving the task-scheduling subsystem. This layer ensures optimistic concurrency through `client.RawPatch` and `Status().Update` operations (see `updateStatus` at lines 401-409).

### Provider Layer

The server-side SDK resides in [`server/src/services/k8s/batchsandbox_provider.py`](https://github.com/alibaba/OpenSandbox/blob/main/server/src/services/k8s/batchsandbox_provider.py). The **BatchSandboxProvider** class offers a high-level Python API that translates user requests into valid BatchSandbox CRs. It constructs the pod specification including init containers for the `execd` installer, wraps user entrypoints with [`bootstrap.sh`](https://github.com/alibaba/OpenSandbox/blob/main/bootstrap.sh), injects optional egress sidecars, and merges user-supplied templates via `BatchSandboxTemplateManager` (lines 24-31 in [`batchsandbox_template.py`](https://github.com/alibaba/OpenSandbox/blob/main/batchsandbox_template.py)).

## CRD Specification and Status Contract

The `BatchSandboxSpec` struct defines the declarative contract between users and the OpenSandbox system. Key fields include:

- **replicas**: The desired number of sandbox pods to maintain
- **poolRef**: Optional reference to a pre-warmed **Pool** resource; when present, the controller operates in pool mode
- **template**: A standard `corev1.PodTemplateSpec` that the controller expands into real Pods with volumes and container specifications
- **expireTime**: Absolute deletion deadline for automatic cleanup
- **taskTemplate**: A Task spec automatically scheduled after pod creation, processed by the controller's task-scheduler

The corresponding `BatchSandboxStatus` (lines 82-103) tracks the observed replica count, allocation and ready counts, and per-task counters including `TaskRunning`, `TaskSucceed`, and failure states. This status sub-resource enables the controller to report accurate state without modifying the spec.

## Provider Manifest Generation Process

When the OpenSandbox service initiates a sandbox, it invokes `BatchSandboxProvider.create_workload(...)` (lines 13-62). The provider executes a deterministic build sequence:

1. **RuntimeClass Selection**: If secure runtime isolation is configured, the provider invokes `SecureRuntimeResolver` to select an appropriate RuntimeClass (lines 84-88) and injects `runtimeClassName` into the pod spec (lines 107-110).

2. **Template Loading**: The provider loads user-supplied templates via `BatchSandboxTemplateManager` (lines 24-31), extracting read-only volumes and volumeMounts through `_extract_template_pod_extras` (lines 176-190).

3. **Init Container Construction**: The `_build_execd_init_container` method (lines 50-84) creates an init container that copies the `execd` binary and [`bootstrap.sh`](https://github.com/alibaba/OpenSandbox/blob/main/bootstrap.sh) into a shared `emptyDir` volume named `opensandbox-bin`.

4. **Main Container Wrapping**: The `_build_main_container` method (lines 96-133) wraps the user entrypoint with [`bootstrap.sh`](https://github.com/alibaba/OpenSandbox/blob/main/bootstrap.sh) inside the main container, ensuring the execd agent is available before user processes start.

5. **Network Policy Application**: When egress control is required, `apply_egress_to_spec` (lines 112-117) injects an egress sidecar container routing outbound traffic through a configurable image. Concurrently, `build_security_context_for_sandbox_container` sets `runAsUser`, `allowPrivilegeEscalation`, and other security contexts (lines 136-144).

6. **CR Creation**: Finally, the provider calls the Kubernetes CustomObjects API to create the `BatchSandbox` resource, merging runtime-generated fields with user templates via `self.template_manager.merge_with_runtime_values`.

## Controller Reconciliation Logic

The **BatchSandboxReconciler** processes every BatchSandbox change through its `Reconcile` method (lines 82-128), implementing the following operational steps:

- **Expiration Handling**: Lines 100-115 check `expireTime` and delete the CR when the deadline passes.

- **Pool Detection**: Line 122 constructs a `PoolStrategy` to determine whether the CR operates in pooled mode based on `spec.poolRef` presence.

- **Pod Discovery**: The `listPods` method (lines 174-210) returns either Pods owned by the CR in direct mode, or the allocated pod set recorded in annotations (`sandbox.opensandbox.io/alloc`) when in pool mode.

- **Scaling Operations**: `scaleBatchSandbox` (lines 260-322) creates missing Pods according to `spec.replicas`, applying per-shard patches from `spec.shardPatches`. It generates Pods from templates using `utils.GetPodFromTemplate` and establishes owner references via `metav1.NewControllerRef`.

- **Status Aggregation**: Lines 131-165 compute `replicas`, `allocated`, and `ready` counts, updating the status sub-resource to reflect current cluster state.

- **Task Scheduling**: When `taskTemplate` is present, the controller instantiates a `TaskScheduler` (lines 188-207) and invokes `scheduleTasks` (lines 203-229) to create Task objects and update `status.task*` counters.

- **Finalizer Management**: Lines 226-258 ensure task resources release before final CR deletion, parsing allocation and release annotations via `parseSandboxAllocation` and `parseSandboxReleased` (lines 520-560).

## Deployment Modes: Direct vs Pool

The BatchSandbox runtime supports two distinct operational modes:

**Direct mode** (default) requires the provider to supply a complete pod specification including image, resources, and environment variables. The controller creates one pod per replica and tracks them individually through standard Kubernetes owner references.

**Pool mode** activates when `spec.poolRef` points to a pre-created **Pool** object containing warm pods. In this mode, the provider only specifies `taskTemplate` (entrypoint and environment), while the controller allocates existing pods from the warm pool. Allocation and release states persist in CR annotations (`sandbox.opensandbox.io/alloc` and `sandbox.opensandbox.io/release`), with the controller reconciling only the active subset of pods.

## Security and Networking Implementation

The architecture implements defense in depth through multiple isolation mechanisms:

- **Egress Sidecar**: The `apply_egress_to_spec` function injects a dedicated sidecar container that routes outbound traffic through a configurable `egress_image`, enabling fine-grained network policy enforcement.

- **Secure RuntimeClass**: Via `SecureRuntimeResolver`, the provider detects secure runtime configurations and injects `runtimeClassName` (e.g., Kata Containers) into the pod spec, providing kernel-level isolation.

- **Security Contexts**: When network policies are attached, the provider invokes `build_security_context_for_sandbox_container` to configure `runAsUser`, disable privilege escalation, and apply additional hardening parameters.

## End-to-End Execution Flow

A complete sandbox lifecycle follows this sequence:

1. The OpenSandbox service receives an API call and invokes `BatchSandboxProvider.create_workload`.
2. The provider constructs a BatchSandbox CR with init containers, main containers, optional egress sidecars, and RuntimeClass specifications.
3. The controller detects the new CR and either creates required Pods or selects them from a warm pool.
4. Pods initialize, executing the init container to install `execd` into `opensandbox-bin`, then launch [`bootstrap.sh`](https://github.com/alibaba/OpenSandbox/blob/main/bootstrap.sh) to start the user process.
5. If a **TaskTemplate** is defined, the controller's task-scheduler creates Task objects, monitors their lifecycle, and updates status counters.
6. Upon `expireTime` arrival or explicit deletion requests, the controller deletes Pods, cleans up pool allocation annotations, and removes the CR.

## Key Implementation Files

The following source files constitute the complete BatchSandbox runtime implementation:

- [`kubernetes/apis/sandbox/v1alpha1/batchsandbox_types.go`](https://github.com/alibaba/OpenSandbox/blob/main/kubernetes/apis/sandbox/v1alpha1/batchsandbox_types.go) — CRD definition for spec and status schemas
- [`kubernetes/internal/controller/batchsandbox_controller.go`](https://github.com/alibaba/OpenSandbox/blob/main/kubernetes/internal/controller/batchsandbox_controller.go) — Core reconciliation, scaling, and lifecycle logic
- [`server/src/services/k8s/batchsandbox_provider.py`](https://github.com/alibaba/OpenSandbox/blob/main/server/src/services/k8s/batchsandbox_provider.py) — High-level Python API for CR generation
- [`server/src/services/k8s/batchsandbox_template.py`](https://github.com/alibaba/OpenSandbox/blob/main/server/src/services/k8s/batchsandbox_template.py) — Template loading and merging utilities
- [`kubernetes/internal/controller/strategy/pool_strategy.go`](https://github.com/alibaba/OpenSandbox/blob/main/kubernetes/internal/controller/strategy/pool_strategy.go) — Pool allocation policies
- [`kubernetes/internal/controller/strategy/task_scheduling_strategy.go`](https://github.com/alibaba/OpenSandbox/blob/main/kubernetes/internal/controller/strategy/task_scheduling_strategy.go) — Task scheduling implementation
- [`kubernetes/pkg/client/listers/sandbox/v1alpha1/batchsandbox.go`](https://github.com/alibaba/OpenSandbox/blob/main/kubernetes/pkg/client/listers/sandbox/v1alpha1/batchsandbox.go) — Generated client listers for cache access

## Code Examples

### Python: Creating a Sandbox in Direct Mode

```python
from src.services.k8s.batchsandbox_provider import BatchSandboxProvider
from src.api.schema import ImageSpec
from datetime import datetime, timedelta

provider = BatchSandboxProvider(k8s_client, template_file_path="batch.yaml")

provider.create_workload(
    sandbox_id="demo-sbx",
    namespace="default",
    image_spec=ImageSpec(uri="python:3.11-slim"),
    entrypoint=["python", "app.py"],
    env={"ENV": "prod"},
    resource_limits={"cpu": "2", "memory": "4Gi"},
    labels={"app": "demo"},
    expires_at=datetime.utcnow() + timedelta(hours=6),
    execd_image="alibaba/execd:latest",
)

```

*Relevant source*: `BatchSandboxProvider.create_workload` — lines 13-62.

### YAML: BatchSandbox CR in Pool Mode

```yaml
apiVersion: sandbox.opensandbox.io/v1alpha1
kind: BatchSandbox
metadata:
  name: pooled-sbx
  namespace: default
spec:
  replicas: 1
  poolRef: demo-pool
  expireTime: "2026-04-01T00:00:00Z"
  taskTemplate:
    spec:
      process:
        command: ["/bin/sh", "-c", "/opt/opensandbox/bin/bootstrap.sh python app.py"]
        env:
        - name: ENV
          value: prod

```

*CRD schema reference*: `BatchSandboxSpec` — lines 23-71.

### Go: Pod Scaling Logic

```go
// Inside scaleBatchSandbox (lines 260-322)
for i := 0; i < int(*batchSandbox.Spec.Replicas); i++ {
    if _, ok := indexedPodMap[i]; !ok {
        needCreateIndex = append(needCreateIndex, i)
    }
}
for _, idx := range needCreateIndex {
    pod, err := utils.GetPodFromTemplate(
        podTemplateSpec, 
        batchSandbox,
        metav1.NewControllerRef(batchSandbox, sandboxv1alpha1.SchemeBuilder.GroupVersion.WithKind("BatchSandbox")),
    )
    if err != nil {
        return err
    }
    r.Create(ctx, pod)
}

```

*Relevant source*: `scaleBatchSandbox` — lines 260-322.

## Summary

- The BatchSandbox runtime implements a three-layer architecture separating concerns between API definition (CRD), operational logic (Controller), and user interface (Provider).
- The controller manages complex lifecycle operations including expiration, pool-based allocation, task scheduling, and optimistic concurrency control.
- Two deployment modes exist: Direct mode for full pod specification control, and Pool mode for warm-start optimization using pre-allocated pods.
- Security implementations include RuntimeClass injection for kernel isolation, egress sidecars for network policy enforcement, and automated security context configuration.
- All operations are traceable through specific source files in the alibaba/OpenSandbox repository, with reconciliation logic concentrated in the Go controller and manifest generation handled by the Python provider.

## Frequently Asked Questions

### What is the role of the BatchSandboxProvider in the OpenSandbox architecture?

The **BatchSandboxProvider** serves as the bridge between the OpenSandbox service API and the Kubernetes control plane. Implemented in [`server/src/services/k8s/batchsandbox_provider.py`](https://github.com/alibaba/OpenSandbox/blob/main/server/src/services/k8s/batchsandbox_provider.py), it transforms high-level user requests into valid BatchSandbox CRs by constructing pod specifications, selecting RuntimeClasses, merging templates, and invoking the Kubernetes CustomObjects API. It handles the complexity of init container injection, bootstrap script wrapping, and security context configuration.

### How does the BatchSandbox controller handle pod scaling and lifecycle management?

The controller uses the **BatchSandboxReconciler** to watch CR changes and execute reconciliation loops. For scaling, the `scaleBatchSandbox` method (lines 260-322) compares desired replicas against existing pods, creating missing instances through `utils.GetPodFromTemplate`. For lifecycle management, it handles expiration via `expireTime` checks (lines 100-115), manages finalizers for resource cleanup (lines 226-258), and aggregates status counters through `updateStatus` (lines 401-409) using optimistic concurrency controls.

### What is the difference between Direct mode and Pool mode in BatchSandbox?

**Direct mode** requires complete pod specifications including images and resources, with the controller creating fresh pods for each replica. **Pool mode** leverages pre-warmed **Pool** resources referenced via `spec.poolRef`, allowing the controller to allocate existing warm pods and inject only entrypoints and environment variables. Pool mode uses annotations (`sandbox.opensandbox.io/alloc`) to track allocations, significantly reducing cold-start latency for short-lived workloads.

### How does BatchSandbox ensure security isolation in Kubernetes?

The architecture implements multiple isolation layers: **RuntimeClass injection** (via `SecureRuntimeResolver`) enables kernel-level isolation through technologies like Kata Containers; **egress sidecars** enforce network policies by routing traffic through controlled proxy containers; and **security contexts** automatically configure `runAsUser`, `allowPrivilegeEscalation`, and other hardening parameters when network policies are detected. These mechanisms combine to provide defense in depth for sandboxed workloads.