# WORKER vs KUBERNETES Backend Kinds in Iris: Execution Models Explained

> Explore Iris WORKER vs KUBERNETES backend kinds. Understand their distinct execution models for managing workloads and discover which fits your needs.

- Repository: [The Marin Project/marin](https://github.com/marin-community/marin)
- Tags: deep-dive
- Published: 2026-08-29

---

**The `WORKER` backend runs the Iris scheduler locally and manages workers via RPC, while the `KUBERNETES` backend delegates placement to Kueue and reconciles Pods directly.**

Iris, the distributed task orchestrator in [marin-community/marin](https://github.com/marin-community/marin), supports two distinct backend execution models through its `BackendKind` enum. Understanding the differences between `WORKER` and `KUBERNETES` backend kinds is critical for choosing the right deployment architecture for your cluster. This guide breaks down the technical distinctions, code paths, and use cases for each backend type.

---

## BackendKind Enum Definition

The `BackendKind` enum in [`iris/cluster/controller/backend.py`](https://github.com/marin-community/marin/blob/main/iris/cluster/controller/backend.py) provides the foundational distinction:

```python
from enum import StrEnum

class BackendKind(StrEnum):
    """The execution mechanism owned by a controller's backend."""
    WORKER = "worker‑daemon"
    KUBERNETES = "kubernetes"

```

Each `BackendDescriptor` carries this `kind` field, which determines the controller's entire runtime behavior. The backend descriptor also includes `backend_id`, `advertised_attributes`, `scale_groups`, and `display_name`, but the `kind` field is the primary selector for conditional logic throughout the controller stack.

---

## Execution Model: Local Scheduler vs. Kubernetes Delegation

### WORKER Backend Runs Iris Scheduler Locally

A `WORKER` backend operates in **Iris-centric mode**. The controller runs the native Iris scheduler and communicates with workers directly through RPC calls. This model gives Iris full control over task placement, worker health, and autoscaling.

Key characteristics of the `WORKER` backend:

- **Scheduler location**: Runs inside the controller process
- **Worker communication**: Direct RPC via worker network addresses
- **State ownership**: Maintains a `ControllerDB` worker store tracking active workers, health status, and resource availability
- **Autoscaling**: Uses `WorkerHealthTracker` for worker-specific reconcile and autoscaling logic

In [`iris/cluster/controller/controller.py`](https://github.com/marin-community/marin/blob/main/iris/cluster/controller/controller.py), worker-only actions are guarded with explicit kind checks:

```python
if descriptor.kind is BackendKind.WORKER:
    # Health tracking, autoscaling, and worker store updates

    self._update_worker_health()
    self._trigger_autoscale()

```

### KUBERNETES Backend Delegates to Kueue

A `KUBERNETES` backend operates in **substrate-delegated mode**. Instead of running its own scheduler, the controller hands placement decisions to [Kueue](https://kueue.sigs.k8s.io/), the Kubernetes-native job queueing system. The controller's role shifts to Pod lifecycle management.

Key characteristics of the `KUBERNETES` backend:

- **Scheduler location**: External (Kueue manages queueing and placement)
- **Worker communication**: Uses `attempt_uid` to rebuild pod names; no network addresses needed
- **State ownership**: **No worker store** — placement state lives in the Kubernetes control plane
- **Autoscaling**: Relies on cluster-level autoscaling (Karpenter, Cluster Autoscaler, etc.)

The controller skips worker-specific reconcile logic for Kubernetes backends, as seen in [`iris/cluster/controller/controller.py`](https://github.com/marin-community/marin/blob/main/iris/cluster/controller/controller.py) around lines 1040-1052:

```python
if descriptor.kind is BackendKind.KUBERNETES:
    # Direct Pod reconciliation, no worker health tracking

    self._reconcile_pods()

```

---

## State Handling: Worker Store vs. Cluster Substrate

The presence or absence of a **worker store** is one of the most significant architectural differences between the two backend kinds.

| Component | WORKER | KUBERNETES |
|-----------|--------|------------|
| Worker store (`ControllerDB`) | ✅ Required | ❌ Absent |
| Health tracking | `WorkerHealthTracker` class | Kubernetes readiness probes |
| Resource accounting | Manual via worker store | Kueue quotas and flavors |
| Worker lifecycle | Explicit join/leave RPC | Pod creation/deletion events |

For `WORKER` backends, the `ControllerDB` maintains active worker records with their network addresses, capabilities, and health status. This enables fine-grained scheduling decisions and rapid failure detection.

For `KUBERNETES` backends, the controller trusts the cluster substrate. Worker identity maps to Pod identity through `attempt_uid`-derived names, and health is determined by Kubernetes-native mechanisms rather than custom tracking.

---

## Dashboard Capabilities: What Metrics Each Backend Exposes

The `dashboard_backend_descriptor` function in [`iris/cluster/controller/backend.py`](https://github.com/marin-community/marin/blob/main/iris/cluster/controller/backend.py) selects capabilities based on backend kind:

```python
def dashboard_backend_descriptor(controller):
    desc = controller.backend_descriptor
    if desc.kind is BackendKind.WORKER:
        return DashboardDescriptor(
            name=desc.display_name,
            capabilities=["workers", "autoscaler"]  # if autoscaling configured

        )
    else:  # KUBERNETES

        return DashboardDescriptor(
            name=desc.display_name,
            capabilities=["cluster"]
        )

```

This means:

- **WORKER backends** expose worker-specific metrics and autoscaling status
- **KUBERNETES backends** expose only cluster-level metrics, delegating detailed visibility to Kubernetes tooling (kubectl, Kueue dashboards, etc.)

---

## Task Targeting: Address-Based vs. UID-Based RPC Routing

Task RPC addressing differs fundamentally between the two backends:

### WORKER: Network Address Targeting

```python

# TaskTarget contains a direct network address

task_target = TaskTarget(address="10.0.0.15:9090")

# RPC routed to specific worker daemon

```

### KUBERNETES: attempt_uid to Pod Name Resolution

```python

# TaskTarget carries attempt_uid, controller rebuilds pod name

task_target = TaskTarget(attempt_uid="task-abc-123-xyz")

# Controller constructs: f"{task_name}-{attempt_uid}-{container_name}"

# No static address needed — Kubernetes DNS resolves the Pod

```

This design lets Kubernetes backends work with ephemeral Pod identities while maintaining stable task references through `attempt_uid`.

---

## Practical Examples: Configuring Each Backend Kind

### Creating a WORKER Backend Descriptor

```python
from iris.cluster.controller.backend import BackendDescriptor, BackendKind

worker_backend = BackendDescriptor(
    backend_id="local‑worker‑01",
    kind=BackendKind.WORKER,
    advertised_attributes={"gpu": {"v100"}},
    scale_groups=frozenset({"default"}),
    display_name="Local Worker Backend",
)

# Controller behavior:

#   * Native Iris scheduler runs locally

#   * RPC to workers via stored network addresses

#   * Worker health tracked via WorkerHealthTracker

#   * Dashboard shows "workers" + "autoscaler" tabs

```

### Creating a KUBERNETES Backend Descriptor

```python
from iris.cluster.controller.backend import BackendDescriptor, BackendKind

k8s_backend = BackendDescriptor(
    backend_id="k8s‑cluster‑01",
    kind=BackendKind.KUBERNETES,
    advertised_attributes={"gpu": {"a100"}},
    scale_groups=frozenset({"high‑mem"}),
    display_name="K8s Cluster Backend",
)

# Controller behavior:

#   * Placement delegated to Kueue

#   * Pods reconciled directly via Kubernetes API

#   * No worker store maintained

#   * Dashboard shows "cluster" tab only

```

### Runtime Backend Inspection

```python
from iris.cluster.controller.backend import dashboard_backend_descriptor

def print_capabilities(controller):
    desc = dashboard_backend_descriptor(controller)
    print(f"Backend '{desc.name}' supports: {', '.join(desc.capabilities)}")

# WORKER output: "Backend 'Local Worker Backend' supports: workers, autoscaler"

# KUBERNETES output: "Backend 'K8s Cluster Backend' supports: cluster"

```

---

## When to Use Each Backend Kind

| Scenario | Recommended Backend | Rationale |
|----------|---------------------|-----------|
| Small-to-medium cluster (<100 nodes) | `WORKER` | Direct control, lower operational complexity, no Kubernetes required |
| Large Kubernetes fleet | `KUBERNETES` | Leverages existing Kueue infrastructure, native autoscaling, multi-tenant queues |
| Custom hardware schedulers | `WORKER` | Native scheduler can be extended with custom placement policies |
| Serverless/bursty workloads | `KUBERNETES` | Kueue + Karpenter handles rapid scale-out without controller bottlenecks |
| Hybrid cloud | `KUBERNETES` | Consistent abstraction across cloud providers |

---

## Key Source Files and Their Roles

| File Path | Purpose |
|-----------|---------|
| [`iris/cluster/controller/backend.py`](https://github.com/marin-community/marin/blob/main/iris/cluster/controller/backend.py) | Defines `BackendKind`, `BackendDescriptor`, `dashboard_backend_descriptor()` |
| [`iris/cluster/controller/controller.py`](https://github.com/marin-community/marin/blob/main/iris/cluster/controller/controller.py) | Main controller loop with kind-based branching (lines 540-588 for WORKER, 1040-1052 for KUBERNETES) |
| [`iris/cluster/controller/service.py`](https://github.com/marin-community/marin/blob/main/iris/cluster/controller/service.py) | RPC routing logic that adapts to backend kind |
| [`iris/tests/journeys/backend.py`](https://github.com/marin-community/marin/blob/main/iris/tests/journeys/backend.py) | Integration tests validating both backend behaviors |
| [`iris/tests/cluster/controller/test_direct_controller.py`](https://github.com/marin-community/marin/blob/main/iris/tests/cluster/controller/test_direct_controller.py) | Unit tests for conditional backend paths |

The module docstring in [`iris/cluster/controller/backend.py`](https://github.com/marin-community/marin/blob/main/iris/cluster/controller/backend.py) (lines 18-22) explicitly documents this architectural split:

> *"The two backend kinds have deliberately different mechanisms: a **worker backend** runs the Iris scheduler and worker RPCs, while a **Kubernetes backend** hands placement to Kueue and reconciles Pods directly."*

---

## Summary

- **`WORKER` backends** run the native Iris scheduler, maintain a worker store, track health via `WorkerHealthTracker`, and expose worker/autoscaler dashboard capabilities
- **`KUBERNETES` backends** delegate scheduling to Kueue, have no worker store, reconcile Pods directly, and expose only cluster-level dashboard metrics
- The `BackendKind` enum in [`iris/cluster/controller/backend.py`](https://github.com/marin-community/marin/blob/main/iris/cluster/controller/backend.py) drives conditional logic throughout the controller stack
- Task targeting uses network addresses for `WORKER` and `attempt_uid`-based pod names for `KUBERNETES`
- Choose `WORKER` for direct control and smaller deployments; choose `KUBERNETES` for large-scale Kubernetes-native operation

---

## Frequently Asked Questions

### Can a single Iris controller switch between WORKER and KUBERNETES backends at runtime?

No. The `BackendDescriptor` is set at controller initialization and immutable thereafter. The controller's entire execution path—including scheduler instantiation, state management, and RPC handling—is determined by the `kind` field. To change backends, you must restart the controller with a new descriptor.

### Does the KUBERNETES backend support custom Iris scheduling policies?

No. Custom scheduling policies require the native Iris scheduler, which only runs with `BackendKind.WORKER`. The `KUBERNETES` backend intentionally relinquishes scheduling control to Kueue, inheriting its queueing and placement semantics. For custom policies in Kubernetes environments, consider extending Kueue's flavors or using `WORKER` backends with a Kubernetes-sidecar deployment model.

### How does autoscaling differ between the two backend kinds?

`WORKER` backends use Iris-native autoscaling via `WorkerHealthTracker` and explicit scale-group logic in the controller. `KUBERNETES` backends rely entirely on cluster-level autoscaling: Karpenter, Cluster Autoscaler, or node pools responding to pending Pod conditions. The controller does not perform worker-specific autoscale calculations for Kubernetes backends.