# How Iris Handles Scheduling Policy and Per‑User Budget Enforcement

> Iris enforces scheduling policy and per-user budget enforcement via a three-stage pipeline. Learn how it dynamically downgrades priority and round-robins tasks to prevent monopolization.

- Repository: [The Marin Project/marin](https://github.com/marin-community/marin)
- Tags: how-to-guide
- Published: 2026-08-29

---

**Iris enforces scheduling policy and per‑user budget enforcement through a three‑stage pipeline that gates pending tasks, dynamically downgrades priority bands when users exceed spend limits, and round‑robins tasks within bands to prevent monopolization.**

The **marin-community/marin** repository implements Iris as a cluster controller that makes placement decisions every tick by combining hierarchical priority bands with real‑time budget accounting. This article examines the source code to explain how the system builds scheduling contexts, applies budget‑driven band adjustments, and ensures fair resource allocation across competing users.

## Building the Scheduling Context

Every controller tick begins with `build_scheduling_context` in [`lib/iris/src/iris/cluster/controller/scheduling/policy.py`](https://github.com/marin-community/marin/blob/main/lib/iris/src/iris/cluster/controller/scheduling/policy.py) (lines 66‑84). This function aggregates the cluster state required for policy decisions:

- **Healthy workers** and their available capacities.
- **Pending tasks** mapped to their requested static priority bands.
- **Per‑user spend** calculated via `compute_user_spend` and current budget limits fetched through `reads.get_all_user_budget_limits`.
- **Default budget limits** (`UserBudgetDefaults`) for users lacking explicit configuration rows.

The resulting context object serves as the immutable input for all downstream scheduling logic, ensuring that every decision uses a consistent snapshot of cluster health and user entitlement.

## Gating and Filtering Tasks

Before priority calculation, `apply_scheduling_gates` (same file, lines 25‑34) filters the pending task set to create `GatedCandidates`. This step enforces hard constraints such as:

- **Deadline violations** — tasks that cannot complete before their deadlines are dropped.
- **Per‑job caps** — enforcing `max_tasks_per_job_per_cycle` to prevent thundering herds.

Only tasks surviving this gate proceed to the priority computation stage, guaranteeing that the scheduler never considers invalid or excessive requests.

## Priority Band Calculation with Budget Enforcement

The `compute_scheduling_order` function (lines 78‑100) translates user spend data into effective scheduling priority through a two‑phase process.

### Budget‑Driven Band Downgrade

First, the system builds a **task‑to‑band map** by calling `compute_effective_band` from [`lib/iris/src/iris/cluster/controller/budget.py`](https://github.com/marin-community/marin/blob/main/lib/iris/src/iris/cluster/controller/budget.py) (lines 67‑88):

```python
task_band_map = {
    task.task_id: compute_effective_band(
        requested_bands.get(task.job_id, job_pb2.PRIORITY_BAND_INTERACTIVE),
        task.task_id.user,
        user_spend,
        user_budget_limits,
        defaults,
    )
}

```

When a user’s cumulative `resource_value` spend exceeds their configured `budget_limit`, `compute_effective_band` automatically downgrades the task to the **BATCH** band. This **per‑user budget enforcement** mechanism ensures that heavy consumers receive lower priority without manual intervention.

### System and Production Band Protection

Crucially, the downgrade logic preserves critical infrastructure work. **SYSTEM** and **PRODUCTION** bands are never downgraded regardless of spend, maintaining cluster stability and ensuring that high‑priority system‑wide tasks always preempt lower‑priority user workloads.

## Fair Scheduling Across Users

Within each priority band, `interleave_by_user` (lines 91‑112) implements a **round‑robin scheduling** strategy. The function orders tasks so that users who have consumed less of their budget receive earlier slots. This prevents any single user from monopolizing a priority band while respecting the hierarchical band structure.

## Task Assignment and Preemption

After establishing the ordered task list, the controller invokes `Scheduler.find_assignments` (implemented in [`lib/iris/src/iris/cluster/controller/scheduling/scheduler.py`](https://github.com/marin-community/marin/blob/main/lib/iris/src/iris/cluster/controller/scheduling/scheduler.py)) to place tasks on workers. This phase respects:

- **Per‑worker limits** such as `max_building_tasks` and `max_assignments_per_worker`.
- **Device‑variant matching**, ensuring that solo pre‑emptors only evict victims with identical accelerator variants.

### Preemption Hierarchy

When workers lack capacity, `run_preemption_pass` (lines 13‑45 in [`policy.py`](https://github.com/marin-community/marin/blob/main/policy.py)) enforces a strict eviction hierarchy:

1. **SYSTEM** tasks preempt all lower bands.
2. **PRODUCTION** tasks preempt INTERACTIVE and BATCH.
3. **INTERACTIVE** tasks preempt BATCH only.
4. **BATCH** tasks never preempt others.

Preemptions are only permitted across different priority bands; same‑band tasks compete exclusively through the ordering established earlier.

### Transactional Persistence

Once assignments and preemptions are selected, the controller persists decisions in a single database transaction using `writes.tasks.assign_task` and `writes.tasks.preempt_task`. This atomic write guarantees that a pre‑emptor never re‑competes for the slot it just freed, eliminating scheduling races.

## Summary

- Iris implements **scheduling policy and per‑user budget enforcement** through a declarative three‑stage pipeline defined in [`policy.py`](https://github.com/marin-community/marin/blob/main/policy.py).
- **Budget limits** trigger automatic downgrades to the BATCH band via `compute_effective_band` in [`budget.py`](https://github.com/marin-community/marin/blob/main/budget.py), while protecting SYSTEM and PRODUCTION workloads.
- **Fairness** is guaranteed by round‑robin interleaving within bands, preventing monopolization by heavy consumers.
- **Preemption** follows a strict hierarchy that respects band priority, and all decisions are committed atomically to maintain consistency.

## Frequently Asked Questions

### How does Iris prevent a single user from consuming all cluster resources?

Within each priority band, `interleave_by_user` round‑robins tasks based on current spend, ensuring that users with lower utilization get earlier scheduling slots. Additionally, when a user exceeds their `budget_limit`, all their tasks downgrade to the BATCH band, where they compete with other deprioritized workloads and can be preempted by higher‑priority bands.

### What happens to SYSTEM and PRODUCTION tasks when a user exceeds their budget?

SYSTEM and PRODUCTION bands are **never downgraded** regardless of spend. The `compute_effective_band` function explicitly preserves these bands to maintain cluster infrastructure stability and ensure that critical system workloads retain highest priority even when individual users exhaust their allocations.

### How does Iris handle task deadlines during scheduling?

The `apply_scheduling_gates` function filters out tasks that cannot meet their deadlines before they enter the priority calculation phase. This gating happens early in the pipeline (lines 25‑34 of [`policy.py`](https://github.com/marin-community/marin/blob/main/policy.py)), ensuring that the scheduler never wastes cycles on infeasible assignments and that deadline‑sensitive jobs fail fast rather than queue indefinitely.

### Where can I find tests validating the budget enforcement logic?

The repository includes comprehensive validation in [`lib/iris/tests/test_budget.py`](https://github.com/marin-community/marin/blob/main/lib/iris/tests/test_budget.py), which covers spend tracking and band downgrade behavior. Scheduling fairness across priority bands is verified in [`lib/iris/tests/cluster/controller/test_scheduling_fairness.py`](https://github.com/marin-community/marin/blob/main/lib/iris/tests/cluster/controller/test_scheduling_fairness.py), ensuring that the hierarchy and interleaving logic function correctly under diverse load patterns.