# How Worker Host Selection Works in OpenAI Symphony: Algorithm and Implementation

> Discover how Symphony's Orchestrator performs worker host selection using a capacity-aware algorithm that filters slots, respects preferences, and finds the least-loaded host.

- Repository: [OpenAI/symphony](https://github.com/openai/symphony)
- Tags: internals
- Published: 2026-05-08

---

**Symphony’s Orchestrator selects SSH-enabled worker hosts using a capacity-aware algorithm that filters available slots, respects caller preferences, and falls back to the least-loaded host, returning `:no_worker_capacity` when all hosts are saturated.**

Symphony, OpenAI's code orchestration framework, distributes Linear issue execution across remote SSH workers to parallelize development tasks. The **worker host selection** logic, implemented in the `SymphonyElixir.Orchestrator` module, balances load across configured machines while respecting per-host concurrency limits and explicit user preferences according to the source code in `openai/symphony`.

## The Host Selection Algorithm

The core selection logic resides in `SymphonyElixir.Orchestrator.select_worker_host/2` at lines 973–990 of [`elixir/lib/symphony_elixir/orchestrator.ex`](https://github.com/openai/symphony/blob/main/elixir/lib/symphony_elixir/orchestrator.ex). The algorithm follows a strict four-step pipeline to determine where an issue should execute.

### Reading Configured SSH Hosts

The process begins by reading the system configuration via `Config.settings!().worker.ssh_hosts`. If this list is empty, the orchestrator runs the issue locally by returning `nil`. Otherwise, the algorithm proceeds to evaluate each configured host for availability.

### Filtering by Capacity

For each candidate host, the orchestrator calls `worker_host_slots_available?/2` (lines 1025–1033) to enforce the per-host concurrency limit defined by `worker.max_concurrent_agents_per_host`. This helper function calculates current load using `running_worker_host_count/2` (lines 1010–1014), which counts active agents on the specific host. Only hosts with running counts below their configured limit survive this filter, producing the `available_hosts` list.

### Priority and Fallback Logic

With the filtered list, the algorithm applies the following priority order:

1. **Empty capacity**: If `available_hosts` is empty, return `:no_worker_capacity` to signal the orchestrator should retry later.
2. **Caller preference**: If the caller supplied a `preferred_worker_host` and `preferred_worker_host_available?/2` (lines 994–998) confirms it exists in `available_hosts`, that host is selected immediately.
3. **Least-loaded**: Otherwise, `least_loaded_worker_host/2` (lines 1001–1008) orders candidates by `running_worker_host_count/2` and selects the host with the smallest count, breaking ties by the host’s original index position.

## Configuration and Concurrency Limits

The selection algorithm depends on settings loaded by `SymphonyElixir.Config` at lines 30–34 of [`elixir/lib/symphony_elixir/config.ex`](https://github.com/openai/symphony/blob/main/elixir/lib/symphony_elixir/config.ex). The relevant configuration keys include:

- **`worker.ssh_hosts`**: List of SSH-enabled worker machines available for task execution.
- **`worker.max_concurrent_agents_per_host`**: Integer defining how many simultaneous agents may run on a single host before it is considered saturated.

These values are parsed from the [`WORKFLOW.md`](https://github.com/openai/symphony/blob/main/WORKFLOW.md) configuration file and drive the capacity calculations in the Orchestrator.

## Key Helper Functions

The implementation relies on several private helper functions to maintain clean separation of concerns:

- **`worker_host_slots_available?/2`**: Checks per-host capacity against `max_concurrent_agents_per_host` (lines 1025–1033).
- **`running_worker_host_count/2`**: Returns the number of active agents on a specific host (lines 1010–1014).
- **`least_loaded_worker_host/2`**: Sorts hosts by load and selects the minimum with stable tie-breaking (lines 1001–1008).
- **`preferred_worker_host_available?/2`**: Validates that a preferred host exists and has capacity (lines 994–998).
- **`worker_slots_available?/1,2`**: High-level guards used by the orchestrator loop to check global capacity before attempting selection (lines 1017–1023).

## Practical Implementation Examples

### Dispatching an Issue Runtime

The following snippet mirrors the production dispatch flow at lines 683–689 of [`orchestrator.ex`](https://github.com/openai/symphony/blob/main/orchestrator.ex):

```elixir
case SymphonyElixir.Orchestrator.select_worker_host(state, preferred_host) do
  :no_worker_capacity ->
    Logger.debug("No SSH worker slots available … will retry later")
    # Retry logic handled by orchestrator loop

  worker_host ->
    # Spawn agent on chosen host (or nil for local execution)

    SymphonyElixir.AgentRunner.run(issue, recipient,
      attempt: attempt,
      worker_host: worker_host
    )
end

```

### Testing Host Selection

For unit testing, the module exposes `select_worker_host_for_test/2` at lines 331–335, which forwards to the private selector:

```elixir
defmodule OrchestratorTest do
  use ExUnit.Case, async: true

  test "prefers the requested host when it has capacity" do
    state = %SymphonyElixir.Orchestrator.State{
      max_concurrent_agents: 10,
      running: %{}
      # Config stubbed to return SSH hosts via Mox

    }

    assert "host-a.example.com" ==
             SymphonyElixir.Orchestrator.select_worker_host_for_test(state,
               "host-a.example.com")
  end
end

```

### Checking Per-Host Availability

External dashboards or tooling can inspect capacity using:

```elixir
def available_worker?(state, host) do
  SymphonyElixir.Orchestrator.worker_host_slots_available?(state, host)
end

```

## Summary

- **Worker host selection** in Symphony is handled by `SymphonyElixir.Orchestrator.select_worker_host/2` in [`elixir/lib/symphony_elixir/orchestrator.ex`](https://github.com/openai/symphony/blob/main/elixir/lib/symphony_elixir/orchestrator.ex).
- The algorithm filters configured SSH hosts by their current load against `worker.max_concurrent_agents_per_host`.
- Caller preferences take precedence over the default least-loaded strategy.
- When all hosts are saturated, the system returns `:no_worker_capacity` to trigger retry logic.
- Configuration originates from `SymphonyElixir.Config` parsing the `worker` section of [`WORKFLOW.md`](https://github.com/openai/symphony/blob/main/WORKFLOW.md).

## Frequently Asked Questions

### How does Symphony handle worker host selection when all SSH hosts are at capacity?

When every configured host exceeds its `max_concurrent_agents_per_host` limit, `select_worker_host/2` returns the atom `:no_worker_capacity`. The orchestrator interprets this signal to defer the issue and retry the selection later, preventing over-commitment of remote resources.

### Can I force a specific worker host for a particular issue execution?

Yes. Pass the desired hostname as the second argument (`preferred_worker_host`) to `select_worker_host/2`. If that host appears in the available list and has free slots, `preferred_worker_host_available?/2` validates it and the algorithm selects it immediately, bypassing the least-loaded logic.

### What determines which host is chosen when multiple workers have equal load?

When multiple hosts share the same number of running agents, `least_loaded_worker_host/2` breaks ties using the host’s index position in the original `worker.ssh_hosts` configuration list. This ensures deterministic, stable selection across identical load conditions.

### Where is the maximum concurrent agents per host configured?

The limit is defined in the [`WORKFLOW.md`](https://github.com/openai/symphony/blob/main/WORKFLOW.md) configuration file under the `worker.max_concurrent_agents_per_host` key, loaded at runtime by `SymphonyElixir.Config` (lines 30–34 of [`elixir/lib/symphony_elixir/config.ex`](https://github.com/openai/symphony/blob/main/elixir/lib/symphony_elixir/config.ex)). This value is compared against the current agent count by `worker_host_slots_available?/2` during the selection process.