How Worker Host Selection Works in OpenAI Symphony: Algorithm and Implementation

Symphony’s Orchestrator selects SSH-enabled worker hosts using a capacity-aware algorithm that filters available slots, respects caller preferences, and falls back to the least-loaded host, returning :no_worker_capacity when all hosts are saturated.

Symphony, OpenAI's code orchestration framework, distributes Linear issue execution across remote SSH workers to parallelize development tasks. The worker host selection logic, implemented in the SymphonyElixir.Orchestrator module, balances load across configured machines while respecting per-host concurrency limits and explicit user preferences according to the source code in openai/symphony.

The Host Selection Algorithm

The core selection logic resides in SymphonyElixir.Orchestrator.select_worker_host/2 at lines 973–990 of elixir/lib/symphony_elixir/orchestrator.ex. The algorithm follows a strict four-step pipeline to determine where an issue should execute.

Reading Configured SSH Hosts

The process begins by reading the system configuration via Config.settings!().worker.ssh_hosts. If this list is empty, the orchestrator runs the issue locally by returning nil. Otherwise, the algorithm proceeds to evaluate each configured host for availability.

Filtering by Capacity

For each candidate host, the orchestrator calls worker_host_slots_available?/2 (lines 1025–1033) to enforce the per-host concurrency limit defined by worker.max_concurrent_agents_per_host. This helper function calculates current load using running_worker_host_count/2 (lines 1010–1014), which counts active agents on the specific host. Only hosts with running counts below their configured limit survive this filter, producing the available_hosts list.

Priority and Fallback Logic

With the filtered list, the algorithm applies the following priority order:

1. Empty capacity: If available_hosts is empty, return :no_worker_capacity to signal the orchestrator should retry later.
2. Caller preference: If the caller supplied a preferred_worker_host and preferred_worker_host_available?/2 (lines 994–998) confirms it exists in available_hosts, that host is selected immediately.
3. Least-loaded: Otherwise, least_loaded_worker_host/2 (lines 1001–1008) orders candidates by running_worker_host_count/2 and selects the host with the smallest count, breaking ties by the host’s original index position.

Configuration and Concurrency Limits

The selection algorithm depends on settings loaded by SymphonyElixir.Config at lines 30–34 of elixir/lib/symphony_elixir/config.ex. The relevant configuration keys include:

• worker.ssh_hosts: List of SSH-enabled worker machines available for task execution.
• worker.max_concurrent_agents_per_host: Integer defining how many simultaneous agents may run on a single host before it is considered saturated.

These values are parsed from the WORKFLOW.md configuration file and drive the capacity calculations in the Orchestrator.

Key Helper Functions

The implementation relies on several private helper functions to maintain clean separation of concerns:

• worker_host_slots_available?/2: Checks per-host capacity against max_concurrent_agents_per_host (lines 1025–1033).
• running_worker_host_count/2: Returns the number of active agents on a specific host (lines 1010–1014).
• least_loaded_worker_host/2: Sorts hosts by load and selects the minimum with stable tie-breaking (lines 1001–1008).
• preferred_worker_host_available?/2: Validates that a preferred host exists and has capacity (lines 994–998).
• worker_slots_available?/1,2: High-level guards used by the orchestrator loop to check global capacity before attempting selection (lines 1017–1023).

Practical Implementation Examples

Dispatching an Issue Runtime

The following snippet mirrors the production dispatch flow at lines 683–689 of orchestrator.ex:

case SymphonyElixir.Orchestrator.select_worker_host(state, preferred_host) do
  :no_worker_capacity ->
    Logger.debug("No SSH worker slots available … will retry later")
    # Retry logic handled by orchestrator loop

  worker_host ->
    # Spawn agent on chosen host (or nil for local execution)

    SymphonyElixir.AgentRunner.run(issue, recipient,
      attempt: attempt,
      worker_host: worker_host
    )
end

Testing Host Selection

For unit testing, the module exposes select_worker_host_for_test/2 at lines 331–335, which forwards to the private selector:

defmodule OrchestratorTest do
  use ExUnit.Case, async: true

  test "prefers the requested host when it has capacity" do
    state = %SymphonyElixir.Orchestrator.State{
      max_concurrent_agents: 10,
      running: %{}
      # Config stubbed to return SSH hosts via Mox

    }

    assert "host-a.example.com" ==
             SymphonyElixir.Orchestrator.select_worker_host_for_test(state,
               "host-a.example.com")
  end
end

Checking Per-Host Availability

External dashboards or tooling can inspect capacity using:

def available_worker?(state, host) do
  SymphonyElixir.Orchestrator.worker_host_slots_available?(state, host)
end

Summary

• Worker host selection in Symphony is handled by SymphonyElixir.Orchestrator.select_worker_host/2 in elixir/lib/symphony_elixir/orchestrator.ex.
• The algorithm filters configured SSH hosts by their current load against worker.max_concurrent_agents_per_host.
• Caller preferences take precedence over the default least-loaded strategy.
• When all hosts are saturated, the system returns :no_worker_capacity to trigger retry logic.
• Configuration originates from SymphonyElixir.Config parsing the worker section of WORKFLOW.md.

Frequently Asked Questions

How does Symphony handle worker host selection when all SSH hosts are at capacity?

When every configured host exceeds its max_concurrent_agents_per_host limit, select_worker_host/2 returns the atom :no_worker_capacity. The orchestrator interprets this signal to defer the issue and retry the selection later, preventing over-commitment of remote resources.

Can I force a specific worker host for a particular issue execution?

Yes. Pass the desired hostname as the second argument (preferred_worker_host) to select_worker_host/2. If that host appears in the available list and has free slots, preferred_worker_host_available?/2 validates it and the algorithm selects it immediately, bypassing the least-loaded logic.

What determines which host is chosen when multiple workers have equal load?

When multiple hosts share the same number of running agents, least_loaded_worker_host/2 breaks ties using the host’s index position in the original worker.ssh_hosts configuration list. This ensures deterministic, stable selection across identical load conditions.

Where is the maximum concurrent agents per host configured?

The limit is defined in the WORKFLOW.md configuration file under the worker.max_concurrent_agents_per_host key, loaded at runtime by SymphonyElixir.Config (lines 30–34 of elixir/lib/symphony_elixir/config.ex). This value is compared against the current agent count by worker_host_slots_available?/2 during the selection process.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}