# Symphony SSH Workers Architecture: How OpenAI Symphony Manages Remote Agents

> Discover the Symphony SSH workers architecture. OpenAI Symphony uses a 4 layer system for distributed agent execution across remote hosts including host discovery, SSH command wrapping, remote execution, and load balancing.

- Repository: [OpenAI/symphony](https://github.com/openai/symphony)
- Tags: architecture
- Published: 2026-05-08

---

**Symphony SSH workers enable distributed agent execution across remote hosts via a four-layer architecture comprising configuration-driven host discovery, a dedicated SSH command wrapper, workspace-level remote execution, and intelligent orchestrator-based load balancing.**

OpenAI's Symphony framework extends agent execution beyond local machines through its **Symphony SSH workers architecture**, which allows the orchestrator to dispatch tasks to remote hosts over SSH. This architecture decouples agent runtime from the orchestrator node, enabling scalable distributed workflows while maintaining strict control over concurrency through per-host capacity limits.

## Configuration Schema for Remote Workers

Symphony discovers available remote workers through the `worker` section of its configuration schema, defined in [`elixir/lib/symphony_elixir/config/schema.ex`](https://github.com/openai/symphony/blob/main/elixir/lib/symphony_elixir/config/schema.ex). The system expects **`Config.settings!().worker.ssh_hosts`** to contain an array of host connection strings.

The configuration supports two key parameters:

- **`ssh_hosts`**: An array of strings defining remote targets (e.g., `"ssh://ci-runner@ci-host.example.com"` or `"localhost:2222"` shorthand)
- **`max_concurrent_agents_per_host`**: Integer defining the upper limit of simultaneous agents allowed per host

Configure SSH workers in your settings file:

```elixir

# config/config.exs or symfony.yaml equivalent

%{
  worker: %{
    ssh_hosts: ["ci-runner@ci-host.example.com", "localhost:2222"],
    max_concurrent_agents_per_host: 4
  }
}

```

## The SSH Command Wrapper

The `SymphonyElixir.SSH` module ([`elixir/lib/symphony_elixir/ssh.ex`](https://github.com/openai/symphony/blob/main/elixir/lib/symphony_elixir/ssh.ex)) serves as the low-level interface to the system SSH binary. It discovers the local `ssh` executable, parses host strings (handling both `"host:port"` shorthand and IPv6 bracket notation), and supports custom SSH configurations via the **`SYMPHONY_SSH_CONFIG`** environment variable.

Key functions include:

- **`SSH.run/3`**: Executes commands remotely via `System.cmd/3` and returns `{:ok, {output, exit_status}}` or error tuples
- **`SSH.start_port/3`**: Opens a port for streaming output when continuous data transfer is required
- **`SSH.ssh_args/2`**: Builds the argument list including optional config file injection
- **`SSH.parse_target/1`**: Normalizes host strings, extracting host and port components
- **`SSH.remote_shell_command/1`**: Wraps user commands for remote shell execution

## Remote Workspace Execution

When agents require workspace operations on remote hosts, `Workspace.run_remote_command/3` (lines 429-440 in [`elixir/lib/symphony_elixir/workspace.ex`](https://github.com/openai/symphony/blob/main/elixir/lib/symphony_elixir/workspace.ex)) coordinates the execution. This function receives the target `worker_host` and hook command, wraps the command as `cd $WORKSPACE && $HOOK_CMD` to ensure proper directory context, and invokes `SSH.run/3` with timeout parameters derived from orchestrator settings.

This abstraction ensures that workspace creation, cleanup, and hook execution behave identically whether running locally or over SSH.

## Orchestrator Load Balancing and Dispatch

The orchestrator ([`elixir/lib/symphony_elixir/orchestrator.ex`](https://github.com/openai/symphony/blob/main/elixir/lib/symphony_elixir/orchestrator.ex)) implements intelligent worker selection through **`select_worker_host/2`** (lines 331-338). This function examines the `ssh_hosts` list from configuration, checks current agent counts against `max_concurrent_agents_per_host`, and returns a host string when capacity exists, or `:no_worker_capacity` when all hosts are saturated.

Upon selecting a host, **`spawn_issue_on_worker_host/5`** (lines 693-698) initiates the remote workflow by spawning an `AgentRunner` with the `worker_host` parameter set. The runner subsequently invokes workspace functions that ultimately reach `SSH.run/3`.

## Complete Execution Flow

The Symphony SSH workers architecture operates through four distinct phases:

1. **Startup Phase**: The orchestrator loads `Config.settings!()`. Non-empty `worker.ssh_hosts` activate remote worker mode.
2. **Dispatch Phase**: `Orchestrator.select_worker_host/2` evaluates host capacity and selects a target, or returns `:no_worker_capacity` if all hosts are busy.
3. **Execution Phase**: The selected host passes to `AgentRunner`, which calls `Workspace.run_remote_command/3`, triggering `SSH.run/3` to build the final command line (including optional `-F $SYMPHONY_SSH_CONFIG`) and execute via the system SSH binary.
4. **Resolution Phase**: Output and exit status return through the SSH wrapper to the workspace layer, which updates issue state accordingly.

## Implementation Examples

Execute a command on a specific worker manually:

```elixir
host = "ci-runner@ci-host.example.com"
cmd = "git clone https://github.com/openai/example.git /tmp/example && make test"

case SymphonyElixir.SSH.run(host, cmd, stderr_to_stdout: true) do
  {:ok, {output, 0}} -> 
    IO.puts("Success:\n#{output}")
  
  {:ok, {output, code}} -> 
    IO.puts("Failed with exit #{code}:\n#{output}")
  
  {:error, reason} -> 
    IO.puts("SSH error: #{inspect(reason)}")
end

```

Handle orchestrator dispatch logic:

```elixir

# Within Orchestrator.dispatch_issue/4

case select_worker_host(state, preferred_worker_host) do
  :no_worker_capacity ->
    Logger.debug("No SSH slots available – falling back to local execution")
  
  nil ->
    # Local execution path when worker_host is nil

    
  host ->
    spawn_issue_on_worker_host(state, issue, attempt, recipient, host)
end

```

## Summary

- **Configuration-driven discovery**: Symphony reads `Config.settings!().worker.ssh_hosts` to identify available remote targets and enforces `max_concurrent_agents_per_host` limits.
- **Robust SSH abstraction**: The `SymphonyElixir.SSH` module handles host parsing, custom config injection via `SYMPHONY_SSH_CONFIG`, and standardized command execution through `SSH.run/3`.
- **Workspace transparency**: `Workspace.run_remote_command/3` enables seamless remote execution of workspace hooks with proper directory context and timeout handling.
- **Intelligent orchestration**: The orchestrator balances load across SSH workers through `select_worker_host/2`, returning `:no_worker_capacity` when hosts are saturated rather than overloading remote machines.

## Frequently Asked Questions

### How does Symphony handle SSH authentication for remote workers?

Symphony delegates SSH authentication to the underlying system SSH binary. You can specify identity files, keys, or other SSH options through the **`SYMPHONY_SSH_CONFIG`** environment variable, which the `SymphonyElixir.SSH` module injects as `-F $SYMPHONY_SSH_CONFIG` when building SSH commands.

### What happens when all SSH workers are at capacity?

When `Orchestrator.select_worker_host/2` determines that all configured hosts have reached their `max_concurrent_agents_per_host` limit, it returns `:no_worker_capacity`. The orchestrator can then fall back to local execution or queue the issue until a remote slot becomes available.

### Can SSH workers use non-standard ports or IPv6 addresses?

Yes. The `SSH.parse_target/1` function in [`elixir/lib/symphony_elixir/ssh.ex`](https://github.com/openai/symphony/blob/main/elixir/lib/symphony_elixir/ssh.ex) handles both `"host:port"` shorthand notation and IPv6 bracket notation (e.g., `"[::1]:2222"`), allowing flexible network configurations beyond standard port 22 SSH connections.

### Where does the command execution context get set for remote hooks?

The `Workspace.run_remote_command/3` function (lines 429-440 in [`elixir/lib/symphony_elixir/workspace.ex`](https://github.com/openai/symphony/blob/main/elixir/lib/symphony_elixir/workspace.ex)) automatically wraps commands with `cd $WORKSPACE && $HOOK_CMD`, ensuring that remote hooks execute within the correct workspace directory regardless of the SSH user's default shell location.