Symphony SSH Workers Architecture: How OpenAI Symphony Manages Remote Agents

Symphony SSH workers enable distributed agent execution across remote hosts via a four-layer architecture comprising configuration-driven host discovery, a dedicated SSH command wrapper, workspace-level remote execution, and intelligent orchestrator-based load balancing.

OpenAI's Symphony framework extends agent execution beyond local machines through its Symphony SSH workers architecture, which allows the orchestrator to dispatch tasks to remote hosts over SSH. This architecture decouples agent runtime from the orchestrator node, enabling scalable distributed workflows while maintaining strict control over concurrency through per-host capacity limits.

Configuration Schema for Remote Workers

Symphony discovers available remote workers through the worker section of its configuration schema, defined in elixir/lib/symphony_elixir/config/schema.ex. The system expects Config.settings!().worker.ssh_hosts to contain an array of host connection strings.

The configuration supports two key parameters:

• ssh_hosts: An array of strings defining remote targets (e.g., "ssh://[email protected]" or "localhost:2222" shorthand)
• max_concurrent_agents_per_host: Integer defining the upper limit of simultaneous agents allowed per host

Configure SSH workers in your settings file:

# config/config.exs or symfony.yaml equivalent

%{
  worker: %{
    ssh_hosts: ["[email protected]", "localhost:2222"],
    max_concurrent_agents_per_host: 4
  }
}

The SSH Command Wrapper

The SymphonyElixir.SSH module (elixir/lib/symphony_elixir/ssh.ex) serves as the low-level interface to the system SSH binary. It discovers the local ssh executable, parses host strings (handling both "host:port" shorthand and IPv6 bracket notation), and supports custom SSH configurations via the SYMPHONY_SSH_CONFIG environment variable.

Key functions include:

• SSH.run/3: Executes commands remotely via System.cmd/3 and returns {:ok, {output, exit_status}} or error tuples
• SSH.start_port/3: Opens a port for streaming output when continuous data transfer is required
• SSH.ssh_args/2: Builds the argument list including optional config file injection
• SSH.parse_target/1: Normalizes host strings, extracting host and port components
• SSH.remote_shell_command/1: Wraps user commands for remote shell execution

Remote Workspace Execution

When agents require workspace operations on remote hosts, Workspace.run_remote_command/3 (lines 429-440 in elixir/lib/symphony_elixir/workspace.ex) coordinates the execution. This function receives the target worker_host and hook command, wraps the command as cd $WORKSPACE && $HOOK_CMD to ensure proper directory context, and invokes SSH.run/3 with timeout parameters derived from orchestrator settings.

This abstraction ensures that workspace creation, cleanup, and hook execution behave identically whether running locally or over SSH.

Orchestrator Load Balancing and Dispatch

The orchestrator (elixir/lib/symphony_elixir/orchestrator.ex) implements intelligent worker selection through select_worker_host/2 (lines 331-338). This function examines the ssh_hosts list from configuration, checks current agent counts against max_concurrent_agents_per_host, and returns a host string when capacity exists, or :no_worker_capacity when all hosts are saturated.

Upon selecting a host, spawn_issue_on_worker_host/5 (lines 693-698) initiates the remote workflow by spawning an AgentRunner with the worker_host parameter set. The runner subsequently invokes workspace functions that ultimately reach SSH.run/3.

Complete Execution Flow

The Symphony SSH workers architecture operates through four distinct phases:

1. Startup Phase: The orchestrator loads Config.settings!(). Non-empty worker.ssh_hosts activate remote worker mode.
2. Dispatch Phase: Orchestrator.select_worker_host/2 evaluates host capacity and selects a target, or returns :no_worker_capacity if all hosts are busy.
3. Execution Phase: The selected host passes to AgentRunner, which calls Workspace.run_remote_command/3, triggering SSH.run/3 to build the final command line (including optional -F $SYMPHONY_SSH_CONFIG) and execute via the system SSH binary.
4. Resolution Phase: Output and exit status return through the SSH wrapper to the workspace layer, which updates issue state accordingly.

Implementation Examples

Execute a command on a specific worker manually:

host = "[email protected]"
cmd = "git clone https://github.com/openai/example.git /tmp/example && make test"

case SymphonyElixir.SSH.run(host, cmd, stderr_to_stdout: true) do
  {:ok, {output, 0}} -> 
    IO.puts("Success:\n#{output}")
  
  {:ok, {output, code}} -> 
    IO.puts("Failed with exit #{code}:\n#{output}")
  
  {:error, reason} -> 
    IO.puts("SSH error: #{inspect(reason)}")
end

Handle orchestrator dispatch logic:

# Within Orchestrator.dispatch_issue/4

case select_worker_host(state, preferred_worker_host) do
  :no_worker_capacity ->
    Logger.debug("No SSH slots available – falling back to local execution")
  
  nil ->
    # Local execution path when worker_host is nil

    
  host ->
    spawn_issue_on_worker_host(state, issue, attempt, recipient, host)
end

Summary

• Configuration-driven discovery: Symphony reads Config.settings!().worker.ssh_hosts to identify available remote targets and enforces max_concurrent_agents_per_host limits.
• Robust SSH abstraction: The SymphonyElixir.SSH module handles host parsing, custom config injection via SYMPHONY_SSH_CONFIG, and standardized command execution through SSH.run/3.
• Workspace transparency: Workspace.run_remote_command/3 enables seamless remote execution of workspace hooks with proper directory context and timeout handling.
• Intelligent orchestration: The orchestrator balances load across SSH workers through select_worker_host/2, returning :no_worker_capacity when hosts are saturated rather than overloading remote machines.

Frequently Asked Questions

How does Symphony handle SSH authentication for remote workers?

Symphony delegates SSH authentication to the underlying system SSH binary. You can specify identity files, keys, or other SSH options through the SYMPHONY_SSH_CONFIG environment variable, which the SymphonyElixir.SSH module injects as -F $SYMPHONY_SSH_CONFIG when building SSH commands.

What happens when all SSH workers are at capacity?

When Orchestrator.select_worker_host/2 determines that all configured hosts have reached their max_concurrent_agents_per_host limit, it returns :no_worker_capacity. The orchestrator can then fall back to local execution or queue the issue until a remote slot becomes available.

Can SSH workers use non-standard ports or IPv6 addresses?

Yes. The SSH.parse_target/1 function in elixir/lib/symphony_elixir/ssh.ex handles both "host:port" shorthand notation and IPv6 bracket notation (e.g., "[::1]:2222"), allowing flexible network configurations beyond standard port 22 SSH connections.

Where does the command execution context get set for remote hooks?

The Workspace.run_remote_command/3 function (lines 429-440 in elixir/lib/symphony_elixir/workspace.ex) automatically wraps commands with cd $WORKSPACE && $HOOK_CMD, ensuring that remote hooks execute within the correct workspace directory regardless of the SSH user's default shell location.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}