How the Symphony Orchestrator Manages Concurrent Agent Limits

TLDR: The Symphony orchestrator prevents resource exhaustion by enforcing three independent limits—global, per-state, and per-host—evaluating available slots through the should_dispatch_issue?/5 function before dispatching any agent to handle Linear issues.

The OpenAI Symphony project provides an Elixir-based runtime that coordinates AI agents across Linear issue workflows. To maintain system stability, the Symphony orchestrator concurrent agent limits implementation applies a hierarchical gating mechanism that checks every potential dispatch against configurable thresholds defined in Config.settings!().

Three-Tier Concurrent Agent Limits

The orchestrator applies three distinct boundaries that operate simultaneously at different scopes.

Global Concurrent Agents

The global limit controls the total number of agents running across the entire orchestrator instance. This value populates State.max_concurrent_agents from Config.settings!().agent.max_concurrent_agents. The function available_slots/1 calculates remaining capacity by subtracting the current running agent count from this global maximum.

Per-State Concurrent Agents

The per-state limit restricts concurrency for issues sharing the same Linear state (e.g., "In Progress" or "Review"). Implemented in state_slots_available?/2, this check uses Config.max_concurrent_agents_for_state/1 to retrieve the limit, then compares it against the count from running_issue_count_for_state/2. This prevents any single workflow state from consuming all available resources.

Per-Host Concurrent Agents

When SSH workers are configured via Config.settings!().worker.ssh_hosts, the per-host limit ensures individual worker machines do not become bottlenecks. The function worker_host_slots_available?/2 utilizes Config.settings!().worker.max_concurrent_agents_per_host and counts matching entries via running_worker_host_count/2 to enforce capacity per specific host.

Dispatch Decision Logic for Agent Concurrency

Within orchestrator.ex, the dispatch_issue/2 function gates every potential agent startup through should_dispatch_issue?/5 (approximately lines 55-66). The orchestrator evaluates limits in strict sequence:

1. Global slot check — Verifies available_slots(state) > 0
2. State-specific check — Confirms state_slots_available?(issue, state.running) returns true
3. Worker-host check — Validates worker_slots_available?(state) or worker_slots_available?(state, preferred_host) when hosts are specified

If any limit has been reached, the orchestrator skips dispatch for that issue, leaving it in the candidate queue for the next polling cycle.

Runtime Configuration and Dynamic Limits

The limits are dynamic rather than static constants. On every tick, refresh_runtime_config/1 (lines 96-103) reloads configuration values, allowing administrators to adjust max_concurrent_agents thresholds without restarting the orchestrator. This ensures runtime adaptation to changing load conditions or infrastructure capacity.

Inspecting Concurrency Limits at Runtime

You can verify the current concurrency state interactively in an IEx session. The following snippet demonstrates how to check each limit tier using actual orchestrator functions:

# 1️⃣ Load the current orchestrator state (assumes the orchestrator is already started)

{:ok, state} = SymphonyElixir.Orchestrator.snapshot()

# 2️⃣ Global limit – how many agents may run overall?

global_limit = state.max_concurrent_agents || SymphonyElixir.Config.settings!().agent.max_concurrent_agents
IO.puts("Global max agents: #{global_limit}")
IO.puts("Currently running: #{map_size(state.running)}")
IO.puts("Global slots available: #{SymphonyElixir.Orchestrator.available_slots(state)}")

# 3️⃣ Per‑state limit – e.g. for the state of an issue we are about to dispatch

issue = %SymphonyElixir.Linear.Issue{
  id: "abc-123",
  state: "in progress",
  identifier: "PROJ‑42"
}
state_slots = SymphonyElixir.Orchestrator.state_slots_available?(issue, state.running)
IO.puts("State‑specific slots available: #{state_slots}")

# 4️⃣ Per‑host limit – assuming SSH workers are configured

worker_slots = SymphonyElixir.Orchestrator.worker_slots_available?(state)
IO.puts("Worker‑host slots available: #{worker_slots}")

# 5️⃣ Full dispatch decision (mirrors the real orchestrator logic)

can_dispatch? = SymphonyElixir.Orchestrator.should_dispatch_issue?(
  issue,
  state,
  SymphonyElixir.Orchestrator.active_state_set(),
  SymphonyElixir.Orchestrator.terminal_state_set()
)

IO.puts("Can the orchestrator dispatch this issue now? #{can_dispatch?}")

This retrieves a runtime snapshot via snapshot/0, displays the global concurrency window, evaluates a mock issue against per-state restrictions, checks worker-host capacity, and executes the composite predicate used in production dispatch decisions.

Key Implementation Files

• elixir/lib/symphony_elixir/orchestrator.ex — Contains the polling loop, limit-checking logic (available_slots/1, state_slots_available?/2, worker_host_slots_available?/2), and the dispatch decision tree.
• elixir/lib/symphony_elixir/config.ex — Provides runtime configuration access for agent.max_concurrent_agents, agent.max_concurrent_agents_per_state, and worker.max_concurrent_agents_per_host.
• elixir/lib/symphony_elixir/agent_runner.ex — Handles actual agent process startup after orchestrator approval.
• elixir/lib/symphony_elixir/workflow.ex — Defines issue-state sets (active_states, terminal_states) used to filter dispatch candidates.
• elixir/lib/symphony_elixir/tracker.ex — Supplies the candidate issue list evaluated against concurrency limits.

Summary

• The Symphony orchestrator enforces three concurrent limits: global, per-state, and per-host.
• Dispatch decisions require passing all three checks via should_dispatch_issue?/5.
• Limits are dynamically reloadable through refresh_runtime_config/1 without system restarts.
• Configuration originates from Config.settings!() and applies to the Elixir runtime in orchestrator.ex.
• Failed checks defer dispatch to subsequent polling cycles, preventing resource overcommitment.

Frequently Asked Questions

How does the Symphony orchestrator prevent exceeding concurrent agent limits?

The orchestrator evaluates every potential dispatch through the should_dispatch_issue?/5 function, which sequentially checks global slots (available_slots/1), per-state capacity (state_slots_available?/2), and per-host availability (worker_host_slots_available?/2). If any layer reports insufficient capacity, the dispatch aborts immediately.

Can concurrent agent limits be changed without restarting the Symphony orchestrator?

Yes. The orchestrator reloads configuration values on every poll tick via refresh_runtime_config/1, reading updated thresholds from Config.settings!(). This allows runtime adjustment of max_concurrent_agents and related parameters while agents continue running.

What happens when a concurrency limit is reached during dispatch?

When any limit check fails within should_dispatch_issue?/5, the orchestrator skips the current issue and leaves it in the candidate queue. The issue remains eligible for dispatch in subsequent polling cycles once slots become available through agent completions.

Where are the concurrent agent limits configured in the Symphony codebase?

Limits are defined in the application configuration and accessed through elixir/lib/symphony_elixir/config.ex. Specifically, agent.max_concurrent_agents controls global limits, agent.max_concurrent_agents_per_state governs per-state restrictions, and worker.max_concurrent_agents_per_host manages per-host SSH worker concurrency.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}