# How the Symphony Orchestrator Manages Concurrent Agent Limits

> Discover how the Symphony orchestrator manages concurrent agent limits using global, per-state, and per-host controls to prevent resource exhaustion. Learn about efficient issue dispatching.

- Repository: [OpenAI/symphony](https://github.com/openai/symphony)
- Tags: internals
- Published: 2026-05-08

---

**TLDR:** The Symphony orchestrator prevents resource exhaustion by enforcing three independent limits—global, per-state, and per-host—evaluating available slots through the `should_dispatch_issue?/5` function before dispatching any agent to handle Linear issues.

The OpenAI Symphony project provides an Elixir-based runtime that coordinates AI agents across Linear issue workflows. To maintain system stability, the **Symphony orchestrator concurrent agent limits** implementation applies a hierarchical gating mechanism that checks every potential dispatch against configurable thresholds defined in `Config.settings!()`.

## Three-Tier Concurrent Agent Limits

The orchestrator applies three distinct boundaries that operate simultaneously at different scopes.

### Global Concurrent Agents

The **global limit** controls the total number of agents running across the entire orchestrator instance. This value populates `State.max_concurrent_agents` from `Config.settings!().agent.max_concurrent_agents`. The function `available_slots/1` calculates remaining capacity by subtracting the current running agent count from this global maximum.

### Per-State Concurrent Agents

The **per-state limit** restricts concurrency for issues sharing the same Linear state (e.g., "In Progress" or "Review"). Implemented in `state_slots_available?/2`, this check uses `Config.max_concurrent_agents_for_state/1` to retrieve the limit, then compares it against the count from `running_issue_count_for_state/2`. This prevents any single workflow state from consuming all available resources.

### Per-Host Concurrent Agents

When SSH workers are configured via `Config.settings!().worker.ssh_hosts`, the **per-host limit** ensures individual worker machines do not become bottlenecks. The function `worker_host_slots_available?/2` utilizes `Config.settings!().worker.max_concurrent_agents_per_host` and counts matching entries via `running_worker_host_count/2` to enforce capacity per specific host.

## Dispatch Decision Logic for Agent Concurrency

Within [`orchestrator.ex`](https://github.com/openai/symphony/blob/main/orchestrator.ex), the `dispatch_issue/2` function gates every potential agent startup through `should_dispatch_issue?/5` (approximately lines 55-66). The orchestrator evaluates limits in strict sequence:

1. **Global slot check** — Verifies `available_slots(state) > 0`
2. **State-specific check** — Confirms `state_slots_available?(issue, state.running)` returns true
3. **Worker-host check** — Validates `worker_slots_available?(state)` or `worker_slots_available?(state, preferred_host)` when hosts are specified

If any limit has been reached, the orchestrator skips dispatch for that issue, leaving it in the candidate queue for the next polling cycle.

## Runtime Configuration and Dynamic Limits

The limits are **dynamic** rather than static constants. On every tick, `refresh_runtime_config/1` (lines 96-103) reloads configuration values, allowing administrators to adjust `max_concurrent_agents` thresholds without restarting the orchestrator. This ensures runtime adaptation to changing load conditions or infrastructure capacity.

## Inspecting Concurrency Limits at Runtime

You can verify the current concurrency state interactively in an IEx session. The following snippet demonstrates how to check each limit tier using actual orchestrator functions:

```elixir

# 1️⃣ Load the current orchestrator state (assumes the orchestrator is already started)

{:ok, state} = SymphonyElixir.Orchestrator.snapshot()

# 2️⃣ Global limit – how many agents may run overall?

global_limit = state.max_concurrent_agents || SymphonyElixir.Config.settings!().agent.max_concurrent_agents
IO.puts("Global max agents: #{global_limit}")
IO.puts("Currently running: #{map_size(state.running)}")
IO.puts("Global slots available: #{SymphonyElixir.Orchestrator.available_slots(state)}")

# 3️⃣ Per‑state limit – e.g. for the state of an issue we are about to dispatch

issue = %SymphonyElixir.Linear.Issue{
  id: "abc-123",
  state: "in progress",
  identifier: "PROJ‑42"
}
state_slots = SymphonyElixir.Orchestrator.state_slots_available?(issue, state.running)
IO.puts("State‑specific slots available: #{state_slots}")

# 4️⃣ Per‑host limit – assuming SSH workers are configured

worker_slots = SymphonyElixir.Orchestrator.worker_slots_available?(state)
IO.puts("Worker‑host slots available: #{worker_slots}")

# 5️⃣ Full dispatch decision (mirrors the real orchestrator logic)

can_dispatch? = SymphonyElixir.Orchestrator.should_dispatch_issue?(
  issue,
  state,
  SymphonyElixir.Orchestrator.active_state_set(),
  SymphonyElixir.Orchestrator.terminal_state_set()
)

IO.puts("Can the orchestrator dispatch this issue now? #{can_dispatch?}")

```

This retrieves a runtime snapshot via `snapshot/0`, displays the global concurrency window, evaluates a mock issue against per-state restrictions, checks worker-host capacity, and executes the composite predicate used in production dispatch decisions.

## Key Implementation Files

- **[`elixir/lib/symphony_elixir/orchestrator.ex`](https://github.com/openai/symphony/blob/main/elixir/lib/symphony_elixir/orchestrator.ex)** — Contains the polling loop, limit-checking logic (`available_slots/1`, `state_slots_available?/2`, `worker_host_slots_available?/2`), and the dispatch decision tree.
- **[`elixir/lib/symphony_elixir/config.ex`](https://github.com/openai/symphony/blob/main/elixir/lib/symphony_elixir/config.ex)** — Provides runtime configuration access for `agent.max_concurrent_agents`, `agent.max_concurrent_agents_per_state`, and `worker.max_concurrent_agents_per_host`.
- **[`elixir/lib/symphony_elixir/agent_runner.ex`](https://github.com/openai/symphony/blob/main/elixir/lib/symphony_elixir/agent_runner.ex)** — Handles actual agent process startup after orchestrator approval.
- **[`elixir/lib/symphony_elixir/workflow.ex`](https://github.com/openai/symphony/blob/main/elixir/lib/symphony_elixir/workflow.ex)** — Defines issue-state sets (`active_states`, `terminal_states`) used to filter dispatch candidates.
- **[`elixir/lib/symphony_elixir/tracker.ex`](https://github.com/openai/symphony/blob/main/elixir/lib/symphony_elixir/tracker.ex)** — Supplies the candidate issue list evaluated against concurrency limits.

## Summary

- The Symphony orchestrator enforces **three concurrent limits**: global, per-state, and per-host.
- Dispatch decisions require passing **all three checks** via `should_dispatch_issue?/5`.
- Limits are **dynamically reloadable** through `refresh_runtime_config/1` without system restarts.
- Configuration originates from `Config.settings!()` and applies to the Elixir runtime in [`orchestrator.ex`](https://github.com/openai/symphony/blob/main/orchestrator.ex).
- Failed checks defer dispatch to subsequent polling cycles, preventing resource overcommitment.

## Frequently Asked Questions

### How does the Symphony orchestrator prevent exceeding concurrent agent limits?

The orchestrator evaluates every potential dispatch through the `should_dispatch_issue?/5` function, which sequentially checks global slots (`available_slots/1`), per-state capacity (`state_slots_available?/2`), and per-host availability (`worker_host_slots_available?/2`). If any layer reports insufficient capacity, the dispatch aborts immediately.

### Can concurrent agent limits be changed without restarting the Symphony orchestrator?

Yes. The orchestrator reloads configuration values on every poll tick via `refresh_runtime_config/1`, reading updated thresholds from `Config.settings!()`. This allows runtime adjustment of `max_concurrent_agents` and related parameters while agents continue running.

### What happens when a concurrency limit is reached during dispatch?

When any limit check fails within `should_dispatch_issue?/5`, the orchestrator skips the current issue and leaves it in the candidate queue. The issue remains eligible for dispatch in subsequent polling cycles once slots become available through agent completions.

### Where are the concurrent agent limits configured in the Symphony codebase?

Limits are defined in the application configuration and accessed through [`elixir/lib/symphony_elixir/config.ex`](https://github.com/openai/symphony/blob/main/elixir/lib/symphony_elixir/config.ex). Specifically, `agent.max_concurrent_agents` controls global limits, `agent.max_concurrent_agents_per_state` governs per-state restrictions, and `worker.max_concurrent_agents_per_host` manages per-host SSH worker concurrency.