How the Archon Workflow DAG Executor Handles Concurrent Node Execution

The Archon workflow DAG executor processes independent workflow nodes in parallel by organizing them into topological layers and executing each layer concurrently using Promise.allSettled, ensuring dependencies are respected while maximizing throughput.

Archon is an open-source workflow automation framework that orchestrates complex AI-driven pipelines as directed acyclic graphs (DAGs). The executor's concurrency model allows independent nodes to run simultaneously while maintaining strict dependency ordering. Understanding how the Archon workflow DAG executor manages parallel execution helps developers optimize performance and resource utilization in production environments.

Building Topological Execution Layers

Before any concurrent execution begins, the Archon workflow DAG executor constructs a valid execution order by building topological layers. In packages/workflows/src/dag-executor.ts (lines 665-698), the buildTopologicalLayers function analyzes the dependency graph to group nodes into layers where each node in a layer has all its dependencies satisfied by nodes in previous layers.

This layering guarantees that the executor never runs a node before its inputs are ready. Nodes with no dependencies occupy layer 0, nodes depending only on layer 0 occupy layer 1, and so on. The executor then processes these layers sequentially while running all nodes within a single layer concurrently.

Concurrent Execution Within Layers

The core concurrency mechanism resides in the layer processing logic found at lines 25-27 of dag-executor.ts. For each topological layer, the executor launches all nodes simultaneously using Promise.allSettled, waiting for every node in the layer to complete before advancing to the next layer.

Detecting Parallel vs. Sequential Layers

The executor dynamically determines whether a layer requires parallel processing through a simple length check at lines 19-22. If layer.length > 1, the executor treats the layer as parallel; otherwise, it processes the single node sequentially. This optimization avoids unnecessary overhead for sequential chains while maximizing throughput for independent operations.

Session Isolation for Concurrent Nodes

Critical to the Archon workflow DAG executor's design is how it handles AI sessions during parallel execution. At lines 21-24, parallel layers always trigger lastSequentialSessionId = undefined, forcing each concurrent node to receive a fresh AI session. This isolation prevents session state corruption when multiple nodes simultaneously interact with language models.

For sequential layers (lines 38-41), the executor reuses the session from the previous node unless the node explicitly requests a fresh context via node.context === 'fresh'. This hybrid approach balances performance (session reuse) with correctness (isolation for concurrent operations).

Result Aggregation and Failure Handling

After Promise.allSettled resolves, the executor iterates through results at lines 46-56, storing each output in a Map<string, NodeOutput> via nodeOutputs.set(nodeId, output). Each node promise internally catches its own errors, logs them, creates a node_failed event, and returns a structured failure result. This design ensures the outer Promise.allSettled never rejects, allowing the executor to complete the entire layer before determining whether workflow-wide failure handling is necessary.

The executor tracks layer-wide failures using a layerHadFailure boolean, which influences whether subsequent layers should execute based on the workflow run's configured failure tolerance.

Retry Logic and Cancellation Support

The Archon workflow DAG executor implements sophisticated retry and lifecycle management that operates independently within each concurrent node.

Independent Retry Mechanisms

For nodes configured with retries, the executor implements a retry loop inside the per-node promise (lines 51-86 of dag-executor.ts). Each concurrently running node maintains its own retry counter (up to maxRetries), allowing one node to retry its operation without blocking or affecting other nodes in the same parallel layer. This granular approach ensures that temporary failures in one branch do not delay independent operations.

Workflow Cancellation and Pause Checks

Between layer executions, the executor checks the workflow run's database status at lines 75-80. If the run status changes from running (e.g., cancelled or paused by a user), the executor aborts further layer processing immediately. This checkpointing between layers ensures that cancellation requests are honored at natural synchronization points without interrupting mid-flight node operations.

Practical Example: Parallel Node Configuration

Consider a workflow where two independent data gathering operations must complete before a merge step:

# .archon/workflows/example-parallel.yaml

name: example-parallel
nodes:
  - id: fetch-info
    command: fetch-info
  - id: list-files
    command: list-files
  - id: merge-results
    depends_on: [fetch-info, list-files]
    prompt: |
      Combine the results from {{ $fetch-info.output }} and {{ $list-files.output }}.

When executed:

1. buildTopologicalLayers creates two layers: Layer 0 (fetch-info, list-files) and Layer 1 (merge-results).
2. The Archon workflow DAG executor calls Promise.allSettled on Layer 0, launching both nodes simultaneously with separate AI sessions.
3. After both nodes settle successfully, their outputs populate the nodeOutputs Map.
4. The sequential Layer 1 node accesses both $fetch-info.output and $list-files.output for template substitution.

Summary

• The Archon workflow DAG executor organizes workflows into topological layers where all nodes in a layer have satisfied dependencies.
• Layers containing multiple nodes execute concurrently via Promise.allSettled, while single-node layers run sequentially.
• Parallel nodes always receive fresh AI sessions to prevent state corruption, whereas sequential nodes reuse sessions unless explicitly configured otherwise.
• Retry logic operates independently within each node's promise, allowing granular failure recovery without blocking parallel siblings.
• Cancellation checks occur between layers, ensuring workflows can be safely paused or stopped at synchronization points.

Frequently Asked Questions

How does the Archon workflow DAG executor determine which nodes can run in parallel?

The executor builds topological layers where nodes are grouped by dependency completion. Nodes sharing the same layer have no interdependencies, allowing the executor to launch them simultaneously using Promise.allSettled in packages/workflows/src/dag-executor.ts (lines 25-27).

What happens to AI sessions when nodes run concurrently?

Parallel nodes always receive fresh AI sessions (lastSequentialSessionId = undefined) as implemented at lines 21-24 of dag-executor.ts. This isolation prevents concurrent nodes from corrupting shared session state, while sequential nodes reuse sessions for efficiency unless node.context === 'fresh'.

Can individual nodes retry without blocking other concurrent nodes?

Yes. The executor implements retry loops inside individual node promises (lines 51-86), allowing each node to retry up to maxRetries times independently. Other nodes in the same parallel layer continue execution unaffected by a sibling node's retry cycle.

How does the executor handle workflow cancellation during parallel execution?

The executor checks the workflow run's database status between layers (lines 75-80). If the status changes from running, the executor stops processing subsequent layers. This design allows current parallel nodes to complete while preventing new layers from starting.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}