SwarmForge Agent Queue Management: Key Scripts and File‑Based Architecture Explained

SwarmForge manages AI agent workloads through six core Bash/Babashka scripts—swarm_handoff.sh, ready_for_next.sh, ready_for_next_batch.sh, done_with_current.sh, pack_web.sh, and handoff_lib.bb—that enqueue, dequeue, and complete hand‑offs via a persistent file‑based queue under .swarmforge/handoffs/.

SwarmForge is an open‑source framework for orchestrating AI agent teams. At its heart lies a lightweight agent queue management system that moves work items—called hand‑offs—through a project‑local directory structure. Unlike database‑backed queues, SwarmForge uses plain files, making the entire workflow version‑controllable and portable across POSIX environments.

The Hand‑Off Queue Lifecycle

The queue operates in four stages: inbox (pending), in‑process (active), outbox (completed), and archival. Scripts transition hand‑off files between these directories while a shared Babashka library enforces naming conventions and validation.

Stage	Directory	Purpose
Enqueue	.swarmforge/handoffs/inbox	New tasks await pickup
Dequeue	.swarmforge/handoffs/in-process	Agent actively working
Complete	.swarmforge/handoffs/outbox	Finished tasks with results

Core Agent Queue Management Scripts

swarm_handoff.sh: Enqueue New Tasks

Located at swarmforge/scripts/swarm_handoff.sh, this script creates hand‑off files in the inbox. It encodes priority, timestamp, and direction (e.g., coder → cleaner) directly into the filename for deterministic ordering.

The script delegates header construction to handoff_lib.bb, then writes the payload to .swarmforge/handoffs/inbox.

swarm_handoff.sh \
  --from coder \
  --to cleaner \
  --task "htw-console-app" \
  --priority normal \
  --body "Refactor the console entry point"

ready_for_next.sh: Dequeue Single Tasks

Invoked by an agent's tmux pane when idle, swarmforge/scripts/ready_for_next.sh scans the inbox for the earliest file matching the caller's role. It atomically moves the file to in‑process and returns the parsed payload.

ready_for_next.sh

# Output: {task: "htw-console-app", from: "specifier", to: "coder", ...}

ready_for_next_batch.sh: Dequeue Batched Work

For multi‑step specifications that must stay together, swarmforge/scripts/ready_for_next_batch.sh identifies batch marker files and promotes entire job groups to in‑process. This preserves ordering dependencies across related hand‑offs.

done_with_current.sh: Mark Completion

After finishing work, agents call swarmforge/scripts/done_with_current.sh. The script:

• Writes a completed_at timestamp to the hand‑off header
• Optionally appends result notes
• Moves the file to .swarmforge/handoffs/outbox

done_with_current.sh

# Hand‑off file relocated to outbox with completion metadata

pack_web.sh: Web UI Integration

The dashboard (/dashboard) reads inbox and outbox directories to render queue state. User actions—clicking "New Task"—trigger wrapped calls to swarm_handoff.sh, bridging human oversight with script‑driven queue operations.

handoff_lib.bb: Shared Queue Logic

swarmforge/scripts/handoff_lib.bb is the Babashka central library imported by all queue scripts. It implements:

• Filename encoding rules (timestamp + priority + direction)
• Header parsing and validation
• Timestamp generation and timezone handling

Why File‑Based Queues?

SwarmForge's agent queue management avoids external dependencies. The file system provides:

• Atomic moves for dequeue operations (no race conditions)
• Git traceability (hand‑off history persists in version control)
• Zero setup (works on any POSIX shell)

Complete Workflow Example

A typical hand‑off flows through all scripts:

1. Specifier queues work for coder:

swarm_handoff.sh --from specifier --to coder --task "api-design" --priority high

2. Coder's pane dequeues when ready:

ready_for_next.sh  # Atomically claims the task

3. Coder finishes and completes:

done_with_current.sh  # Archives with timestamp

4. Dashboard shows updated queue state via pack_web.sh reads.

Summary

• swarm_handoff.sh enqueues tasks to .swarmforge/handoffs/inbox with encoded priority and direction.
• ready_for_next.sh and ready_for_next_batch.sh dequeue to in‑process using role‑based matching.
• done_with_current.sh finalizes tasks to outbox with completion metadata.
• pack_web.sh exposes queue state through the web dashboard.
• handoff_lib.bb centralizes all queue logic as a reusable Babashka library.
• The file‑based design eliminates external queue dependencies while remaining race‑safe and version‑controllable.

Frequently Asked Questions

How does SwarmForge prevent race conditions when multiple agents dequeue?

The ready_for_next.sh script uses atomic file moves at the OS level. When an agent finds a matching hand‑off in the inbox, it attempts to move the file to in‑process in a single operation. Only one process succeeds; others retry with the next candidate.

Can hand‑offs have priorities beyond normal and high?

The handoff_lib.bb library encodes priority into the filename for lexical sorting. While swarm_handoff.sh exposes --priority as a parameter, the source code shows the sorting logic relies on prefix ordering—custom priority levels would require extending handoff_lib.bb.

What happens if an agent crashes while processing a hand‑off?

Files remain in in‑process until done_with_current.sh completes. An administrator can manually inspect .swarmforge/handoffs/in-process and restart or requeue stuck items. The dashboard via pack_web.sh surfaces these orphaned tasks.

Is the queue suitable for high‑throughput workloads?

SwarmForge's agent queue management targets small‑to‑medium agent teams (dozens of hand‑offs per minute). The file‑based design trades raw throughput for transparency and simplicity. For massive scale, an external message broker would replace these scripts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}