How the SwarmForge Handoff File Directory Structure Works: Outbox, Inbox, and State Management

SwarmForge uses an atomic file-based message queue inside .swarmforge/handoffs/ where agents write drafts to an outbox and consume tasks from an inbox, while a Babashka daemon (handoffd.bb) manages all transport and state transitions between roles.

The SwarmForge handoff file directory structure implements a durable, git-aware message queue that enables AI agents to exchange work without direct socket or process interaction. In the unclebob/swarm-forge repository, this design splits communication into isolated outbound and inbound areas under each role's worktree, ensuring crash safety through atomic file operations and providing complete audit trails for debugging.

Directory Structure Overview

SwarmForge organizes all handoff state under .swarmforge/handoffs/ within each role's worktree. The structure separates outbound drafts from inbound tasks and maintains archive directories for completed or failed operations.

Outbox Directories (Outbound Queue)

The outbox handles all draft handoffs created by the local agent before they are delivered to recipients:

• .swarmforge/handoffs/outbox/ – The main staging area where validated handoff files land after atomic rename operations. The handoffd.bb daemon watches this folder exclusively for new work.

• .swarmforge/handoffs/outbox/tmp/ – A transient staging directory used only by swarm_handoff.sh. The script writes temporary files here first, flushes them to disk, then performs an atomic rename into the parent outbox/ directory to prevent partial reads during write operations.

• .swarmforge/handoffs/sent/ – The archive location where the daemon moves successfully delivered handoffs. This provides an audit trail of all outbound communications.

• .swarmforge/handoffs/failed/ – The error directory where the daemon stores handoffs that failed validation or delivery, along with diagnostic metadata explaining the failure.

Inbox Directories (Inbound Queue)

The inbox manages the task lifecycle for work received from other roles:

• .swarmforge/handoffs/inbox/new/ – The primary task queue containing handoffs ready for processing. Files here are sorted by filename (encoding priority and timestamp) to ensure deterministic ordering.

• .swarmforge/handoffs/inbox/in_process/ – Holds the single handoff currently being executed by the agent. The ready_for_next_task.sh helper atomically moves files from new/ to this directory before work begins.

• .swarmforge/handoffs/inbox/completed/ – The final archive for handoffs that have finished processing, moved here by done_with_current.sh to maintain a complete history of executed tasks.

The Handoff Lifecycle

The SwarmForge handoff directory structure supports a four-phase lifecycle that ensures durability at every transition.

Step 1: Creating and Validating Drafts

When an agent initiates a handoff, it runs swarm_handoff.sh <draft-file>. According to swarmforge/handoff-protocol.md (lines 48-53), the script performs header validation, generates an immutable id, adds timestamps, and creates a canonical commit abbreviation for git_handoff types.

The script writes the validated content to outbox/tmp/<name>.tmp, ensures the data is flushed to disk, then atomically renames the file to outbox/<name>.handoff. This atomic rename guarantees that the daemon never sees a partially written file.

Step 2: Daemon Processing and Delivery

The handoffd.bb daemon polls each role's outbox/ directory, explicitly ignoring any files in outbox/tmp/. When it discovers a completed .handoff file, it validates the content again, then copies the handoff into each recipient's inbox/new/ directory.

During delivery, the daemon appends a recipient header and an enqueued_at timestamp to track provenance. After successful copy operations, the daemon sends a generic tmux wake-up message ("You have new handoff mail…") to notify the recipient's terminal. Finally, the original outbox file is moved to sent/ (or failed/ if errors occurred), as documented in swarmforge/handoff-protocol.md (lines 17-27).

Step 3: Task Consumption and Processing

To retrieve work, an agent executes ready_for_next.sh, which reads the role's receive mode (task or batch) from .swarmforge/roles.tsv and dispatches to the appropriate helper. In task mode, ready_for_next_task.sh selects the first file in inbox/new/ (sorted lexicographically), atomically moves it to inbox/in_process/, adds a dequeued_at header, and prints the task details to stdout. If no files exist, it outputs NO_TASK.

Step 4: Completion and Archiving

After processing the task, the agent runs done_with_current.sh (or done_with_current_task.sh for task mode). The script verifies the presence of a file in inbox/in_process/, performs any final cleanup or follow-up handoffs, then moves the completed handoff to inbox/completed/ for audit purposes.

Atomic Operations and Crash Safety

The SwarmForge handoff file directory structure relies on atomic filesystem operations to maintain consistency during crashes. All write operations use the write-to-temp-then-rename pattern, ensuring that the daemon and agent helpers never encounter corrupted or partial files.

Because all state persists as regular files in .swarmforge/handoffs/, a system crash only loses in-memory data, not the queue itself. Upon restart, the daemon resumes polling where it left off, and agents can recover their current task from inbox/in_process/ without data loss.

Summary

• SwarmForge handoff files live under .swarmforge/handoffs/ in each role's worktree, split between outbox/ (outbound) and inbox/ (inbound) areas.
• Atomic file operations via outbox/tmp/ ensure the daemon never processes partial handoffs during write operations.
• The handoffd.bb daemon owns all transport logic, copying files from sender outboxes to recipient inbox/new/ directories and archiving to sent/ or failed/.
• Task progression follows a strict path: inbox/new/ → inbox/in_process/ → inbox/completed/, managed by ready_for_next.sh and done_with_current.sh.
• Durability and auditing are built-in through persistent file storage and archive directories that maintain complete history of all handoff attempts.

Frequently Asked Questions

What happens if the daemon crashes during handoff delivery?

If handoffd.bb crashes while copying a handoff to a recipient's inbox, the original file remains in the sender's outbox/ until the daemon restarts and retries. Because the daemon moves the original to sent/ only after confirming successful delivery to all recipients, no handoffs are lost during crashes—though duplicate delivery to some recipients may occur if the crash happened mid-copy, which the receiving agent handles idempotently.

How does SwarmForge prevent duplicate task processing?

SwarmForge prevents duplicates through atomic moves and single-threaded consumption. The ready_for_next_task.sh helper atomically moves a file from inbox/new/ to inbox/in_process/ using filesystem atomicity guarantees. Since only one process can succeed in moving a specific filename, concurrent agents attempting to grab the same task will result in only one successful claim while others receive NO_TASK or the next available item.

Can multiple agents read from the same inbox simultaneously?

While multiple processes could theoretically poll the same inbox/new/ directory, SwarmForge is designed for single-agent-per-role consumption. The atomic move operation ensures that even if multiple agents attempt to pick up work simultaneously, only one will successfully acquire any given handoff file. For distributed scenarios, each role should maintain its own worktree with isolated .swarmforge/handoffs/ directories.

Where are failed handoffs stored for debugging?

Failed handoffs are stored in .swarmforge/handoffs/failed/ with diagnostic metadata attached. When the handoffd.bb daemon encounters validation errors or delivery failures (such as unreachable recipient directories), it moves the problematic handoff file from outbox/ to failed/ and appends error details to aid troubleshooting. This location serves as a dead-letter queue for inspecting malformed handoffs or system configuration issues.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}