How to Debug Handoff Delivery Failures or Missing Messages in SwarmForge

To debug handoff delivery failures in SwarmForge, inspect the daemon log at .swarmforge/daemon/handoffd.log, check the failed directory for .error files, and validate handoff headers using handoff_lib.bb.

SwarmForge orchestrates multi-agent workflows by moving work between roles via handoff files managed by a dedicated daemon. When messages vanish or delivery fails in the unclebob/swarm-forge repository, systematic debugging requires tracing the pipeline from the outbox through the inbox to the final tmux notification.

Understanding the Handoff Architecture

SwarmForge uses a file-based messaging system where the handoffd.bb daemon watches every role’s outbox for new .handoff files. The daemon validates each draft, copies it into each recipient’s inbox → new directory, and sends a tmux wake-up message. Delivery failures can occur during validation, filesystem operations, or the notification phase.

Three Diagnostic Layers for Debugging Handoffs

Trace failures through these three layers to isolate the root cause.

Layer 1: Daemon Logging

The daemon writes a chronological log to .swarmforge/daemon/handoffd.log. Look for entries prefixed with delivered, error, or failed.

In swarmforge/scripts/handoffd.bb, the log! function writes timestamped entries to log-file at lines 34–38. A successful delivery logs delivered <path>, while failures log specific error codes.

Layer 2: Failure Archives

When the daemon cannot deliver a handoff, it moves the draft to the sender’s failed directory and creates a companion .error file containing the reason.

The fail! function in handoffd.bb (lines 17–22) records the failure reason, writes the .error file, and archives the handoff. Common triggers include missing headers, unknown recipients, or filesystem collisions.

Layer 3: Handoff Content Validation

Verify that the handoff file contains all required headers: type, to, priority, and task or commit.

Use swarmforge/scripts/handoff_lib.bb to inspect files. The header-field function (lines 68–74) extracts specific headers, while print-task (lines 107–116) pretty-prints the full task representation for manual verification.

Step-by-Step Debug Workflow

Follow these steps to diagnose delivery failures systematically.

1. Check the Daemon Log

View the last 20 lines of the daemon log to identify recent errors or successful deliveries.

cat .swarmforge/daemon/handoffd.log | tail -n 20

Look for error <path> or failed <path> entries. If you see delivered but the recipient lacks the file, the failure occurred during the notify! step.

2. Inspect the Failed Directory

List failed handoffs and read their error descriptions.

ls .swarmforge/handoffs/<sender>/failed
cat .swarmforge/handoffs/<sender>/failed/<file>.error

The .error file explains why fail! aborted delivery, such as "missing to header" or invalid priority values.

3. Validate Handoff Headers Manually

Use the helper library to inspect the handoff content before it enters the outbox.

# Print full task representation

bb swarmforge/scripts/handoff_lib.bb print-task .swarmforge/handoffs/<sender>/outbox/tmp/<file>.handoff

# Extract specific header (e.g., "to")

bb swarmforge/scripts/handoff_lib.bb header-field .swarmforge/handoffs/<sender>/outbox/tmp/<file>.handoff to

Ensure the to list contains only role names defined in roles.tsv.

4. Confirm Recipient Inbox State

Verify whether the file reached the destination inbox.

ls .swarmforge/handoffs/<recipient>/inbox/new

If the file is missing despite a delivered log entry, the copy succeeded but the tmux notification likely failed.

5. Test Tmux Notification Manually

The notify! function (lines 94–99) sends three tmux commands to wake the recipient session. Test these manually to detect socket or session errors.

tmux -S .swarmforge/tmux-socket send-keys -t <session> -l "You have new handoff mail. If idle, run ready_for_next.sh."
tmux -S .swarmforge/tmux-socket send-keys -t <session> C-m
tmux -S .swarmforge/tmux-socket send-keys -t <session> C-j

Non-zero exit statuses appear in the daemon log (lines 100–105).

6. Verify Role Configuration

The daemon loads role metadata via load-roles (lines 44–56) from .swarmforge/roles.tsv. Typos in role names, missing worktree paths, or incorrect receive modes cause silent delivery aborts.

cat .swarmforge/roles.tsv

7. Check Sequence Handling

Duplicate filenames occur when agents generate handoffs within the same second. The next-sequence function in handoff_lib.bb (lines 38–57) uses a lock directory to ensure uniqueness. If the lock fails, move-with-collision in handoffd.bb (lines 7–15) resolves conflicts by renaming files, potentially causing delivery delays.

Inspect the failed directory for "collision" errors if you suspect timing issues.

Common Pitfalls and Fixes

Symptom	Likely Cause	Fix
"missing to header" in .error file	Draft omitted the to: line or used incorrect capitalization.	Add to: <role_name> with valid role names from roles.tsv.
No delivered entry but error shown	Invalid priority (must be two digits) or malformed type.	Ensure priority: 00–99 and type: git_handoff or type: note.
Recipient never sees file despite delivered log	Incorrect tmux session name or missing socket file.	Verify roles.tsv session column matches active tmux sessions; check .swarmforge/tmux-socket exists.
Duplicate filename collisions	Multiple agents created handoffs simultaneously and lock directory failed.	Restart the daemon by killing handoffd.pid and rerunning ./swarm.
Handoff missing from inbox/new	Role configured for batch receive mode but agent runs ready_for_next.sh expecting single tasks.	Use ready_for_next_batch.sh for batch mode; see handoff-protocol.md lines 44–58 for mode definitions.

Quick Diagnostic Script

Run this script from the project root to automate the initial investigation.

#!/usr/bin/env bash
set -euo pipefail

echo "=== Daemon Log (last 20 lines) ==="
tail -n 20 .swarmforge/daemon/handoffd.log

if ls .swarmforge/handoffs/*/failed/*.handoff 2>/dev/null | grep -q .; then
  echo; echo "=== Failed Handoffs ==="
  for f in .swarmforge/handoffs/*/failed/*.handoff; do
    echo "- $(basename "$f")"
    cat "${f}.error"
    echo
  done
fi

OUT=$(ls .swarmforge/handoffs/*/outbox/*.handoff 2>/dev/null | head -n 1 || true)
if [[ -n "$OUT" ]]; then
  echo "=== Sample Outbox Handoff ==="
  bb swarmforge/scripts/handoff_lib.bb print-task "$OUT"
fi

Save as debug_handoff.sh, make it executable (chmod +x debug_handoff.sh), and execute while the swarm is active.

Summary

• Monitor .swarmforge/daemon/handoffd.log for delivered, error, or failed entries generated by the log! function.
• Inspect the failed directory and companion .error files created by fail! to understand validation failures.
• Validate handoff headers using handoff_lib.bb functions header-field and print-task to ensure required fields (to, type, priority) are present.
• Test tmux notifications manually if logs show delivery but recipients report missing messages.
• Verify roles.tsv configuration matches active tmux sessions and correct receive modes (task vs batch).

Frequently Asked Questions

Why does the daemon log show "delivered" but the recipient never sees the file?

This indicates the file copy succeeded but the tmux notification failed. The notify! function (lines 94–99 in handoffd.bb) sends wake-up commands to the recipient's tmux session. If the session name in roles.tsv does not match the actual tmux session, or if the socket file .swarmforge/tmux-socket is missing, the recipient never receives the alert. Verify the session name and socket path, then test the tmux commands manually.

What causes the "missing to header" error in handoff files?

The daemon's validation requires a to: header specifying the recipient role. If the handoff draft omits this header, uses wrong capitalization, or lists non-existent roles, the fail! function (lines 17–22 in handoffd.bb) moves the file to the failed directory with a descriptive .error file. Always include to: <role_name> where the role exists in roles.tsv.

How do I fix duplicate filename collisions in the handoff system?

When multiple agents create handoffs within the same second, the next-sequence function in handoff_lib.bb (lines 38–57) uses a lock directory to generate unique filenames. If the lock fails due to filesystem issues or stale locks, filenames collide. The move-with-collision handler (lines 7–15 in handoffd.bb) resolves this by renaming files, which may delay delivery. Restart the daemon by removing handoffd.pid and relaunching ./swarm to clear stale locks.

Where does the daemon store error logs for failed deliveries?

Failed deliveries are not logged to a separate error log; instead, the daemon moves the undeliverable handoff file to .swarmforge/handoffs/<sender>/failed/ and creates a text file with the same name plus the .error extension. This file contains the specific reason for failure written by the fail! function. Check this directory when the main handoffd.log shows failed entries or when recipients report missing messages without corresponding delivered entries.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}