What Is the Purpose of the SwarmForge Window Watchdog?

The SwarmForge window watchdog is a background process that guarantees the continuity of agent terminal windows by detecting disappeared sessions, automatically recreating them after a configurable threshold, and triggering a full swarm shutdown if the cleanup window itself is lost.

The unclebob/swarm-forge repository implements this watchdog to make long-running AI agent swarms resilient to terminal-backend quirks. By continuously monitoring the window-state file and verifying tmux session existence, the watchdog eliminates the need for manual intervention when windows vanish unexpectedly.

Core Responsibilities of the Window Watchdog

The watchdog operates as a persistence layer between SwarmForge’s agent orchestration and the underlying terminal multiplexer. Its implementation in swarmforge/scripts/swarm-window-watchdog.bb follows a strict monitor-detect-recover-escalate lifecycle.

Monitoring Window State

When SwarmForge initializes, each visible agent window is attached to a dedicated tmux session. The watchdog continuously reads the window-state file—a TSV (tab-separated values) format that records each window’s index, stable ID, tmux session name, and title.

According to the source code, the state file structure is parsed at startup to build an internal registry of tracked windows:

;; From swarmforge/scripts/swarm-window-watchdog.bb
;; Lines 13-20: Parsing the window-state TSV
(let [windows (mapv (fn [line]
                      (let [[idx id session title] (str/split line #"\t")]
                        {:index idx
                         :stable-id id
                         :session session
                         :title title}))
                    (str/split-lines (slurp window-state-file)))]
  ...)

Detecting Missing Windows

For every tracked window except the special cleanup window, the watchdog verifies existence by invoking the terminal_window_exists function. This check queries the terminal backend to confirm whether the tmux session and window ID combination is still valid.

The detection logic uses a consecutive failure counter rather than immediate action to avoid false positives during transient backend delays:

;; Lines 90-101: Existence check implementation
(if (terminal_window_exists backend tmux-socket session stable-id)
  (assoc window :missing-count 0)
  (update window :missing-count inc))

Automatic Recovery

Once a window’s missing-count reaches the missing-threshold (default value of 3 consecutive checks), the watchdog initiates recovery. It attempts to recreate the window via terminal_open_session, then atomically updates both the state file and the accompanying window-ids file with the new stable ID.

The recovery sequence preserves the agent’s terminal history by reattaching to the same tmux session name while generating a new stable identifier:

;; Lines 101-107: Window recreation logic
(let [new-window-id (terminal-out script-dir working-dir tmux-socket backend
                                  "terminal_open_session" session title cleanup-window-id)]
  (when-not (str/blank? new-window-id)
    (rewrite-window-id! window-state-file window-ids-file index new-window-id)))

Fatal Shutdown Condition

The cleanup window receives special treatment. If this primary control window cannot be located for the same missing-threshold duration, the watchdog treats the condition as fatal. It immediately tears down all tmux sessions and tracked windows, terminating the entire swarm to prevent orphaned processes and inconsistent state.

This escalation path is implemented in lines 112-119 of the watchdog script, ensuring that a lost control plane never leaves agent processes running invisibly.

Running and Configuring the Watchdog

The watchdog is typically launched automatically by swarmforge.bb, but it can be invoked manually for testing or debugging specific window states.

Manual Invocation

To test the window ID rewriting logic without starting the full monitoring loop:

bb swarmforge/scripts/swarm-window-watchdog.bb \
  windows.tsv window-ids 2 new-b \
  --rewrite-window-id windows.tsv window-ids 2 new-b

Background Process Integration

The main SwarmForge orchestrator starts the watchdog as a background process, redirecting logs to the state directory:

;; From swarmforge.bb
(process/process [(str (fs/path (:script-dir ctx) "swarm-window-watchdog.sh"))
                  (:window-state-file ctx)
                  (:window-ids-file ctx)
                  cleanup-owner-index
                  (:tmux-socket ctx)
                  (:working-dir ctx)
                  backend]))

Key Configuration Parameters

• missing-threshold: Number of consecutive failed existence checks before triggering recovery (default: 3).
• window-state-file: Path to the TSV tracking window metadata.
• window-ids-file: Path to the companion file mapping indices to stable IDs.
• cleanup-owner-index: Integer identifier for the control window that, if lost, triggers total shutdown.

Summary

• The SwarmForge window watchdog maintains agent terminal continuity by monitoring tmux session health through the window-state TSV file.
• It detects disappeared windows via the terminal_window_exists check and tracks consecutive failures against a configurable missing-threshold.
• It recovers automatically by invoking terminal_open_session and rewriting stable IDs in both state files, preserving session history.
• It escalates to full shutdown when the cleanup window is lost, preventing orphaned tmux sessions and ensuring system consistency.
• The implementation resides primarily in swarmforge/scripts/swarm-window-watchdog.bb and is invoked by swarmforge.bb during swarm initialization.

Frequently Asked Questions

What triggers the SwarmForge window watchdog to recreate a window?

The watchdog recreates a window when terminal_window_exists returns false for three consecutive monitoring cycles (the default missing-threshold). This threshold prevents premature recreation during brief terminal-backend hiccups while ensuring actual disappeared windows are restored promptly.

How does the watchdog handle the cleanup window differently from agent windows?

While standard agent windows trigger automatic recreation when missing, the cleanup window triggers a fatal shutdown of the entire swarm. According to the source code in swarm-window-watchdog.bb lines 112-119, losing the cleanup window indicates a catastrophic terminal backend failure, prompting the watchdog to terminate all tmux sessions rather than attempt recovery.

Can I run the window watchdog independently for testing?

Yes. You can invoke the Babashka script directly to test specific functions like --rewrite-window-id without starting the monitoring loop. The repository includes test coverage in test/swarmforge/script_test.clj that verifies the ID rewriting logic using this manual invocation pattern.

Where does the watchdog log its activity?

When launched by swarmforge.bb, the watchdog writes its operational log to <state-dir>/window-watchdog.log. This log captures detection events, recreation attempts, and any fatal shutdown sequences for debugging swarm stability issues.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}