How the Window Watchdog Mechanism Recovers Closed Agent Windows in Swarm Forge

The window watchdog mechanism monitors agent windows via a Babashka script that tracks missing window counts and automatically respawns closed windows once a configurable threshold is reached.

The unclebob/swarm-forge repository implements a robust window watchdog mechanism to ensure agent windows remain available during long-running operations. This Clojure-based watchdog continuously validates window states against a persistent table and triggers recovery actions when sessions disappear unexpectedly.

How the Window Watchdog Reads Window State

The watchdog operates on a tab-separated window state file that maps indices to active terminal sessions. In swarmforge/scripts/swarm-window-watchdog.bb, the rows function loads window-state-file into a vector of maps containing :index, :window-id, :session, and :title keys.

This persistent table enables the script to track which tmux sessions should be visible in agent windows. The watchdog receives a cleanup-owner-index parameter that identifies which window owns the cleanup process, allowing it to validate the anchor point before checking dependent windows.

Validating the Cleanup Owner Window

Before scanning agent windows, the watchdog verifies the cleanup owner remains healthy. It retrieves the cleanup-row using the provided index and confirms the tmux session still exists via the tmux-session? predicate.

The script then calls terminal_ok? to execute the backend command terminal_window_exists against the cleanup window itself. If the owner window has disappeared, the watchdog bypasses normal recovery and proceeds directly to emergency teardown rather than attempting to recreate orphaned windows.

Detecting Closed Windows with the Missing Count Threshold

For every non-owner row with a living tmux session, the watchdog checks window existence using the terminal_window_exists backend command. The mechanism tolerates transient failures by maintaining a missing-counts map that tracks consecutive detection failures per window.

• Existing windows reset their missing count to 0 immediately upon successful detection.
• Missing windows increment their counter.

When a window's missing count reaches the missing-threshold (defaulting to 3 checks), the watchdog triggers the recovery sequence. This threshold prevents false positives from brief UI lag or momentary focus changes while ensuring actual closures are caught promptly.

Recovering Closed Agent Windows Automatically

Upon detecting a persistently closed window, the watchdog executes terminal_open_session with the original session name, title, and the cleanup window's ID as the parent. This backend command spawns a replacement terminal and returns a new-window-id.

The script then invokes rewrite-window-id! to atomically update the window identifier in both the window-state-file and the auxiliary window-ids file. This persistence ensures that subsequent watchdog iterations reference the correct window handle, maintaining continuity for the agent environment even after unexpected closures.

Loop Control and Emergency Shutdown

After processing all rows, the watchdog pauses execution with (Thread/sleep 2000) before recursing with updated missing-counts. This 2-second interval balances responsiveness against system resource consumption.

If the cleanup owner itself remains missing beyond the threshold, the script calls kill-all-sessions! to terminate all related tmux sessions and exits the loop. This safety mechanism prevents zombie processes when the controlling window is destroyed, ensuring clean teardown of the entire agent swarm.

Running the Window Watchdog Script

Launch the watchdog from the repository root by invoking the wrapper script with the required arguments:

# Arguments: <window-state-file> <window-ids-file> <cleanup-owner-index> \

#            <tmux-socket> <working-dir> [backend]

./swarmforge/scripts/swarm-window-watchdog.sh \
    /tmp/window-state.tsv /tmp/window-ids.tsv 0 /tmp/tmux.sock "$(pwd)" terminal-app

To manually update a window ID after external recreation (as performed automatically during recovery):

./swarmforge/scripts/swarm-window-watchdog.sh \
    --rewrite-window-id /tmp/window-state.tsv /tmp/window-ids.tsv 3 new-id-123

Before full cleanup, the watchdog internally invokes the hand-off daemon stopper:

(stop-handoff-daemon! script-dir working-dir)

Summary

• The window watchdog mechanism in swarmforge/scripts/swarm-window-watchdog.bb continuously monitors agent windows by reading a tab-separated state file and validating entries against live tmux sessions.
• It implements a missing count threshold (default 3) to distinguish between transient glitches and actual window closures.
• Recovery uses terminal_open_session to recreate missing windows and rewrite-window-id! to persist new identifiers across the state and ID files.
• The script sleeps 2000ms between checks and triggers kill-all-sessions! if the cleanup owner disappears, ensuring robust lifecycle management.

Frequently Asked Questions

What triggers the window watchdog to recover a closed agent window?

The watchdog increments a per-window missing-counts counter each time terminal_window_exists fails to locate the window. When this counter reaches the configurable missing-threshold (default 3 consecutive failures), the script executes terminal_open_session to spawn a replacement and updates the state files via rewrite-window-id!.

How does the watchdog handle the cleanup owner window disappearing?

If the window designated by cleanup-owner-index remains missing beyond the threshold, the watchdog calls kill-all-sessions! to terminate all associated tmux sessions and stops the hand-off daemon. This prevents orphaned processes when the controlling window is destroyed rather than attempting to recreate the owner.

Where does the window watchdog persist window state information?

The script maintains state in two files: window-state-file (tab-separated values with columns for index, window-id, session, and title) and window-ids-file. The rewrite-window-id! function atomically updates both files to ensure consistency between the watchdog's internal map and the persistent store.

What is the default check interval for the window watchdog mechanism?

The watchdog sleeps for 2000 milliseconds (2 seconds) between iterations using Thread/sleep 2000. This interval balances rapid failure detection with minimal CPU overhead, as implemented in the main loop of swarmforge/scripts/swarm-window-watchdog.bb.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}