How the Window Watchdog Reopens Closed Agent Windows in SwarmForge

The SwarmForge window watchdog monitors tmux window states via a TSV mapping file and automatically recreates any closed agent windows after three consecutive detection cycles, updating the window registry to maintain swarm continuity.

SwarmForge manages distributed agent sessions through individual tmux windows, but accidental closures can disrupt workflows. The window watchdog (swarm-window-watchdog.bb) implements a resilient monitoring loop that detects vanished windows and respawns them without manual intervention. This mechanism ensures that temporary terminal failures or accidental window closures do not terminate active agent sessions.

How the Window Watchdog Tracks Agent Windows

The watchdog operates on a state file that maps each agent to its tmux window ID, session, and title. This persistent registry enables the system to verify window existence across polling cycles.

Reading the Window State File

At the start of each iteration, the watchdog parses the window state TSV using the rows function (lines 13‑19 in swarm-window-watchdog.bb). This file maintains the relationship between index, window-id, tmux-session, and title for every agent surface.

The function reads the tab-separated values into memory, creating a vector of row maps that the watchdog uses to verify each window's current status.

Detecting Missing Windows

For every non-cleanup window, the watchdog invokes the backend command terminal_window_exists through the terminal-ok? helper (lines 90‑100). This shell call checks whether the tmux window ID registered in the state file still exists in the target session.

If the command returns a non-zero exit code, the watchdog flags the window as missing and increments its internal counter. Present windows reset their counters to zero immediately.

The Threshold-Based Reopening Logic

Rather than reopening windows on the first missed detection, SwarmForge implements a threshold mechanism to avoid false positives during brief tmux latency spikes.

Incrementing Missing Window Counters

The watchdog maintains a missing-counts hash map that tracks consecutive detection failures for each window index. When terminal-ok? reports a window as missing, the counter increments; when the window exists, the counter resets to zero (lines 94‑108).

The constant missing-threshold (line 8) defaults to 3, meaning the watchdog tolerates two consecutive detection failures before taking action. This debouncing prevents unnecessary window recreation during temporary system load.

Recreating Windows with terminal_open_session

Once a window's missing count reaches the threshold, the watchdog executes terminal_open_session (lines 103‑104) to spawn a replacement. This command creates a fresh tmux window within the same session, using the original title and the cleanup window as a parent reference.

After successful creation, the watchdog calls rewrite-window-id! (lines 29‑35) to atomically update the state file with the new window ID. This ensures subsequent polling cycles reference the correct tmux window rather than the defunct ID.

(let [new-window-id (terminal-out script-dir working-dir tmux-socket backend
                                  "terminal_open_session" session title cleanup-window-id)]
  (when-not (str/blank? new-window-id)
    (rewrite-window-id! window-state-file window-ids-file index new-window-id))
  (assoc counts index 0))

Launching and Configuring the Watchdog

The watchdog runs as a background process spawned by the main SwarmForge controller after initial window creation.

Starting the Watchdog Process

In swarmforge.bb, the open-terminal-surfaces! function launches the watchdog via process/process (lines 41‑48). The system only starts the watchdog if the terminal backend supports window tracking, verified by terminal_backend_tracks_windows.

The process receives the window state file path, tmux socket, working directory, and backend type as arguments. All output redirects to window-watchdog.log for debugging.

(process/process [(str (fs/path (:script-dir ctx) "swarm-window-watchdog.sh"))
                  (str (:window-state-file ctx))
                  (str (:window-ids-file ctx))
                  "1"
                  (:tmux-socket ctx)
                  (str (:working-dir ctx))
                  (:terminal-backend ctx)]
                 {:out (str (:window-watchdog-log ctx))
                  :err :out})

Configuring the Missing Window Threshold

The missing-threshold variable at line 8 of swarm-window-watchdog.bb controls the sensitivity of the reopening logic. Operators can modify this constant to require more or fewer consecutive failures before recreation, though the default value of 3 balances responsiveness against false positives.

Handling the Cleanup Window Failure

The watchdog treats the cleanup window as a special sentinel. If this parent window disappears, the system assumes a critical failure has occurred. After detecting the cleanup window as missing, the watchdog eventually invokes kill-all-sessions! (lines 112‑115) to terminate the entire swarm gracefully, preventing orphaned agent processes from consuming resources indefinitely.

Summary

• The window watchdog in swarm-window-watchdog.bb polls tmux window existence every two seconds using terminal_window_exists checks.
• Missing windows accumulate counters up to a configurable missing-threshold (default 3) before triggering recreation.
• Reopened windows invoke terminal_open_session and update the state file via rewrite-window-id! to maintain accurate mappings.
• The watchdog launches as a background process from swarmforge.bb only when the terminal backend supports window tracking.
• Loss of the cleanup window triggers kill-all-sessions! to prevent resource leaks across the swarm.

Frequently Asked Questions

What triggers the window watchdog to reopen a window?

The watchdog reopens a window when the terminal_window_exists command returns a failure code for three consecutive polling cycles (the default missing-threshold). This threshold prevents premature recreation during brief tmux unavailability or system latency.

How many times does the watchdog check before reopening a window?

By default, the watchdog checks three times before reopening, as defined by the missing-threshold constant in swarm-window-watchdog.bb. Each check occurs approximately every two seconds, meaning a window typically recreates within six seconds of confirmed closure.

What happens if the cleanup window itself closes?

If the cleanup window—the parent reference used when spawning replacement windows—disappears, the watchdog assumes catastrophic session failure. It executes kill-all-sessions! to terminate all remaining tmux sessions and prevent orphaned agent processes from running indefinitely without supervision.

Where does the watchdog log its activity?

The watchdog writes all standard output and error streams to window-watchdog.log in the working directory. This log captures detection events, window recreation attempts, and any errors encountered during terminal_open_session calls, as configured in the process launcher within swarmforge.bb.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}