What Is the Terminal Watchdog in SwarmForge and How Does It Manage Window State?

The terminal watchdog in SwarmForge is a background supervisor process that monitors tmux sessions and terminal windows, automatically recreating missing windows or gracefully terminating the entire environment when the owner window disappears.

The terminal watchdog in SwarmForge ensures resilient terminal-window management by acting as a self-healing supervisor for the development environment. Located in the unclebob/swarm-forge repository, this Babashka-based process reads persistent state files and verifies window integrity every two seconds. It bridges the gap between tmux session management and the host terminal application, providing automatic recovery from transient failures.

Core Responsibilities of the Terminal Watchdog

The watchdog operates as a continuous polling loop implemented in swarmforge/scripts/swarm_window_watchdog.bb. It maintains the mapping between tmux sessions and their associated terminal windows, enforcing consistency through several distinct mechanisms.

Reading Window State

The watchdog begins each iteration by loading the current window configuration from a tab-separated state file. The rows function (lines 13-19 in swarm_window_watchdog.bb) parses window-state.tsv, extracting each window's index, tmux session name, window ID, and title.

This state file serves as the source of truth for the entire SwarmForge environment, allowing the watchdog to track which windows should exist regardless of the current process state.

Detecting Missing Windows

For every non-owner window entry, the watchdog verifies existence through two consecutive checks (lines 88-100):

1. tmux-session? – Validates that the tmux session identifier still exists in the tmux socket
2. terminal-ok? – Executes the backend-specific terminal_window_exists command via the adapter script to confirm the terminal window is still alive

If either check fails, the window is marked as missing and enters the recovery queue.

Automatic Recovery with Retry Logic

The watchdog implements a configurable tolerance threshold to avoid false positives from transient system delays. The missing-threshold parameter defaults to 3 attempts, decrementing only after consecutive failures.

When a window exceeds this threshold (lines 101-108), the watchdog:

• Invokes terminal_open_session through the adapter to spawn a replacement window
• Captures the new window ID from the backend
• Calls rewrite-window-id! to update both window-state.tsv and window-ids.txt with the new identifier

This state mutation ensures the rest of the SwarmForge environment immediately recognizes the recovered window.

Owner Window Protection and Global Cleanup

The row designated by cleanup-owner-index represents the primary "owner" window that anchors the entire SwarmForge session. When this specific window disappears (lines 112-115), the watchdog initiates a graceful teardown rather than attempting recovery:

1. Invokes stop_handoff_daemon! to halt the hand-off coordination service
2. Executes kill-all-sessions! (lines 65-72), which iterates through all tracked tmux sessions and terminal windows, explicitly closing each before terminating

This design prevents zombie processes and orphaned tmux sessions when the main control window is lost.

Implementation Architecture

The watchdog follows a simple but robust polling pattern. The main loop sleeps for 2 seconds between iterations (Thread/sleep 2000 at line 110), providing sufficient responsiveness without excessive CPU usage.

The system relies on three primary components:

• swarmforge/scripts/swarm_window_watchdog.bb – Core Babashka implementation containing the monitoring logic, state file parsers, and recovery routines
• swarmforge/scripts/swarm-window-watchdog.sh – Thin Zsh wrapper that forwards arguments to the BB script
• swarmforge/scripts/swarm-terminal-adapter.sh – Backend abstraction layer providing terminal_window_exists, terminal_open_session, and other terminal-specific operations

Runtime state persists in:

• window-state.tsv – Contains index, tmux session, window ID, and title mappings
• window-ids.txt – Simple flat list of window identifiers kept synchronized with the TSV file

Running and Invoking the Watchdog

You can launch the watchdog manually for debugging or invoke it programmatically from other Babashka scripts.

Manual Execution via Shell

Execute the wrapper script with the required state file paths and parameters:

# From the Swarm Forge repository root

./swarmforge/scripts/swarm-window-watchdog.sh \
  /path/to/window-state.tsv \
  /path/to/window-ids.txt \
  0                     # cleanup-owner-index

  /tmp/tmux.sock        # tmux socket path

  $(pwd)                # working directory

  terminal-app          # backend adapter (optional)

The script forwards these arguments to swarm_window_watchdog.bb, which initializes the monitoring loop.

Programmatic Invocation from Clojure

Integrate the watchdog into larger Babashka workflows by spawning it as a subprocess:

(require '[babashka.process :as proc])

(let [args ["/tmp/window-state.tsv"
            "/tmp/window-ids.txt"
            "0"                    ; cleanup-owner-index
            "/tmp/tmux.sock"
            (str (fs/pwd))
            "terminal-app"]]
  (proc/sh ["bb" "-f" "swarmforge/scripts/swarm_window_watchdog.bb"]
           :inherit true
           :args args))

This pattern allows parent scripts to manage the watchdog lifecycle while maintaining isolated terminal state monitoring.

Summary

• The terminal watchdog in SwarmForge acts as a self-healing supervisor for terminal window management, implemented in swarmforge/scripts/swarm_window_watchdog.bb.
• It polls every 2 seconds, checking tmux session validity and terminal window existence via the terminal_window_exists adapter function.
• Missing windows trigger an automatic recovery sequence using terminal_open_session and rewrite-window-id!, respecting a configurable missing-threshold (default 3).
• If the owner window (identified by cleanup-owner-index) disappears, the watchdog executes kill-all-sessions! to prevent orphaned processes.
• State files (window-state.tsv and window-ids.txt) provide persistent tracking across process restarts and window recreations.

Frequently Asked Questions

How does the terminal watchdog detect when a window has crashed?

The watchdog detects crashed windows by verifying both the tmux session existence and the terminal window visibility. According to the source code in swarm_window_watchdog.bb (lines 88-100), it first checks if the tmux session exists using tmux-session?, then validates the terminal window through the terminal_window_exists function provided by the adapter script. If either check fails, the window is flagged for recovery or counting toward the missing threshold.

What happens when the owner window disappears in SwarmForge?

When the owner window disappears, the watchdog initiates a complete teardown of the SwarmForge environment rather than attempting recovery. As implemented in lines 112-115 of swarm_window_watchdog.bb, the watchdog counts missing occurrences of the window identified by cleanup-owner-index. Once the count reaches the threshold, it calls stop_handoff_daemon! and kill-all-sessions! (lines 65-72), which explicitly closes every tracked tmux session and terminal window before exiting.

Can I adjust how many times the watchdog retries before opening a new window?

Yes, the retry behavior is controlled by the missing-threshold parameter, which defaults to 3 attempts. This threshold determines how many consecutive polling cycles a window can be missing before the watchdog executes terminal_open_session and updates the state files via rewrite-window-id!. You can modify this value in the source code or potentially pass it as a configuration parameter depending on your invocation method.

Where does the terminal watchdog store its state between checks?

The watchdog persists state in two runtime files: window-state.tsv (a tab-separated file storing index, tmux session, window ID, and title) and window-ids.txt (a simple list of window identifiers). The rows function reads window-state.tsv at the start of each loop (lines 13-19), while rewrite-window-id! updates both files whenever a window is recreated, ensuring the watchdog maintains accurate state across recovery operations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}