How Herdr Live Handoff Transfers Running Panes Between Server Instances

Herdr transfers running panes by duplicating PTY file descriptors and passing them to a newly spawned server instance via Unix SCM_RIGHTS sockets, allowing the replacement process to assume control of existing terminal sessions without interrupting executing commands.

The herdr live handoff mechanism enables zero-downtime server upgrades by migrating active terminal sessions from one process to another. When triggered through the JSON API, the current server freezes pane output, serializes workspace state into a HandoffManifest, and transfers the underlying PTY file descriptors to a replacement binary. This ensures that running panes continue executing under the new instance while client sessions reconnect seamlessly.

The 12-Step Handoff Protocol

The live handoff implemented in src/server/headless.rs follows a strict coordination sequence to prevent data loss. Each step maps to specific functions in the Herdr codebase:

1. Request Initiation – A client sends {"method":"server.live_handoff","params":{…}} to the API socket, routed to HeadlessServer::perform_live_handoff at lines 71–73 of src/api/server.rs.

2. Socket Preparation – The server generates a unique Unix socket (herdr‑handoff‑<pid>.sock) with 0o600 permissions and a random validation token. This logic resides in src/server/handoff.rs at lines 51–59.

3. PTY Stream Freezing – Each TerminalRuntime calls pause_handoff_reader with a 2-second timeout to halt PTY data consumption, preventing output corruption during the transfer (see src/server/headless.rs lines 15–24).

4. State Snapshot – The workspace layout, focus states, and pane configurations are captured via crate::persist::capture (lines 26–37 of src/server/headless.rs).

5. Runtime Metadata Collection – For every pane, the server constructs a HandoffRuntimeState containing child_pid, dimensions (rows, cols), keyboard protocol flags, and optional scrollback history. This structure is defined in src/handoff_runtime.rs at lines 5–22.

6. Manifest Construction – handoff::manifest_for assembles the HandoffManifest struct (lines 33–43 of src/server/handoff.rs), which includes version metadata, the snapshot, and the per-pane runtime vector.

7. Import Server Spawn – The old server executes a new Herdr binary with --handoff-import <socket> <token> arguments via spawn_handoff_import (lines 56–84 of src/server/handoff.rs). The child process runs headless and awaits the manifest.

8. File Descriptor Duplication – Each TerminalRuntime invokes duplicate_handoff_fd() (referenced in the loop at lines 80–88 of perform_live_handoff), performing a Unix dup on the PTY fd and storing it in a Vec<RawFd>.

9. Connection Validation – The old server accepts the import server's connection on the temporary socket, verifies the token, and transmits the JSON manifest, expecting a "validated" response (lines 31–49 of src/server/handoff.rs).

10. FD Transfer via SCM_RIGHTS – send_fds_and_wait_restored transmits the duplicated PTY fds using SCM_RIGHTS control messages and blocks until the import server replies "restored" (lines 63–71 of src/server/handoff.rs).

11. Commit and Cleanup – Upon receiving "ready" and "restored" signals, the old server deletes its public API and client sockets, invokes preserve_for_handoff on each runtime to maintain PTY liveness, and sends "committed" to the new server (lines 66–78 of src/server/headless.rs).

12. Graceful Shutdown – The original process sets self.shutting_down = true, marks should_quit = true, and exits, leaving the new server to continue serving clients with the inherited pane processes (lines 74–80 of src/server/headless.rs).

Core Implementation Details

The perform_live_handoff Orchestrator

The primary entry point in src/server/headless.rs orchestrates the entire migration:

fn perform_live_handoff(
    &mut self,
    params: crate::api::schema::ServerLiveHandoffParams,
) -> io::Result<()> {
    self.handoff_in_progress = true;
    self.disconnect_all_clients_for_handoff();
    
    // Freeze PTY streams to prevent data races
    for terminal_id in pane_by_terminal.keys() {
        self.app.terminal_runtimes[terminal_id]
            .pause_handoff_reader(Duration::from_secs(2))?;
    }
    
    // Capture workspace and build manifest
    let snapshot = crate::persist::capture(...);
    let mut handoff_entries = Vec::new();
    for (terminal_id, runtime) in self.app.terminal_runtimes.iter() {
        handoff_entries.push((terminal_id.clone(), runtime.handoff_runtime_state(pane_id)));
    }
    let manifest = crate::server::handoff::manifest_for(...);
    
    // Spawn replacement and transfer state
    let mut import_child = crate::server::handoff::spawn_handoff_import(...)?;
    let mut fds = Vec::new();
    for (terminal_id, _) in &handoff_entries {
        fds.push(self.app.terminal_runtimes[terminal_id].duplicate_handoff_fd()?);
    }
    
    let mut stream = crate::server::handoff::accept_and_validate_on(...)?;
    crate::server::handoff::send_fds_and_wait_restored(&mut stream, &fds)?;
    // Commit and shutdown...
}

This method ensures atomicity: if any step fails before the commit phase, the server invokes rollback_handoff_before_commit to re-enable PTY readers and restore public sockets.

Serializing Pane State with HandoffManifest

The manifest structure transferred between processes contains version compatibility checks and session metadata:

pub(crate) struct HandoffManifest {
    pub version: u32,
    pub source_version: String,
    pub source_protocol: u32,
    pub expected_version: Option<String>,
    pub expected_protocol: Option<u32>,
    pub snapshot: crate::persist::SessionSnapshot,
    pub panes: Vec<crate::handoff_runtime::HandoffRuntimeState>,
}

Each HandoffRuntimeState (defined in src/handoff_runtime.rs) preserves critical terminal attributes:

pub(crate) struct HandoffRuntimeState {
    pub pane_id: u32,
    pub child_pid: u32,
    pub rows: u16,
    pub cols: u16,
    pub cell_width_px: u32,
    pub cell_height_px: u32,
    pub keyboard_protocol_flags: u16,
    pub keyboard_protocol_ansi: Option<String>,
    pub input_state: Option<crate::pane::InputState>,
    pub initial_history_ansi: Option<String>,
}

Transferring PTY File Descriptors via SCM_RIGHTS

The actual transfer running panes mechanism relies on Unix domain socket ancillary data. The send_fds function (lines 69–100 of src/server/handoff.rs) packs duplicated file descriptors into a control message using SCM_RIGHTS, while the import server uses recv_fds (lines 104–149) to receive them. This kernel-assisted handoff ensures that the new process possesses valid file descriptors pointing to the same underlying pseudo-terminals as the original server.

Practical Usage

CLI Trigger

Execute a live handoff from the command line:

herdr server live-handoff

This forwards the JSON RPC request to the running server, which then executes the 12-step protocol. Once complete, clients can reconnect via herdr attach to resume sessions exactly where they left off.

Direct JSON API

For scripted automation, send a line-terminated JSON request to the API socket (herdr.sock):

{
  "id": "handoff-1",
  "method": "server.live_handoff",
  "params": {
    "import_exe": "/usr/local/bin/herdr",
    "expected_version": "0.9.3",
    "expected_protocol": 9
  }
}

The server validates the optional expected_version and expected_protocol fields against the spawned binary before proceeding, preventing version mismatches during the handoff.

Error Handling and Rollback Safety

If any phase fails before the commit signal, rollback_handoff_before_commit (lines 82–90 of src/server/headless.rs) executes automatically:

• Re-enables PTY readers via the TerminalRuntime pause mechanism
• Clears the handoff_in_progress flag
• Restores the original public sockets so existing clients maintain connectivity

This design guarantees that running panes never terminate unexpectedly; they either migrate successfully to the new server or continue under the old server if the transfer aborts.

Summary

• Herdr live handoff migrates active terminal sessions between server processes using a 12-step Unix socket protocol.
• PTY file descriptors are duplicated and transferred via SCM_RIGHTS, allowing the new server to control existing processes without interruption.
• HandoffManifest and HandoffRuntimeState serialize layout, dimensions, and process metadata from src/server/handoff.rs and src/handoff_runtime.rs.
• perform_live_handoff in src/server/headless.rs orchestrates the transfer with automatic rollback on failure.
• Clients reconnect after handoff completion to find panes in identical states, including scrollback history and keyboard protocols.

Frequently Asked Questions

What happens to running processes during a herdr live handoff?

Running processes continue executing uninterrupted because the herdr live handoff mechanism transfers only the controlling PTY file descriptors to the new server process. The underlying child processes (identified by child_pid in HandoffRuntimeState) remain the same operating system entities; only the server process managing them changes. The original server calls preserve_for_handoff to ensure PTYs remain valid in the new process before shutting down.

How does Herdr ensure file descriptors are transferred securely?

Herdr employs a token-validated Unix socket with 0o600 permissions created specifically for the handoff transaction. Before transferring file descriptors via SCM_RIGHTS, the old server verifies that the connecting import server presents the correct random token generated during socket preparation. This prevents unauthorized processes from intercepting the PTY fds during the accept_and_validate_on phase in src/server/handoff.rs.

What causes a handoff to rollback instead of commit?

A handoff rolls back if any step fails before the commit phase, including: socket creation errors, PTY duplication failures, manifest validation mismatches, or timeout during the send_fds_and_wait_restored call. When triggered, rollback_handoff_before_commit re-enables all PTY readers and restores the original server's public sockets, ensuring clients experience only a brief pause rather than a session termination.

Can clients reconnect during a live handoff?

Clients must disconnect during the handoff because disconnect_all_clients_for_handoff drops existing connections to prevent state synchronization conflicts. However, once the new server signals "committed" and the old process removes its sockets, clients can immediately reconnect via herdr attach or new TCP/Unix socket connections to the new server instance, resuming their running panes without data loss.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}