# Execution Semantics for Task Recovery and Watchdog Verification in Paperclip

> Discover Paperclip's task recovery and watchdog verification. Learn about its two-phase execution model, fingerprint-based change detection, and safety checks for robust agent management.

- Repository: [Paperclip/paperclip](https://github.com/paperclipai/paperclip)
- Tags: internals
- Published: 2026-08-18

---

**Paperclip's task recovery and watchdog verification system uses a two-phase execution model where a watchdog evaluates stopped subtrees via fingerprint-based change detection and enforces strict safety checks to prevent recursive loops and unauthorized agent assignments.**

Paperclip implements deterministic task recovery through a specialized **task watchdog** that monitors stopped workflow subtrees and decides when runs can safely resume. According to the `paperclipai/paperclip` source code, this system combines cryptographic fingerprinting for idempotency with multi-layered verification logic to ensure reliable automation. Below is a comprehensive breakdown of the execution semantics, implementation details, and verification safeguards.

## How Task Recovery Works in Paperclip

When a workflow run stops due to a halted subtree, Paperclip initiates a recovery protocol centered on the watchdog mechanism. The process operates in two interconnected phases: **task recovery** (the evaluation loop) and **watchdog verification** (the safety layer).

### Phase 1: Task Recovery and Watchdog Evaluation

The core recovery logic resides in [`server/src/services/task-watchdogs.ts`](https://github.com/paperclipai/paperclip/blob/main/server/src/services/task-watchdogs.ts). When a run stops, Paperclip performs the following operations:

1. **Stop fingerprint generation** — The system captures a cryptographic hash (`task_watchdog_stop:<hash>`) that uniquely identifies the exact state of the stopped subtree at termination (line 317)

2. **Watchdog attachment** — A new watchdog record is created and bound to the parent issue, storing the initial fingerprint as `lastReviewedStopSnapshot`

3. **Periodic re-evaluation** — Triggered by heartbeat events (`/heartbeat-runs/:runId/watchdog-decisions`), the watchdog wakes via `watchdogWakeContext` and reloads the subtree using `loadWatchdogSubtreeIssues` (lines 924–925)

4. **Change detection** — The watchdog recomputes the current fingerprint and compares it against `lastReviewedStopSnapshot`. If they match, the watchdog exits early to avoid redundant processing

5. **Classifier invocation** — When change is detected, `collectClassifierInput` inspects the subtree for pending reviews, pending interactions, and approval requirements to determine whether to **resume** the run or maintain **observe-only** mode

The idempotency key format ensures safe retries without duplicate side effects:

```ts
// From server/src/services/task-watchdogs.ts#L25
function taskWatchdogWakeIdempotencyKey(watchdogId: string, stopFingerprint: string) {
  return `task_watchdog:${watchdogId}:${stopFingerprint}`;
}

```

### Phase 2: Watchdog Verification and Safety Checks

Before any watchdog can operate, Paperclip enforces four critical verification constraints (lines 374–382):

| Verification Rule | Implementation Purpose | Failure Behavior |
|-------------------|------------------------|----------------|
| **No recursive watching** | Watchdog origin issues cannot themselves be watched | Prevents infinite watchdog loops |
| **Company boundary enforcement** | Watchdog and watched issue must share `companyId` | Maintains multi-tenant isolation |
| **Parent requirement** | Watched issue must have `parentId` | Ensures valid subtree topology |
| **Agent invokability** | Assigned agent must be explicitly invokable | Throws: "Cannot assign watchdog to an agent that is not invokable" (line 579) |

Additional post-run verification includes terminal state handling. When a watchdog issue reaches a terminal status, `markTerminalWatchdogIssueReviewed` (lines 1269–1283) marks it as reviewed to prevent further evaluation cycles.

## Complete Watchdog Execution Flow

The following sequence illustrates the end-to-end recovery semantics:

```ts
// Simplified evaluation logic from server/src/services/task-watchdogs.ts
async function evaluateWatchdog(input: {
  watchdog: IssueWatchdogRow;
  activeRuns: RunRow[];
  queuedWakeRequests: WakeRequestRow[];
}) {
  // Load current subtree state
  const currentStopSnapshot = await computeStopFingerprint(input.watchdog);
  
  // Idempotency check against last reviewed state
  if (input.watchdog.lastReviewedFingerprint === currentStopSnapshot) {
    return { state: "not_applicable", reason: "Already reviewed." };
  }
  
  // Classifier determines safety for resumption
  const classifierInput = await collectClassifierInput(input.watchdog);
  // ... classifier logic returns "resume" or "observe_only"
}

```

The watchdog persists its observation state across evaluations, updating both `lastObservedStopSnapshot` (current view) and `lastReviewedStopSnapshot` (actioned view) to maintain accurate change detection across wake cycles.

## Key Implementation Files

Understanding the codebase structure helps navigate the recovery system:

- **[`server/src/services/task-watchdogs.ts`](https://github.com/paperclipai/paperclip/blob/main/server/src/services/task-watchdogs.ts)** — Core implementation containing `createTaskWatchdog`, `watchdogWakeContext`, fingerprint computation, and verification logic

- **[`ui/src/lib/recovery-display.ts`](https://github.com/paperclipai/paperclip/blob/main/ui/src/lib/recovery-display.ts)** — Client-side rendering of recovery status indicators (e.g., "observe only" badges)

- **[`ui/src/api/heartbeats.ts`](https://github.com/paperclipai/paperclip/blob/main/ui/src/api/heartbeats.ts)** — API surface for watchdog wake triggers via `watchdog-decisions` endpoint

- **[`ui/src/lib/issue-properties-panel-key.ts`](https://github.com/paperclipai/paperclip/blob/main/ui/src/lib/issue-properties-panel-key.ts)** — Propagates watchdog configuration to the issue panel UI

- **[`ui/src/api/issues.ts`](https://github.com/paperclipai/paperclip/blob/main/ui/src/api/issues.ts)** — REST operations for watchdog lifecycle management (`GET/PUT/DELETE /issues/:id/watchdog`)

## Summary

- **Fingerprint-based idempotency** — Stop snapshots (`task_watchdog_stop:<hash>`) guarantee that watchdogs only act when subtree state materially changes, preventing redundant classifier invocations

- **Strict verification hierarchy** — Recursive watching prohibition, company isolation, parent requirement, and agent invokability checks form a defense-in-depth safety model

- **Deterministic evaluation loop** — The `watchdogWakeContext` → `loadWatchdogSubtreeIssues` → `collectClassifierInput` → state persistence flow ensures consistent, observable recovery decisions

- **Single-watchdog invariant** — Design constraints enforce exactly one active watchdog per company-issue tree, eliminating race conditions in distributed scenarios

## Frequently Asked Questions

### How does Paperclip prevent a watchdog from watching itself?

The verification logic in [`server/src/services/task-watchdogs.ts`](https://github.com/paperclipai/paperclip/blob/main/server/src/services/task-watchdogs.ts) (lines 374–382) explicitly checks that the watchdog origin issue is not already under watch. This "watchdog origin issues cannot themselves be watched" rule blocks recursive attachment attempts before database insertion, preventing infinite loops where a watchdog would trigger itself.

### What happens when a stopped subtree hasn't changed between watchdog evaluations?

When `computeStopFingerprint` produces a hash matching `lastReviewedStopSnapshot`, the watchdog returns early with state `"not_applicable"` and reason `"Already reviewed."` This idempotency mechanism, implemented in the core evaluation loop, avoids unnecessary classifier computation and database writes for stable subtrees.

### Why must the assigned agent be "invokable" for watchdog operation?

Paperclip enforces agent capability verification (line 579) to ensure that only properly configured automation agents can trigger workflow resumption. Non-invokable agents lack the execution context required for safe subtree reactivation; attempting assignment throws a descriptive error protecting against misconfiguration.

### Where is the watchdog resume decision actually made?

The resume-or-pause decision emerges from `collectClassifierInput` in [`server/src/services/task-watchdogs.ts`](https://github.com/paperclipai/paperclip/blob/main/server/src/services/task-watchdogs.ts) (lines 924–925), which aggregates subtree metadata including pending reviews and approval states. This classifier output feeds into the watchdog state machine that emits resume events only when all safety conditions are satisfied.