# How the Paperclip Heartbeat System Triggers Agent Execution: A Deep Architectural Guide

> Understand how the Paperclip heartbeat system triggers agent execution. Learn about its three-phase pipeline: wakeup requests, queue processing, and agent launching.

- Repository: [Paperclip/paperclip](https://github.com/paperclipai/paperclip)
- Tags: deep-dive
- Published: 2026-08-18

---

**The Paperclip heartbeat system triggers agent execution through a three-phase pipeline: external events create wakeup requests, a background queue processor claims ready runs while respecting concurrency limits, and an execution engine launches agents with full runtime instrumentation.**

The **heartbeat ecosystem** in Paperclip transforms asynchronous events—issue comments, scheduled timers, manual approvals—into running agent executions. This article examines the implementation in the `paperclipai/paperclip` repository, tracing the path from wakeup signal to completed run.

## Core Architecture: Three Phases of Execution

The heartbeat service operates as a continuous loop with three tightly coupled phases: wakeup reception, queue processing, and run execution. Each phase is implemented in [`server/src/services/heartbeat.ts`](https://github.com/paperclipai/paperclip/blob/main/server/src/services/heartbeat.ts) with specific responsibilities for reliability and observability.

### Phase 1: Wakeup Reception and Registration

External events enter the system through `heartbeat.trackWakeup(agentId, opts)`. This method registers the wakeup in `activeWakeupPromises` and forwards to `enqueueWakeup` for persistence.

```typescript
// server/src/services/heartbeat.ts
// trackWakeup implementation around L13935
public trackWakeup(
  agentId: string,
  opts: {
    reason?: AgentWakeupReason;
    priority?: number;
    metadata?: Record<string, unknown>;
  } = {}
): void {
  const promise = this.enqueueWakeup(agentId, opts);
  this.activeWakeupPromises.add(promise);
  promise.finally(() => this.activeWakeupPromises.delete(promise));
}

```

The **wakeup request** is stored in the `agentWakeupRequests` table, creating a durable signal that survives process restarts. Priority and metadata allow fine-grained control over execution ordering.

### Phase 2: Queue Processing and Run Claiming

A background timer (started via `heartbeat.start()`) repeatedly invokes `startNextQueuedRunForAgent(agentId)`. This function implements the core dispatch logic with strict concurrency controls.

```typescript
// Pseudocode based on implementation around L13822
async startNextQueuedRunForAgent(agentId: string): Promise<void> {
  // Single-dispatcher guarantee per agent
  await this.withAgentStartLock(agentId, async () => {
    const agent = await this.loadAgent(agentId);
    const freeSlots = this.computeFreeSlots(agent.heartbeatPolicy.maxConcurrentRuns);
    
    if (freeSlots <= 0) return;
    
    const readyRuns = await this.selectReadyRuns(agentId, freeSlots);
    for (const run of readyRuns) {
      const claimed = await this.claimQueuedRun(run.id);
      if (claimed) {
        const execPromise = this.executeRun(run.id);
        this.activeRunExecutionPromises.add(execPromise);
        execPromise.finally(() => this.activeRunExecutionPromises.delete(execPromise));
      }
    }
  });
}

```

Key safeguards in this phase include:

- **Scheduling suppression** — prevents runs outside allowed time windows
- **Work-tree cut-off** — blocks execution when repository state is unstable
- **Agent start lock** — `withAgentStartLock` guarantees exactly one dispatcher per agent
- **Dependency sorting** — runs are ordered by dependency readiness, issue priority, and creation time

### Phase 3: Run Execution and Runtime Instrumentation

Once claimed, `executeRun(runId)` transforms the queued run into an active agent execution. This is the heavyweight phase involving database state transitions, runtime preparation, and adapter invocation.

```typescript
// Core execution flow beginning around L13952
async executeRun(runId: string): Promise<void> {
  // 1. State validation and claim
  const run = await this.reloadAndValidateRun(runId);
  
  // 2. Runtime preparation
  const agent = await this.loadAgent(run.agentId);
  const runtimeState = await this.ensureRuntimeState(run);
  
  // 3. Environment construction
  const scratch = await this.prepareHeartbeatRunScratch(run, agent);
  const { taskKey, codec, issueContext } = this.deriveExecutionParams(run, agent);
  
  // 4. Adapter invocation
  const adapter = this.getServerAdapter(agent.adapterType);
  const execution = await adapter.start({
    runtimeState,
    scratch,
    taskKey,
    codec,
    streamingCallbacks: this.createLogStreamers(runId)
  });
  
  // 5. Completion and event publishing
  await this.finalizeRun(runId, execution.result);
  await this.publishLiveEvent({ type: 'run_completed', runId, result: execution.result });
}

```

**Critical persistence guarantee**: All database writes complete before the returned promise resolves. This enables deterministic test teardown via `drainActiveRunExecutions()`.

## Coordination Mechanisms for Reliability

### Graceful Shutdown and Test Isolation

The heartbeat system tracks all active work through three internal sets:

- `activeRunExecutions` — currently running agent processes
- `activeRunExecutionPromises` — in-flight `executeRun` calls
- `activeWakeupPromises` — pending wakeup registrations

```typescript
// Awaiting complete quiescence
async drainActiveRunExecutions(): Promise<void> {
  await Promise.all([...this.activeWakeupPromises]);
  while (
    this.activeRunExecutions.size > 0 || 
    this.activeRunExecutionPromises.size > 0
  ) {
    await Promise.race([
      ...this.activeRunExecutionPromises,
      sleep(100)
    ]);
  }
}

```

### Workspace Contention Handling

When multiple agents target the same workspace, `WorkspaceBusyDeferral` implements bounded exponential backoff:

```typescript
// Retry configuration from source
const WORKSPACE_BUSY_RETRY_BASE_DELAY_MS = 1000;
const WORKSPACE_BUSY_RETRY_MAX_DELAY_MS = 30000;
const WORKSPACE_BUSY_RETRY_MAX_ATTEMPTS = 5;

```

### Policy Enforcement Injection

Before execution, multiple services validate and prepare the run context:

| Service | Responsibility |
|---------|---------------|
| `costService` | Estimates and tracks token consumption |
| `budgetService` | Enforces spending limits per agent/organization |
| `secretService` | Binds and injects configured secrets |
| `trustService` | Validates low-trust environment requirements |

## Practical Usage Examples

### Programmatic Wakeup Trigger

```typescript
import { heartbeat } from '@paperclipai/server';

// Fire-and-forget wakeup from external event handler
app.post('/github/webhook', async (req, res) => {
  const { agentId, issueNumber } = parseWebhook(req.body);
  
  heartbeat.trackWakeup(agentId, {
    reason: 'issue_commented',
    priority: issueNumber,
    metadata: { commentId: req.body.comment.id }
  });
  
  res.status(202).send('Wakeup queued');
});

```

### CLI Invocation

The `paperclipai heartbeat-run` command uses the same internal path:

```bash

# Direct agent execution via CLI

$ paperclipai heartbeat-run --agent-id=agent-abc123 --wait

# With priority override

$ paperclipai heartbeat-run --agent-id=agent-abc123 --priority=10 --reason=manual_trigger

```

### Test Harness with Drain

```typescript
import { createTestHeartbeat } from '@paperclipai/server/testing';

describe('agent execution', () => {
  let heartbeat: HeartbeatService;
  
  beforeEach(() => {
    heartbeat = createTestHeartbeat();
  });
  
  afterEach(async () => {
    // Ensures no orphaned database writes
    await heartbeat.drainActiveRunExecutions();
  });
  
  it('processes wakeup through completion', async () => {
    heartbeat.trackWakeup('test-agent', { reason: 'test' });
    
    // Wait for natural completion
    await heartbeat.drainActiveRunExecutions();
    
    const runs = await listRuns('test-agent');
    expect(runs[0].status).toBe('completed');
  });
});

```

## Key Source Files

| File | Role | Link |
|------|------|------|
| [`server/src/services/heartbeat.ts`](https://github.com/paperclipai/paperclip/blob/main/server/src/services/heartbeat.ts) | Core service: wakeup handling, queue dispatch, execution orchestration | [github.com/paperclipai/paperclip/blob/master/server/src/services/heartbeat.ts](https://github.com/paperclipai/paperclip/blob/master/server/src/services/heartbeat.ts) |
| [`server/src/services/heartbeat-run-summary.ts`](https://github.com/paperclipai/paperclip/blob/main/server/src/services/heartbeat-run-summary.ts) | Result formatting, output trimming, PR comment generation | [heartbeat-run-summary.ts](https://github.com/paperclipai/paperclip/blob/master/server/src/services/heartbeat-run-summary.ts) |
| [`packages/shared/src/types/heartbeat.ts`](https://github.com/paperclipai/paperclip/blob/main/packages/shared/src/types/heartbeat.ts) | Public TypeScript definitions for payloads and status enums | [heartbeat types](https://github.com/paperclipai/paperclip/blob/master/packages/shared/src/types/heartbeat.ts) |
| [`cli/src/commands/heartbeat-run.ts`](https://github.com/paperclipai/paperclip/blob/main/cli/src/commands/heartbeat-run.ts) | CLI interface wrapping `trackWakeup` | [CLI command](https://github.com/paperclipai/paperclip/blob/master/cli/src/commands/heartbeat-run.ts) |
| [`ui/src/api/heartbeats.ts`](https://github.com/paperclipai/paperclip/blob/main/ui/src/api/heartbeats.ts) | Frontend polling and real-time status subscription | [UI API](https://github.com/paperclipai/paperclip/blob/master/ui/src/api/heartbeats.ts) |

## Summary

The Paperclip heartbeat system triggers agent execution through:

- **Durable wakeup registration** via `trackWakeup`, storing signals in `agentWakeupRequests` with priority metadata
- **Lock-protected queue processing** in `startNextQueuedRunForAgent`, respecting `maxConcurrentRuns` and dependency ordering
- **Fully instrumented execution** through `executeRun`, with guaranteed persistence before promise resolution
- **Clean coordination primitives** including `drainActiveRunExecutions` for graceful shutdown and test isolation

This architecture ensures that every external event—whether from webhooks, schedules, or manual triggers—reliably converts to an executed agent run with full observability and policy enforcement.

## Frequently Asked Questions

### What happens if an agent is already at max concurrency when a wakeup arrives?

The wakeup is stored in `agentWakeupRequests` and the queue processor will attempt to claim runs on the next polling cycle. No events are dropped; the system continuously retries until capacity becomes available or the request expires based on configured TTL policies.

### How does the heartbeat system handle process crashes during execution?

Run state is persisted to the database at each phase transition (queued → running → completed/failed). On restart, the queue processor rescues runs stuck in `running` state by validating heartbeats from the underlying adapter; orphaned runs are marked failed and can be retried.

### Can multiple heartbeat processes run simultaneously for high availability?

Yes, the `withAgentStartLock` mechanism ensures only one dispatcher operates per agent across all processes. The lock is implemented via database advisory locks or distributed lock service depending on deployment configuration.

### What is the difference between `trackWakeup` and directly calling `executeRun`?

`trackWakeup` enqueues a durable request and returns immediately, allowing the background processor to handle scheduling and concurrency. Direct `executeRun` calls bypass the queue and are used only internally after a run has been claimed and validated.