# How the Callback Notification Delivery System Works in Background Agents

> Understand how the callback notification delivery system works in background agents. Learn about secure HTTP callbacks signed JSON payloads and asynchronous processing for real-time notifications.

- Repository: [Cole Murray/background-agents](https://github.com/ColeMurray/background-agents)
- Tags: how-to-guide
- Published: 2026-07-13

---

**Background Agents implement a secure, HTTP-based callback mechanism where the Control-Plane POSTs signed JSON payloads to bot-specific endpoints, verified via HMAC-SHA256 signatures and processed asynchronously to deliver real-time notifications without blocking execution.**

The ColeMurray/background-agents repository uses this architecture to bridge the Control-Plane's run lifecycle with user-facing interfaces like Slack and Linear. Understanding how the callback notification delivery system functions reveals the security patterns, validation layers, and asynchronous processing that ensure reliable delivery of session completions and tool-call updates.

## Callback Notification Delivery System Architecture

Each bot package mounts a Hono router at `/callbacks` to receive Control-Plane notifications. In [`packages/slack-bot/src/callbacks.ts`](https://github.com/ColeMurray/background-agents/blob/main/packages/slack-bot/src/callbacks.ts), the router exposes four distinct POST endpoints that handle different notification types:

- **`/complete`** – Delivers session-completion notifications when an agent run finishes
- **`/tool_call`** – Receives in-flight status updates for asynchronous tool calls
- **`/automation-complete`** – Handles Slack-triggered automation completions and clears the temporary "eyes" reaction
- **`/automation-skip`** – Sends ephemeral notices when a trigger is ignored due to an active run

The Linear bot mirrors this structure in [`packages/linear-bot/src/callbacks.ts`](https://github.com/ColeMurray/background-agents/blob/main/packages/linear-bot/src/callbacks.ts), adapting the handlers for Linear-specific contexts while reusing the same verification infrastructure.

## Security and Authentication

### HMAC-SHA256 Signature Verification

Every callback payload includes a `signature` field containing an HMAC-SHA256 hash of the JSON payload (excluding the signature itself). The `verifyCallbackSignature` function (lines 50-57 in [`packages/slack-bot/src/callbacks.ts`](https://github.com/ColeMurray/background-agents/blob/main/packages/slack-bot/src/callbacks.ts)) recomputes the hash using the shared secret and compares it using `timingSafeEqual` to prevent timing attacks:

```typescript
// Simplified verification logic from the source
async function verifyCallbackSignature(payload, secret) {
  const { signature, ...rest } = payload;
  const expected = await computeHmacHex(JSON.stringify(rest), secret);
  return timingSafeEqual(signature, expected);
}

```

If verification fails, the router invokes `rejectInvalidCallback` to return a **401 Unauthorized** response immediately.

### Secret Management

The `INTERNAL_CALLBACK_SECRET` is defined in `terraform/environments/production/variables.tf` (line 309) and injected into each worker via Terraform. At runtime, the secret is accessed through `c.env.INTERNAL_CALLBACK_SECRET` within the Hono context environment. This ensures that only the Control-Plane can generate valid signatures, preventing unauthorized callback injection.

## Payload Validation and Schema Enforcement

After authentication, payloads undergo strict schema validation using Zod. The system defines specific schemas for each callback type:

- `completionCallbackSchema` for session completions
- `toolCallCallbackSchema` for tool-call updates

The `rejectInvalidPayload` handler returns **400 Bad Request** for JSON parse errors or schema mismatches. This validation layer ensures type safety before the asynchronous processing begins.

## Asynchronous Processing Pattern

Once validated, callbacks are processed asynchronously using `c.executionCtx.waitUntil(...)` to prevent blocking the HTTP response. This pattern allows the Control-Plane to receive immediate acknowledgment while the bot performs downstream operations.

The main handler functions include:

- **`handleCompletionCallback`** – Fetches the agent response from D1, builds Slack blocks, posts the message, and clears the "thinking" reaction
- **`handleToolCallCallback`** – Processes status updates for pending tool calls
- **`handleAutomationComplete`** – Finalizes automation workflows and updates UI state
- **`handleAutomationSkip`** – Logs skipped triggers with structured context

Each function uses structured logging via `log.info`, `log.warn`, and `log.error` from [`packages/shared/src/logger.ts`](https://github.com/ColeMurray/background-agents/blob/main/packages/shared/src/logger.ts) to maintain observability across the distributed system.

## End-to-End Notification Flow

The complete callback notification delivery system operates through the following sequence:

1. **Control-Plane Dispatch** – When a run completes, the Control-Plane computes the HMAC signature using `INTERNAL_CALLBACK_SECRET` and POSTs the JSON payload to `<bot-worker-url>/callbacks/complete`

2. **Verification Layer** – The bot worker verifies the signature via `verifyCallbackSignature` and validates the payload against `completionCallbackSchema`

3. **Async Processing** – The handler calls `c.executionCtx.waitUntil(...)` to execute `handleCompletionCallback` in the background

4. **Message Construction** – The handler extracts the agent response using `extractAgentResponse`, builds formatted blocks via `buildCompletionBlocks`, and references [`packages/shared/src/activity-status.ts`](https://github.com/ColeMurray/background-agents/blob/main/packages/shared/src/activity-status.ts) for status formatting

5. **Delivery** – The bot posts the message to the original thread using `postMessage` and removes the temporary reaction via `clearThinkingReaction`

```typescript
/** Example: Control-Plane signing a completion callback */
import { createHmac } from "crypto";

const payload = {
  sessionId: "abc123",
  messageId: "1623456789.12345",
  success: true,
  timestamp: Date.now(),
  context: {
    source: "slack",
    channel: "C01SLACKCHAN",
    threadTs: "1623456789.12345",
    repoFullName: "owner/repo",
    model: "gpt-4o",
  },
};

const secret = process.env.INTERNAL_CALLBACK_SECRET!;
payload.signature = createHmac("sha256", secret)
  .update(JSON.stringify({ ...payload, signature: undefined }))
  .digest("hex");

await fetch("https://slack-bot-worker.example.com/callbacks/complete", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify(payload),
});

```

```typescript
// Handler implementation excerpt
async function handleCompletionCallback(payload, env, traceId) {
  const response = await extractAgentResponse(env, payload.sessionId, payload.messageId, traceId);
  const blocks = buildCompletionBlocks(payload.sessionId, response, payload.context, env.WEB_APP_URL);
  await postMessage(env.SLACK_BOT_TOKEN, payload.context.channel, getFallbackText(response), {
    thread_ts: payload.context.threadTs,
    blocks,
  });
  await clearThinkingReaction(env, payload.context.channel, payload.context.reactionMessageTs, traceId);
}

```

## Summary

- **Secure HTTP callbacks** – The Control-Plane communicates with bot workers via POST requests to `/callbacks` endpoints secured with HMAC-SHA256 signatures
- **Multi-layered validation** – Each request passes through signature verification (`verifyCallbackSignature`), timing-safe comparison (`timingSafeEqual`), and Zod schema validation before processing
- **Asynchronous execution** – The `c.executionCtx.waitUntil(...)` pattern ensures non-blocking notification delivery while maintaining fast HTTP responses
- **Distributed secret management** – The `INTERNAL_CALLBACK_SECRET` is managed via Terraform and injected into worker environments to prevent unauthorized callback injection
- **Observable processing** – Structured logging and dedicated handler functions (`handleCompletionCallback`, `handleToolCallCallback`, etc.) provide clear tracing across the notification lifecycle

## Frequently Asked Questions

### How does the callback notification delivery system authenticate requests?

The system uses HMAC-SHA256 signatures generated with a shared `INTERNAL_CALLBACK_SECRET`. The `verifyCallbackSignature` function in [`packages/slack-bot/src/callbacks.ts`](https://github.com/ColeMurray/background-agents/blob/main/packages/slack-bot/src/callbacks.ts) recomputes the hash and validates it using `timingSafeEqual` to prevent timing attacks. Requests with missing or invalid signatures receive a **401 Unauthorized** response via `rejectInvalidCallback`.

### What happens if a callback payload fails validation?

If JSON parsing fails or the payload doesn't match the expected Zod schema (such as `completionCallbackSchema` or `toolCallCallbackSchema`), the router returns a **400 Bad Request** through the `rejectInvalidPayload` handler. This ensures only well-formed, expected data enters the processing pipeline.

### Which endpoints handle different types of callback notifications?

The Hono router in [`packages/slack-bot/src/callbacks.ts`](https://github.com/ColeMurray/background-agents/blob/main/packages/slack-bot/src/callbacks.ts) defines four endpoints: `/callbacks/complete` for session completions, `/callbacks/tool_call` for tool status updates, `/callbacks/automation-complete` for finished automations, and `/callbacks/automation-skip` for ignored triggers. Each endpoint routes to specific handlers like `handleCompletionCallback` or `handleAutomationComplete`.

### How does the system prevent blocking during callback processing?

After validation, handlers invoke `c.executionCtx.waitUntil(...)` to process the callback asynchronously. This allows the HTTP response to return immediately to the Control-Plane while `handleCompletionCallback` or similar functions perform database lookups, build Slack blocks, and post messages in the background.