How the Callback Notification Delivery System Works in Background Agents

Background Agents implement a secure, HTTP-based callback mechanism where the Control-Plane POSTs signed JSON payloads to bot-specific endpoints, verified via HMAC-SHA256 signatures and processed asynchronously to deliver real-time notifications without blocking execution.

The ColeMurray/background-agents repository uses this architecture to bridge the Control-Plane's run lifecycle with user-facing interfaces like Slack and Linear. Understanding how the callback notification delivery system functions reveals the security patterns, validation layers, and asynchronous processing that ensure reliable delivery of session completions and tool-call updates.

Callback Notification Delivery System Architecture

Each bot package mounts a Hono router at /callbacks to receive Control-Plane notifications. In packages/slack-bot/src/callbacks.ts, the router exposes four distinct POST endpoints that handle different notification types:

• /complete – Delivers session-completion notifications when an agent run finishes
• /tool_call – Receives in-flight status updates for asynchronous tool calls
• /automation-complete – Handles Slack-triggered automation completions and clears the temporary "eyes" reaction
• /automation-skip – Sends ephemeral notices when a trigger is ignored due to an active run

The Linear bot mirrors this structure in packages/linear-bot/src/callbacks.ts, adapting the handlers for Linear-specific contexts while reusing the same verification infrastructure.

Security and Authentication

HMAC-SHA256 Signature Verification

Every callback payload includes a signature field containing an HMAC-SHA256 hash of the JSON payload (excluding the signature itself). The verifyCallbackSignature function (lines 50-57 in packages/slack-bot/src/callbacks.ts) recomputes the hash using the shared secret and compares it using timingSafeEqual to prevent timing attacks:

// Simplified verification logic from the source
async function verifyCallbackSignature(payload, secret) {
  const { signature, ...rest } = payload;
  const expected = await computeHmacHex(JSON.stringify(rest), secret);
  return timingSafeEqual(signature, expected);
}

If verification fails, the router invokes rejectInvalidCallback to return a 401 Unauthorized response immediately.

Secret Management

The INTERNAL_CALLBACK_SECRET is defined in terraform/environments/production/variables.tf (line 309) and injected into each worker via Terraform. At runtime, the secret is accessed through c.env.INTERNAL_CALLBACK_SECRET within the Hono context environment. This ensures that only the Control-Plane can generate valid signatures, preventing unauthorized callback injection.

Payload Validation and Schema Enforcement

After authentication, payloads undergo strict schema validation using Zod. The system defines specific schemas for each callback type:

• completionCallbackSchema for session completions
• toolCallCallbackSchema for tool-call updates

The rejectInvalidPayload handler returns 400 Bad Request for JSON parse errors or schema mismatches. This validation layer ensures type safety before the asynchronous processing begins.

Asynchronous Processing Pattern

Once validated, callbacks are processed asynchronously using c.executionCtx.waitUntil(...) to prevent blocking the HTTP response. This pattern allows the Control-Plane to receive immediate acknowledgment while the bot performs downstream operations.

The main handler functions include:

• handleCompletionCallback – Fetches the agent response from D1, builds Slack blocks, posts the message, and clears the "thinking" reaction
• handleToolCallCallback – Processes status updates for pending tool calls
• handleAutomationComplete – Finalizes automation workflows and updates UI state
• handleAutomationSkip – Logs skipped triggers with structured context

Each function uses structured logging via log.info, log.warn, and log.error from packages/shared/src/logger.ts to maintain observability across the distributed system.

End-to-End Notification Flow

The complete callback notification delivery system operates through the following sequence:

1. Control-Plane Dispatch – When a run completes, the Control-Plane computes the HMAC signature using INTERNAL_CALLBACK_SECRET and POSTs the JSON payload to <bot-worker-url>/callbacks/complete

2. Verification Layer – The bot worker verifies the signature via verifyCallbackSignature and validates the payload against completionCallbackSchema

3. Async Processing – The handler calls c.executionCtx.waitUntil(...) to execute handleCompletionCallback in the background

4. Message Construction – The handler extracts the agent response using extractAgentResponse, builds formatted blocks via buildCompletionBlocks, and references packages/shared/src/activity-status.ts for status formatting

5. Delivery – The bot posts the message to the original thread using postMessage and removes the temporary reaction via clearThinkingReaction

/** Example: Control-Plane signing a completion callback */
import { createHmac } from "crypto";

const payload = {
  sessionId: "abc123",
  messageId: "1623456789.12345",
  success: true,
  timestamp: Date.now(),
  context: {
    source: "slack",
    channel: "C01SLACKCHAN",
    threadTs: "1623456789.12345",
    repoFullName: "owner/repo",
    model: "gpt-4o",
  },
};

const secret = process.env.INTERNAL_CALLBACK_SECRET!;
payload.signature = createHmac("sha256", secret)
  .update(JSON.stringify({ ...payload, signature: undefined }))
  .digest("hex");

await fetch("https://slack-bot-worker.example.com/callbacks/complete", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify(payload),
});

// Handler implementation excerpt
async function handleCompletionCallback(payload, env, traceId) {
  const response = await extractAgentResponse(env, payload.sessionId, payload.messageId, traceId);
  const blocks = buildCompletionBlocks(payload.sessionId, response, payload.context, env.WEB_APP_URL);
  await postMessage(env.SLACK_BOT_TOKEN, payload.context.channel, getFallbackText(response), {
    thread_ts: payload.context.threadTs,
    blocks,
  });
  await clearThinkingReaction(env, payload.context.channel, payload.context.reactionMessageTs, traceId);
}

Summary

• Secure HTTP callbacks – The Control-Plane communicates with bot workers via POST requests to /callbacks endpoints secured with HMAC-SHA256 signatures
• Multi-layered validation – Each request passes through signature verification (verifyCallbackSignature), timing-safe comparison (timingSafeEqual), and Zod schema validation before processing
• Asynchronous execution – The c.executionCtx.waitUntil(...) pattern ensures non-blocking notification delivery while maintaining fast HTTP responses
• Distributed secret management – The INTERNAL_CALLBACK_SECRET is managed via Terraform and injected into worker environments to prevent unauthorized callback injection
• Observable processing – Structured logging and dedicated handler functions (handleCompletionCallback, handleToolCallCallback, etc.) provide clear tracing across the notification lifecycle

Frequently Asked Questions

How does the callback notification delivery system authenticate requests?

The system uses HMAC-SHA256 signatures generated with a shared INTERNAL_CALLBACK_SECRET. The verifyCallbackSignature function in packages/slack-bot/src/callbacks.ts recomputes the hash and validates it using timingSafeEqual to prevent timing attacks. Requests with missing or invalid signatures receive a 401 Unauthorized response via rejectInvalidCallback.

What happens if a callback payload fails validation?

If JSON parsing fails or the payload doesn't match the expected Zod schema (such as completionCallbackSchema or toolCallCallbackSchema), the router returns a 400 Bad Request through the rejectInvalidPayload handler. This ensures only well-formed, expected data enters the processing pipeline.

Which endpoints handle different types of callback notifications?

The Hono router in packages/slack-bot/src/callbacks.ts defines four endpoints: /callbacks/complete for session completions, /callbacks/tool_call for tool status updates, /callbacks/automation-complete for finished automations, and /callbacks/automation-skip for ignored triggers. Each endpoint routes to specific handlers like handleCompletionCallback or handleAutomationComplete.

How does the system prevent blocking during callback processing?

After validation, handlers invoke c.executionCtx.waitUntil(...) to process the callback asynchronously. This allows the HTTP response to return immediately to the Control-Plane while handleCompletionCallback or similar functions perform database lookups, build Slack blocks, and post messages in the background.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}