# How Webhooks Notify External Systems of Events in Macro: A Complete Technical Guide

> Learn how Macro webhooks notify external systems of events. Discover the asynchronous pipeline, SQS queuing, and HMAC-SHA256 signed payload delivery.

- Repository: [Macro/macro](https://github.com/macro-inc/macro)
- Tags: how-to-guide
- Published: 2026-08-18

---

**Macro notifies external systems through an asynchronous pipeline that normalizes domain events into HTTP POST requests, queues them via SQS, and delivers them via a worker that signs each payload with HMAC-SHA256.**

Macro's webhook notification system decouples internal domain events from external HTTP callbacks using a trait-based pipeline architecture. When events occur—such as document creation in the Document Storage Service—they transform into signed, queued messages that workers deliver to registered endpoints. This implementation, found in the `macro-inc/macro` repository's `webhook` crate, ensures reliable, retry-aware delivery to any external system.

## The Webhook Notification Pipeline

Macro uses a six-stage pipeline to turn internal events into HTTP calls that external systems consume:

1. **Event Normalisation** – Domain events become `NormalizedWebhookEvent` instances capturing event metadata and broker payloads.
2. **Queue Message Creation** – The system wraps normalized events into `WebhookEventQueueMessage` structs containing the target webhook ID.
3. **Enqueue** – A `WebhookEventEnqueuer` implementation persists messages to an asynchronous queue (typically SQS).
4. **Worker Consumption** – A `WebhookEventWorker` pulls messages, resolves webhook configuration, and delegates to a delivery client.
5. **HTTP Delivery** – The `WebhookDeliveryClient` builds signed POST requests with Macro-specific headers.
6. **Result Recording** – Delivery outcomes are persisted via `WebhookDeliveryRepository`, enabling retry logic and auditability.

All components are defined by traits in the **`webhook`** crate, allowing pluggable implementations for different queue backends, HTTP clients, and storage engines.

## Event Normalization and Enqueueing

When a domain event fires—such as a new document stored in `services/document_storage_service`—the ingestion layer creates a `NormalizedWebhookEvent` that captures the event ID, name, entity type, and original broker payload. This structure is defined in [`crates/webhook/src/domain/models.rs`](https://github.com/macro-inc/macro/blob/main/crates/webhook/src/domain/models.rs).

The ingestion service then constructs a `WebhookEventQueueMessage::new(webhook_id, event)`, which versions the payload for queue compatibility. This message combines the normalized event with the specific webhook ID stored in the database.

The `WebhookEventEnqueuer` trait handles persistence. The SQS implementation in [`crates/webhook/src/outbound/sqs_queue.rs`](https://github.com/macro-inc/macro/blob/main/crates/webhook/src/outbound/sqs_queue.rs) provides `SqsWebhookQueue::enqueue(message)`, which stores the message for asynchronous processing:

```rust
// Normalise a broker event
let normalized = NormalizedWebhookEvent {
    event_id: "evt_123".into(),
    schema_version: 1,
    event_name: "document.created".into(),
    entity_type: "document".into(),
    entity_id: doc.id.clone(),
    ordering_key: "doc_123".into(),
    occurred_at: chrono::Utc::now(),
    broker_envelope: serde_json::json!({ "doc_id": doc.id, "title": doc.title }),
};

// Wrap for the queue
let queue_msg = WebhookEventQueueMessage::new("wh_abc".into(), normalized);

// Enqueue via SQS
let enqueuer = SqsWebhookQueue::new(sqs_client, queue_url);
enqueuer.enqueue(queue_msg).await?;

```

## Asynchronous Delivery Architecture

Once events are queued, the delivery process becomes the responsibility of the worker and delivery client components.

### The WebhookEventWorker Implementation

The `WebhookEventWorker` defined in [`crates/webhook/src/inbound/worker.rs`](https://github.com/macro-inc/macro/blob/main/crates/webhook/src/inbound/worker.rs) continuously polls the queue via `process_message`. For each `WebhookEventQueueMessage`, it:

- Deserializes the queue payload
- Looks up the current webhook configuration (endpoint URL, custom headers, signing secret)
- Calls `WebhookDeliveryClient::deliver(&webhook, &event)`

The worker runs as a background task in services like the Document Storage Service, instantiated in [`services/document_storage_service/src/main.rs`](https://github.com/macro-inc/macro/blob/main/services/document_storage_service/src/main.rs).

### HTTP Request Construction and Signing

The actual HTTP delivery occurs in [`crates/webhook/src/outbound/http_delivery.rs`](https://github.com/macro-inc/macro/blob/main/crates/webhook/src/outbound/http_delivery.rs) within the delivery client implementation. The client builds requests through two key functions:

- **`delivery_headers`** – Assembles required Macro headers (`x-macro-event`, `x-macro-event-id`, `x-macro-timestamp`), merges custom user-defined headers, and calculates the signature header.
- **`delivery_request`** – Creates the `reqwest::Request` with the JSON-encoded broker envelope.

The **`signature_header`** function generates an HMAC-SHA256 signature using the webhook's signing secret, timestamp, and request body. This signature appears in the `x-macro-signature` header, allowing external systems to verify payload authenticity.

The request is sent via `reqwest::Client`, after which `classify_http_status` categorizes the response into a `WebhookHttpOutcome` variant: `Success`, `RetryableFailure`, or `PermanentFailure`.

## Delivery Outcomes and Retry Logic

After attempting delivery, the system persists the result through the `WebhookDeliveryRepository` trait. The PostgreSQL implementation in [`crates/webhook/src/outbound/pg_delivery_repository.rs`](https://github.com/macro-inc/macro/blob/main/crates/webhook/src/outbound/pg_delivery_repository.rs) records the outcome, updates delivery status, and schedules retries for transient failures.

This repository pattern allows the worker to classify responses and implement backoff strategies without blocking the main application flow. External systems can rely on eventual delivery even during temporary network partitions or endpoint downtime.

## Implementation Example

Below is a complete illustration showing how a service enqueues events and how the worker processes them:

```rust
// Service side: Enqueue a document creation event
let normalized = NormalizedWebhookEvent {
    event_id: "evt_123".into(),
    schema_version: 1,
    event_name: "document.created".into(),
    entity_type: "document".into(),
    entity_id: doc.id.clone(),
    ordering_key: "doc_123".into(),
    occurred_at: chrono::Utc::now(),
    broker_envelope: serde_json::json!({ "doc_id": doc.id, "title": doc.title }),
};

let queue_msg = WebhookEventQueueMessage::new("wh_abc".into(), normalized);
let enqueuer = SqsWebhookQueue::new(sqs_client, queue_url);
enqueuer.enqueue(queue_msg).await?;

// Worker side: Background delivery task
let queue = SqsWebhookQueue::new(sqs_client, queue_url);
let delivery = ReqwestWebhookDeliveryClient::new();
let worker = WebhookEventWorker::new(queue, delivery);
worker.run().await; // Continuously processes messages

```

The `WebhookEventWorker` pulls messages, resolves configuration via the repository layer, and invokes `delivery_client.deliver()`, which internally handles request construction, signing, and transmission.

## Key Source Files and Data Models

Understanding the notification flow requires familiarity with these specific files:

- **[`crates/webhook/src/domain/models.rs`](https://github.com/macro-inc/macro/blob/main/crates/webhook/src/domain/models.rs)** – Defines `NormalizedWebhookEvent`, `WebhookEventQueueMessage`, and delivery status enums.
- **[`crates/webhook/src/domain/ports.rs`](https://github.com/macro-inc/macro/blob/main/crates/webhook/src/domain/ports.rs)** – Trait definitions for `WebhookEventEnqueuer`, queue access, and delivery clients.
- **[`crates/webhook/src/outbound/sqs_queue.rs`](https://github.com/macro-inc/macro/blob/main/crates/webhook/src/outbound/sqs_queue.rs)** – SQS implementation of the queue traits.
- **[`crates/webhook/src/outbound/http_delivery.rs`](https://github.com/macro-inc/macro/blob/main/crates/webhook/src/outbound/http_delivery.rs)** – HTTP delivery logic, header preparation, HMAC-SHA256 signing, and response classification.
- **[`crates/webhook/src/inbound/worker.rs`](https://github.com/macro-inc/macro/blob/main/crates/webhook/src/inbound/worker.rs)** – The asynchronous worker that orchestrates message consumption and delivery.
- **[`services/document_storage_service/src/main.rs`](https://github.com/macro-inc/macro/blob/main/services/document_storage_service/src/main.rs)** – Example service wiring the ingestion pipeline.
- **[`services/email_service/src/api/gmail/webhook.rs`](https://github.com/macro-inc/macro/blob/main/services/email_service/src/api/gmail/webhook.rs)** – Reference for inbound webhook endpoints that external systems call.

## Summary

Macro's webhook notification system provides reliable, asynchronous delivery of domain events to external systems through these key mechanisms:

- **Event normalization** converts broker messages into standardized `NormalizedWebhookEvent` structures with versioning support.
- **Queue-based architecture** decouples event generation from delivery using `WebhookEventQueueMessage` and pluggable enqueuers like SQS.
- **Signed delivery** ensures security via HMAC-SHA256 signatures in the `x-macro-signature` header, generated in [`http_delivery.rs`](https://github.com/macro-inc/macro/blob/main/http_delivery.rs).
- **Worker-driven processing** handles HTTP transmission, response classification, and outcome persistence through the `WebhookEventWorker`.
- **Retry logic** at the repository layer allows recovery from transient failures without data loss.

## Frequently Asked Questions

### What triggers a webhook notification in Macro?

Domain events within Macro services—such as document creation in the Document Storage Service—trigger the webhook pipeline. When these events occur, the ingestion layer creates a `NormalizedWebhookEvent` that flows through the queue to registered webhook endpoints. The specific event types depend on which services instantiate the webhook ingestion infrastructure in their [`main.rs`](https://github.com/macro-inc/macro/blob/main/main.rs) files.

### How does Macro secure webhook deliveries to external systems?

Macro secures deliveries through HMAC-SHA256 request signing. The `signature_header` function in [`crates/webhook/src/outbound/http_delivery.rs`](https://github.com/macro-inc/macro/blob/main/crates/webhook/src/outbound/http_delivery.rs) generates a signature using the webhook's signing secret, timestamp, and request body. This signature appears in the `x-macro-signature` header, while `x-macro-timestamp` prevents replay attacks. External systems can verify these signatures to authenticate that payloads originated from Macro.

### What happens when a webhook delivery fails?

The `classify_http_status` function categorizes HTTP responses into `Success`, `RetryableFailure`, or `PermanentFailure` variants of `WebhookHttpOutcome`. Retryable failures (such as 5xx errors or timeouts) are recorded in the `WebhookDeliveryRepository` for later retry, while permanent failures (such as 4xx client errors) are logged without retry attempts. This logic ensures external systems receive eventual delivery during transient outages without wasting resources on permanently broken endpoints.

### Where is the webhook worker implementation located?

The core worker logic resides in [`crates/webhook/src/inbound/worker.rs`](https://github.com/macro-inc/macro/blob/main/crates/webhook/src/inbound/worker.rs), which defines the `WebhookEventWorker` responsible for polling queues and orchestrating delivery. The SQS queue implementation is in [`crates/webhook/src/outbound/sqs_queue.rs`](https://github.com/macro-inc/macro/blob/main/crates/webhook/src/outbound/sqs_queue.rs), while the HTTP delivery client is in [`crates/webhook/src/outbound/http_delivery.rs`](https://github.com/macro-inc/macro/blob/main/crates/webhook/src/outbound/http_delivery.rs). Services instantiate these components in their respective [`main.rs`](https://github.com/macro-inc/macro/blob/main/main.rs) files, such as [`services/document_storage_service/src/main.rs`](https://github.com/macro-inc/macro/blob/main/services/document_storage_service/src/main.rs).