# How Macro Delivers Webhooks for External Events: A Complete Async Pipeline

> Discover how Macro delivers webhooks for external events using an async pipeline. Learn about event reception, SQS enqueuing, and background worker processing with Reqwest.

- Repository: [Macro/macro](https://github.com/macro-inc/macro)
- Tags: deep-dive
- Published: 2026-08-15

---

**Macro delivers webhooks through a decoupled pipeline that receives external HTTP events, enqueues them to SQS, and processes delivery via background workers using a Reqwest-based HTTP client.**

When external services like Gmail, Stripe, or GitHub need to notify Macro, the system must handle burst traffic, guarantee delivery, and prevent duplicate processing. The Macro repository implements this through a multi-stage async architecture that separates fast ingress handling from reliable outbound delivery.

## The Webhook Delivery Pipeline: 5 Stages Explained

Macro's webhook system follows a clear data flow: **HTTP ingress → validation → queueing → worker consumption → HTTP delivery**. Each stage is isolated to ensure resilience and horizontal scalability.

### Stage 1: Ingress Handling via Axum Router

External events first hit HTTP endpoints exposed by individual services. The router lives in [`crates/webhook/src/inbound/axum_router.rs`](https://github.com/macro-inc/macro/blob/main/crates/webhook/src/inbound/axum_router.rs) and routes incoming POST requests to the appropriate handler.

Service-specific handlers validate the payload before any queueing occurs:

- **Gmail webhooks**: [`services/email_service/src/api/gmail/webhook.rs`](https://github.com/macro-inc/macro/blob/main/services/email_service/src/api/gmail/webhook.rs) validates GCP-signed JWTs
- **Stripe webhooks**: [`services/authentication_service/src/api/webhooks/user/stripe_webhook.rs`](https://github.com/macro-inc/macro/blob/main/services/authentication_service/src/api/webhooks/user/stripe_webhook.rs) verifies Stripe signatures

Once validated, the handler constructs a `WebhookEventQueueMessage` containing the `webhook_id` and event payload, then passes it to the outbound queue.

### Stage 2: FIFO Queueing with SQS

The `SqsWebhookQueue` implementation in [`crates/webhook/src/outbound/sqs_queue.rs`](https://github.com/macro-inc/macro/blob/main/crates/webhook/src/outbound/sqs_queue.rs) handles message persistence. Key implementation details:

```rust
// crates/webhook/src/outbound/sqs_queue.rs#L45-L70
pub async fn enqueue(&self, msg: WebhookEventQueueMessage) -> Result<(), QueueError> {
    let body = serde_json::to_string(&msg)?;
    let group_id = msg.webhook_id.clone();
    let dedup_id = format!("{}:{}", msg.webhook_id, msg.event.event_id);
    
    self.send_fifo_message(
        &self.queue_url,
        &body,
        &group_id,      // FIFO ordering key
        &dedup_id,      // prevents duplicate delivery
    ).await
}

```

**FIFO guarantees:**
- **Ordering**: Messages with the same `webhook_id` are processed in sequence
- **Deduplication**: The composite key `{webhook_id}:{event_id}` ensures idempotent processing

### Stage 3: Worker Message Consumption

The background worker polls SQS using `receive_messages` (lines 72-82 in [`sqs_queue.rs`](https://github.com/macro-inc/macro/blob/main/sqs_queue.rs)):

```rust
// crates/webhook/src/outbound/sqs_queue.rs#L72-L82
pub async fn receive_messages(&self) -> Result<Vec<RawWebhookEventQueueMessage>, QueueError> {
    let output = self.sqs
        .receive_message()
        .queue_url(&self.queue_url)
        .max_number_of_messages(self.max_messages)
        .wait_time_seconds(self.wait_time)
        .send()
        .await?;
        
    // Convert SQS messages to RawWebhookEventQueueMessage
    output.messages.map_or(Ok(vec![]), |msgs| {
        msgs.into_iter()
            .map(|m| self.raw_webhook_message(m))
            .collect()
    })
}

```

### Stage 4: HTTP Delivery via Reqwest

The `ReqwestWebhookDeliveryClient` in [`crates/webhook/src/outbound/http_delivery.rs`](https://github.com/macro-inc/macro/blob/main/crates/webhook/src/outbound/http_delivery.rs) performs the actual POST to the subscriber's target URL. It attaches the JSON payload and records delivery outcomes for observability.

### Stage 5: Retry Handling with Visibility Timeout

On transient failures, the worker delays message reprocessing by adjusting SQS visibility timeout:

```rust
// crates/webhook/src/outbound/sqs_queue.rs#L94-L106
pub async fn delay_message(
    &self,
    receipt_handle: &str,
    duration: Duration,
) -> Result<(), QueueError> {
    let seconds = duration.as_secs().min(43200) as i32; // Max 12 hours
    
    self.sqs
        .change_message_visibility()
        .queue_url(&self.queue_url)
        .receipt_handle(receipt_handle)
        .visibility_timeout(seconds)
        .send()
        .await?;
    Ok(())
}

```

## Complete Worker Implementation Pattern

Here's how the stages compose in practice:

```rust
// Enqueue from an inbound handler
let queue = SqsWebhookQueue::new(sqs.clone(), queue_url, 10, 20);
queue.enqueue(WebhookEventQueueMessage {
    webhook_id: webhook.id.clone(),
    event: WebhookEvent {
        event_id: uuid::Uuid::new_v4(),
        payload: serde_json::json!({ "type": "gmail_sync", "data": {} }),
    },
}).await?;

```

```rust
// Worker loop (crates/webhook/src/inbound/worker.rs)
loop {
    let msgs = queue.receive_messages().await?;
    for raw in msgs {
        let delivery = ReqwestWebhookDeliveryClient::new();
        
        match delivery.deliver(&raw.body).await {
            Ok(_) => {
                queue.delete_message(&raw.receipt_handle).await?;
            }
            Err(e) if e.is_transient() => {
                // Exponential backoff: delay and retry
                queue.delay_message(&raw.receipt_handle, Duration::from_secs(30)).await?;
            }
            Err(e) => {
                tracing::error!(error = ?e, "permanent delivery failure");
                queue.delete_message(&raw.receipt_handle).await?;
            }
        }
    }
}

```

## Key Design Decisions in Macro's Webhook System

| Decision | Implementation | Benefit |
|----------|---------------|---------|
| **FIFO SQS** | `SqsWebhookQueue` with custom group/dedup IDs | Exactly-once delivery per event |
| **Visibility timeout retry** | `delay_message()` caps at 12 hours | Bounded retry without separate dead-letter handling |
| **Service-specific validation** | Handlers in `email_service`, `authentication_service` | Proper signature verification per provider |
| **Decoupled queue/delivery** | Separate `SqsWebhookQueue` and `ReqwestWebhookDeliveryClient` traits | Testable, swappable components |

## Summary

- **Ingress routers** in [`crates/webhook/src/inbound/axum_router.rs`](https://github.com/macro-inc/macro/blob/main/crates/webhook/src/inbound/axum_router.rs) receive external HTTP events and validate signatures
- **SQS FIFO queueing** in [`crates/webhook/src/outbound/sqs_queue.rs`](https://github.com/macro-inc/macro/blob/main/crates/webhook/src/outbound/sqs_queue.rs) provides ordering and deduplication via `webhook_id` and composite dedup keys
- **Background workers** poll `receive_messages` and hand off to the delivery layer
- **Reqwest-based delivery** in [`crates/webhook/src/outbound/http_delivery.rs`](https://github.com/macro-inc/macro/blob/main/crates/webhook/src/outbound/http_delivery.rs) POSTs to target URLs with full retry semantics
- **Visibility timeout manipulation** enables delay-based retry without message re-enqueueing

## Frequently Asked Questions

### How does Macro prevent duplicate webhook delivery?

Macro uses SQS FIFO queues with composite deduplication IDs formatted as `{webhook_id}:{event_id}`. Because the `event_id` is generated uniquely per external event and persisted through retries, the same logical event will never be delivered twice to the same subscriber. The FIFO guarantee also preserves ordering for events sharing the same `webhook_id`.

### What happens when a webhook target is temporarily unavailable?

The worker catches transient failures and calls `delay_message()` to increase the SQS visibility timeout up to 12 hours. This hides the message from other workers while keeping it in the queue. After the timeout expires, the message becomes visible again for retry. Permanent failures (non-2xx responses, DNS failures) are logged and the message is deleted to prevent infinite retry loops.

### Can Macro's webhook system use queue implementations other than SQS?

Yes. The queue and delivery traits are abstracted: `SqsWebhookQueue` implements a generic queue interface, and `ReqwestWebhookDeliveryClient` implements a delivery trait. While the production configuration uses AWS SQS, the trait-based design allows substituting Redis streams, RabbitMQ, or in-memory queues for testing or alternative deployments.

### Where is the Gmail webhook signature validation performed?

Gmail-specific validation lives in [`services/email_service/src/api/gmail/webhook.rs`](https://github.com/macro-inc/macro/blob/main/services/email_service/src/api/gmail/webhook.rs), not in the shared webhook crate. This design keeps provider-specific logic (Google's JWT signature format) co-located with the email service that owns the Gmail integration, rather than polluting the generic webhook infrastructure with vendor-specific code paths.