# How Macro Implements Event-Driven Architecture with SQS Queues and Lambda Triggers

> Discover Macro's event-driven architecture using SQS queues and Lambda triggers. Learn how Macro implements durable buffering and lightweight event processing with Rust workers.

- Repository: [Macro/macro](https://github.com/macro-inc/macro)
- Tags: architecture
- Published: 2026-08-16

---

**Macro uses Amazon SQS for durable, decoupled message buffering and AWS Lambda for lightweight event processing, with long-running Rust workers handling complex, stateful workloads.**

The `macro-inc/macro` codebase is a Rust-based microservices platform where services communicate primarily through asynchronous events rather than direct calls. This event-driven architecture separates producers from consumers, enabling independent scaling, failure isolation, and loose coupling between system components.

## Core Components of Macro's Event System

### SQS Queues

**Amazon SQS** serves as the durable transport layer for all inter-service communication. Each service defines its queues in the local development catalog and initializes clients through a shared wrapper.

Queue definitions live in [`tooling/xtask/crates/xtask_local/src/local/resources.rs`](https://github.com/macro-inc/macro/blob/main/tooling/xtask/crates/xtask_local/src/local/resources.rs), where the local-stack environment declares all queues with their environment variable bindings. At runtime, services instantiate the `sqs_client::SQS` wrapper around the AWS SDK:

```rust
use aws_config::load_from_env;
use aws_sdk_sqs::Client as SqsSdkClient;
use sqs_client::SQS;

let aws_cfg = load_from_env().await;
let sqs_sdk = SqsSdkClient::new(&aws_cfg);
let sqs = SQS::new(sqs_sdk);

```

This pattern appears in [`services/search_processing_service/src/main.rs`](https://github.com/macro-inc/macro/blob/main/services/search_processing_service/src/main.rs), where the SQS client is initialized once and shared across the service.

### Lambda Triggers

Lambda functions handle **short-lived, stateless event processing**. Macro's Lambda entry points follow a consistent structure under `services/*_lambda_handler`, using the `lambda_runtime` crate to process `SqsEvent` payloads.

The minimal handler pattern in [`services/upload_extractor_lambda_handler/src/main.rs`](https://github.com/macro-inc/macro/blob/main/services/upload_extractor_lambda_handler/src/main.rs) demonstrates this:

```rust
use lambda_runtime::{service_fn, Error, LambdaEvent};
use sqs_client::SQS;
use serde_json::Value;

#[tokio::main]
async fn main() -> Result<(), Error> {
    lambda_runtime::run(service_fn(handler)).await?;
    Ok(())
}

async fn handler(event: LambdaEvent<Value>, sqs: SQS) -> Result<(), Error> {
    tracing::debug!("Processing SQS event: {:?}", event.payload);
    // Domain-specific extraction logic
    Ok(())
}

```

### SQS Workers

For **long-running, stateful, or batch-processing workloads**, Macro uses `sqs_worker::SQSWorker` instead of Lambda. Workers run inside standard Rust services, manage graceful shutdown, implement retry with backoff, and can maintain persistent connections to databases or search indices.

The worker initialization in [`services/search_processing_service/src/main.rs`](https://github.com/macro-inc/macro/blob/main/services/search_processing_service/src/main.rs) shows the configuration:

```rust
use sqs_worker::SQSWorker;
use std::sync::Arc;

let worker = SQSWorker::new(
    sqs.clone(),
    "my-queue-name",
    /* batch size */ 10,
    /* concurrency */ 4,
);
worker.run().await?;

```

Workers poll continuously, process messages in batches, and can emit new events to downstream queues—enabling complex multi-stage pipelines.

## Complete Event Flow: Document Upload Processing

A typical document upload illustrates how these components interact:

1. **Upload Service** stores the raw file in S3 and sends a "process-upload" message to the designated SQS queue
2. **`upload_extractor_lambda_trigger`** monitors the queue and invokes the extractor Lambda
3. **`upload_extractor_lambda_handler`** receives the SQS event, extracts text, and persists results

The three components share queue definitions, ensuring that extractor failures don't lose events—messages remain visible for redelivery until processed or moved to a dead-letter queue.

## Key Implementation Files

| Path | Purpose |
|------|---------|
| [`services/upload_extractor_lambda_trigger/src/main.rs`](https://github.com/macro-inc/macro/blob/main/services/upload_extractor_lambda_trigger/src/main.rs) | Lambda trigger that forwards SQS messages to processing Lambda |
| [`services/upload_extractor_lambda_handler/src/main.rs`](https://github.com/macro-inc/macro/blob/main/services/upload_extractor_lambda_handler/src/main.rs) | Receives SQS events and executes extraction logic |
| [`services/search_processing_service/src/main.rs`](https://github.com/macro-inc/macro/blob/main/services/search_processing_service/src/main.rs) | SQS client setup, `SQSWorker` creation, and cross-queue publishing |
| [`services/worker_trigger/src/main.rs`](https://github.com/macro-inc/macro/blob/main/services/worker_trigger/src/main.rs) | Service that launches ECS tasks from queue messages |
| [`tooling/xtask/crates/xtask_local/src/local/resources.rs`](https://github.com/macro-inc/macro/blob/main/tooling/xtask/crates/xtask_local/src/local/resources.rs) | LocalStack queue definitions and environment bindings |

## Design Tradeoffs: Lambda vs. Worker

Macro's architecture deliberately supports both execution models:

- **Lambda functions** — Best for quick, isolated tasks with cold-start tolerance; automatic scaling; pay-per-invocation pricing
- **SQS Workers** — Required for sustained throughput, connection pooling, large batch processing, or operations needing graceful shutdown semantics

The decision between them depends on latency requirements, processing duration, and resource needs. Both consume from the same SQS queues, allowing migration between models without protocol changes.

## Summary

- **SQS queues** provide durable, decoupled messaging between all Macro services
- **Lambda triggers** handle lightweight, event-driven processing with minimal operational overhead
- **`SQSWorker`** enables long-running, stateful processing with fine-grained control over concurrency and retry behavior
- **Shared queue definitions** in `tooling/xtask` ensure consistency between local development and production
- **Automatic redelivery** and dead-letter queues make the system resilient to transient failures

## Frequently Asked Questions

### How does Macro handle SQS message failures?

Messages that fail processing become visible again after the visibility timeout expires, triggering automatic redelivery. After a configured number of attempts, messages move to a dead-letter queue for manual inspection. The `sqs_worker` crate implements exponential backoff and graceful shutdown to minimize duplicate processing.

### Can Lambda functions and workers consume from the same queue?

Yes—both can target identical queues, though Macro typically partitions workloads to avoid competition. Lambda excels at fast, bursty traffic while workers handle sustained load. The same message format and queue configuration applies to both consumers.

### What Rust crates power Macro's SQS integration?

The architecture relies on `aws_sdk_sqs` for AWS API calls, `lambda_runtime` for Lambda handlers, and internal crates `sqs_client` and `sqs_worker` for service-level abstractions. These wrappers standardize client initialization, error handling, and worker lifecycle management across the codebase.

### How does local development simulate SQS and Lambda?

Macro uses LocalStack through the `xtask_local` tooling. Queue definitions in [`tooling/xtask/crates/xtask_local/src/local/resources.rs`](https://github.com/macro-inc/macro/blob/main/tooling/xtask/crates/xtask_local/src/local/resources.rs) declare all SQS resources, enabling end-to-end testing of event flows without AWS credentials.