How Macro Implements Event-Driven Architecture with SQS Queues and Lambda Triggers

Macro uses Amazon SQS for durable, decoupled message buffering and AWS Lambda for lightweight event processing, with long-running Rust workers handling complex, stateful workloads.

The macro-inc/macro codebase is a Rust-based microservices platform where services communicate primarily through asynchronous events rather than direct calls. This event-driven architecture separates producers from consumers, enabling independent scaling, failure isolation, and loose coupling between system components.

Core Components of Macro's Event System

SQS Queues

Amazon SQS serves as the durable transport layer for all inter-service communication. Each service defines its queues in the local development catalog and initializes clients through a shared wrapper.

Queue definitions live in tooling/xtask/crates/xtask_local/src/local/resources.rs, where the local-stack environment declares all queues with their environment variable bindings. At runtime, services instantiate the sqs_client::SQS wrapper around the AWS SDK:

use aws_config::load_from_env;
use aws_sdk_sqs::Client as SqsSdkClient;
use sqs_client::SQS;

let aws_cfg = load_from_env().await;
let sqs_sdk = SqsSdkClient::new(&aws_cfg);
let sqs = SQS::new(sqs_sdk);

This pattern appears in services/search_processing_service/src/main.rs, where the SQS client is initialized once and shared across the service.

Lambda Triggers

Lambda functions handle short-lived, stateless event processing. Macro's Lambda entry points follow a consistent structure under services/*_lambda_handler, using the lambda_runtime crate to process SqsEvent payloads.

The minimal handler pattern in services/upload_extractor_lambda_handler/src/main.rs demonstrates this:

use lambda_runtime::{service_fn, Error, LambdaEvent};
use sqs_client::SQS;
use serde_json::Value;

#[tokio::main]
async fn main() -> Result<(), Error> {
    lambda_runtime::run(service_fn(handler)).await?;
    Ok(())
}

async fn handler(event: LambdaEvent<Value>, sqs: SQS) -> Result<(), Error> {
    tracing::debug!("Processing SQS event: {:?}", event.payload);
    // Domain-specific extraction logic
    Ok(())
}

SQS Workers

For long-running, stateful, or batch-processing workloads, Macro uses sqs_worker::SQSWorker instead of Lambda. Workers run inside standard Rust services, manage graceful shutdown, implement retry with backoff, and can maintain persistent connections to databases or search indices.

The worker initialization in services/search_processing_service/src/main.rs shows the configuration:

use sqs_worker::SQSWorker;
use std::sync::Arc;

let worker = SQSWorker::new(
    sqs.clone(),
    "my-queue-name",
    /* batch size */ 10,
    /* concurrency */ 4,
);
worker.run().await?;

Workers poll continuously, process messages in batches, and can emit new events to downstream queues—enabling complex multi-stage pipelines.

Complete Event Flow: Document Upload Processing

A typical document upload illustrates how these components interact:

1. Upload Service stores the raw file in S3 and sends a "process-upload" message to the designated SQS queue
2. upload_extractor_lambda_trigger monitors the queue and invokes the extractor Lambda
3. upload_extractor_lambda_handler receives the SQS event, extracts text, and persists results

The three components share queue definitions, ensuring that extractor failures don't lose events—messages remain visible for redelivery until processed or moved to a dead-letter queue.

Key Implementation Files

Path	Purpose
services/upload_extractor_lambda_trigger/src/main.rs	Lambda trigger that forwards SQS messages to processing Lambda
services/upload_extractor_lambda_handler/src/main.rs	Receives SQS events and executes extraction logic
services/search_processing_service/src/main.rs	SQS client setup, SQSWorker creation, and cross-queue publishing
services/worker_trigger/src/main.rs	Service that launches ECS tasks from queue messages
tooling/xtask/crates/xtask_local/src/local/resources.rs	LocalStack queue definitions and environment bindings

Design Tradeoffs: Lambda vs. Worker

Macro's architecture deliberately supports both execution models:

• Lambda functions — Best for quick, isolated tasks with cold-start tolerance; automatic scaling; pay-per-invocation pricing
• SQS Workers — Required for sustained throughput, connection pooling, large batch processing, or operations needing graceful shutdown semantics

The decision between them depends on latency requirements, processing duration, and resource needs. Both consume from the same SQS queues, allowing migration between models without protocol changes.

Summary

• SQS queues provide durable, decoupled messaging between all Macro services
• Lambda triggers handle lightweight, event-driven processing with minimal operational overhead
• SQSWorker enables long-running, stateful processing with fine-grained control over concurrency and retry behavior
• Shared queue definitions in tooling/xtask ensure consistency between local development and production
• Automatic redelivery and dead-letter queues make the system resilient to transient failures

Frequently Asked Questions

How does Macro handle SQS message failures?

Messages that fail processing become visible again after the visibility timeout expires, triggering automatic redelivery. After a configured number of attempts, messages move to a dead-letter queue for manual inspection. The sqs_worker crate implements exponential backoff and graceful shutdown to minimize duplicate processing.

Can Lambda functions and workers consume from the same queue?

Yes—both can target identical queues, though Macro typically partitions workloads to avoid competition. Lambda excels at fast, bursty traffic while workers handle sustained load. The same message format and queue configuration applies to both consumers.

What Rust crates power Macro's SQS integration?

The architecture relies on aws_sdk_sqs for AWS API calls, lambda_runtime for Lambda handlers, and internal crates sqs_client and sqs_worker for service-level abstractions. These wrappers standardize client initialization, error handling, and worker lifecycle management across the codebase.

How does local development simulate SQS and Lambda?

Macro uses LocalStack through the xtask_local tooling. Queue definitions in tooling/xtask/crates/xtask_local/src/local/resources.rs declare all SQS resources, enabling end-to-end testing of event flows without AWS credentials.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}