# How Macro's 42 Microservices Are Organized and How They Communicate: Architecture Deep Dive

> Discover how Macro organizes its 42 microservices in Rust crates and their communication methods including HTTP APIs SQS queues Redis PubSub and WebSockets.

- Repository: [Macro/macro](https://github.com/macro-inc/macro)
- Tags: architecture
- Published: 2026-08-16

---

**Macro's 42 microservices are organized as independent Rust crates in a single Cargo workspace under `services/`, communicating via HTTP APIs (Axum), AWS SQS queues, Redis Pub/Sub, and WebSocket gateways.**

The **macro-inc/macro** repository implements a **loosely coupled microservices architecture** using Rust's workspace system. Each service is a self-contained binary—some run as long-lived processes, others as AWS Lambda handlers—all built together while keeping dependencies isolated. This design enables horizontal scaling and independent deployment without sacrificing type safety or compile-time guarantees.

## Service Organization in the Cargo Workspace

Macro uses a **monorepo structure** with a **single Cargo workspace** at the root. The workspace's [`Cargo.toml`](https://github.com/macro-inc/macro/blob/main/Cargo.toml) lists approximately 42 member crates, allowing `cargo` to build, test, and lint all services together.

### Directory Structure

Each service follows a consistent layout:

```

services/<service_name>/
 ├─ Cargo.toml                # Declares binary target and dependencies

 ├─ src/
 │   ├─ main.rs               # Entry point (binary) or handler (Lambda)

 │   └─ ...                   # Service-specific modules

 └─ justfile / README.md      # Build and run helpers

```

### Service Categories

The 42 services cluster into six functional groups:

| Category | Example Services | Purpose |
|---|---|---|
| **Core storage** | `document_storage_service`, `document_cognition_service`, `search_service`, `static_file_service` | Persist documents, run OCR/AI cognition, full-text search indexing, static asset serving |
| **Processing pipelines** | `convert_service`, `document_text_extractor`, `image_optimizer`, `docx_unzip_handler`, `upload_extractor_lambda_handler` | Transform uploads: PDF → text, DOCX extraction, image compression, S3-triggered processing |
| **Communication & notifications** | `email_service`, `notification_service`, `connection_gateway`, `websocket-service` | Email delivery, push notifications, real-time client updates |
| **Infrastructure & auth** | `authentication_service`, `mcp_service`, `mcp_auth_proxy` | User authentication, request proxying, multi-tenant connection management |
| **Event-driven workers** | `worker_trigger`, `scheduled_action`, `organization_retention_trigger`, `organization_retention_handler` | React to SQS messages, cron schedules, data retention policies |
| **Auxiliary helpers** | `image_proxy_service`, `search_upload_handler`, `sha_cleanup_worker`, `dataloss_prevention_handler` | Background jobs: image proxying, SHA-based cleanup, DLP scanning |

## Communication Patterns Between Microservices

Macro's services remain **deliberately loosely coupled**, interacting through well-defined mechanisms rather than direct database sharing. Six primary patterns handle cross-service communication:

### HTTP APIs with Axum

Services expose **REST and GraphQL endpoints** using **Axum 0.8**. Other services consume these via `reqwest` or generated clients.

In [`services/static_file_service/src/api/mod.rs`](https://github.com/macro-inc/macro/blob/main/services/static_file_service/src/api/mod.rs):

```rust
use axum::{routing::get, Router};

async fn health_check() -> &'static str {
    "OK"
}

// Build the router for a service
let app = Router::new()
    .route("/healthz", get(health_check))
    .merge(api::router());   // service-specific sub-router

```

### AWS SQS Message Queues

**Event-driven workers** pull from SQS queues and publish downstream messages. The SQS client initializes with `aws_sdk_sqs::Client::new(&aws_config)`.

In [`services/authentication_service/src/main.rs`](https://github.com/macro-inc/macro/blob/main/services/authentication_service/src/main.rs):

```rust
use aws_config::load_from_env;
use aws_sdk_sqs::Client as SqsClient;

// Load AWS config (region, credentials) from environment/Doppler
let aws_config = load_from_env().await;

// Create a typed SQS client
let sqs_client = SqsClient::new(&aws_config);

// Wrap in Macro-specific helper (adds queue name handling)
let sqs = sqs_client::SQS::new(sqs_client);

```

### Redis Pub/Sub

**High-throughput messaging**—rate limiting, token buckets, ephemeral state—uses the `redis` crate for publish/subscribe channels.

In [`services/email_service/src/util/redis/mod.rs`](https://github.com/macro-inc/macro/blob/main/services/email_service/src/util/redis/mod.rs):

```rust
use redis::AsyncCommands;

async fn publish_rate_limit(redis: &redis::Client, key: &str, value: i64) -> redis::RedisResult<()> {
    let mut conn = redis.get_async_connection().await?;
    conn.publish(key, value).await
}

```

### WebSocket Gateway

Real-time updates flow through **connection_gateway**, which maintains WebSocket connections and forwards events to connected clients.

In [`services/connection_gateway/src/inbound/axum_router.rs`](https://github.com/macro-inc/macro/blob/main/services/connection_gateway/src/inbound/axum_router.rs):

```rust
use axum::extract::State;
use crate::gateway::Gateway;

async fn notify(State(gateway): State<Gateway>, payload: String) {
    // Broadcast to all connected clients
    gateway.broadcast(payload).await;
}

```

### S3 Event Triggers and Lambda

File uploads to S3 fire **Lambda handlers** that initiate processing pipelines. The `upload_extractor_lambda_handler` service serves as the entry point.

In [`services/upload_extractor_lambda_handler/src/main.rs`](https://github.com/macro-inc/macro/blob/main/services/upload_extractor_lambda_handler/src/main.rs):

```rust
// Lambda entry point triggered by S3 PUT events
// Deserializes S3 notification, validates, queues downstream work

```

### CloudWatch Scheduled Tasks

Periodic jobs run on timers and interact via SQS or Redis. The `scheduled_action` service implements cron-like scheduling.

In [`services/scheduled_action/src/bins/service.rs`](https://github.com/macro-inc/macro/blob/main/services/scheduled_action/src/bins/service.rs):

```rust
// Timer-based execution that pushes messages to SQS queues
// for downstream worker processing

```

## Shared Infrastructure and Configuration

All 42 services consume **consistent configuration** through the `macro_env_var` crate, which reads secrets and environment variables from **Doppler**. This ensures:

- Uniform AWS region and credential chains
- Synchronized feature flags across services
- Centralized secret rotation without code changes

## Key Source Files for Reference

| Service | File Path | Demonstrates |
|---|---|---|
| Workspace root | [[`Cargo.toml`](https://github.com/macro-inc/macro/blob/main/Cargo.toml)](https://github.com/macro-inc/macro/blob/main/Cargo.toml) | All ~42 service members declared |
| Authentication | [[`services/authentication_service/src/main.rs`](https://github.com/macro-inc/macro/blob/main/services/authentication_service/src/main.rs)](https://github.com/macro-inc/macro/blob/main/services/authentication_service/src/main.rs) | SQS client creation, token validation, HTTP endpoints |
| Email pipeline | [[`services/email_service/src/main.rs`](https://github.com/macro-inc/macro/blob/main/services/email_service/src/main.rs)](https://github.com/macro-inc/macro/blob/main/services/email_service/src/main.rs) | Multi-channel communication: SQS, Redis, Axum |
| Connection gateway | [[`services/connection_gateway/src/inbound/axum_router.rs`](https://github.com/macro-inc/macro/blob/main/services/connection_gateway/src/inbound/axum_router.rs)](https://github.com/macro-inc/macro/blob/main/services/connection_gateway/src/inbound/axum_router.rs) | WebSocket handling and HTTP forwarding |
| Scheduled actions | [[`services/scheduled_action/src/bins/service.rs`](https://github.com/macro-inc/macro/blob/main/services/scheduled_action/src/bins/service.rs)](https://github.com/macro-inc/macro/blob/main/services/scheduled_action/src/bins/service.rs) | CloudWatch-triggered periodic jobs |
| Upload extractor | [[`services/upload_extractor_lambda_handler/src/main.rs`](https://github.com/macro-inc/macro/blob/main/services/upload_extractor_lambda_handler/src/main.rs)](https://github.com/macro-inc/macro/blob/main/services/upload_extractor_lambda_handler/src/main.rs) | S3 event Lambda entry point |
| Document storage | [[`services/document_storage_service/src/main.rs`](https://github.com/macro-inc/macro/blob/main/services/document_storage_service/src/main.rs)](https://github.com/macro-inc/macro/blob/main/services/document_storage_service/src/main.rs) | PostgreSQL and S3 integration |
| Search service | [[`services/search_service/src/main.rs`](https://github.com/macro-inc/macro/blob/main/services/search_service/src/main.rs)](https://github.com/macro-inc/macro/blob/main/services/search_service/src/main.rs) | OpenSearch indexing and query API |

## Summary

- **Organization**: 42 Rust microservices live in `services/` as independent Cargo crates within a single workspace, enabling unified builds with isolated dependencies.
- **Communication**: Services interact through **HTTP (Axum)**, **AWS SQS queues**, **Redis Pub/Sub**, **WebSocket gateways**, and **S3/Lambda event triggers**—never direct database coupling.
- **Scalability**: The event-driven, queue-based design allows horizontal scaling of individual services without affecting the broader platform.
- **Consistency**: Shared `macro_env_var` configuration and standardized project layouts reduce cognitive overhead across the codebase.

## Frequently Asked Questions

### How many microservices does Macro have?

Macro maintains **approximately 42 microservices**, all implemented in Rust and organized as member crates in a single Cargo workspace. The exact count varies as services are added or consolidated, but the workspace structure in [`Cargo.toml`](https://github.com/macro-inc/macro/blob/main/Cargo.toml) provides the authoritative list.

### Why does Macro use a monorepo instead of separate repositories?

The **single Cargo workspace** enables atomic refactors across service boundaries, unified dependency management, and faster cross-service integration testing. Teams iterate on individual services using `cargo test -p <service>` without rebuilding the entire codebase, while CI pipelines leverage workspace-wide caching.

### What messaging protocol does Macro use for asynchronous communication?

Macro uses **AWS SQS as the primary async messaging backbone**, with **Redis Pub/Sub** for high-throughput, low-latency scenarios like rate limiting. SQS provides durable, scalable queue semantics; Redis handles ephemeral, in-process coordination. Both are wrapped in service-specific client crates for type safety.

### How does Macro handle real-time client updates?

The **`connection_gateway` service** maintains persistent WebSocket connections to clients and forwards events from backend services. Other services publish notifications through this gateway rather than managing sockets directly, centralizing connection state and simplifying horizontal scaling of stateless API workers.