How the Macro Connection Gateway Manages Massive WebSocket Connections Through Batching and Auto-Scaling

The Connection Gateway uses a Rust client library that batches messages into single HTTP requests, combined with auto-scaling AWS Fargate containers, to handle thousands of concurrent WebSocket upgrades without per-socket overhead.

The Connection Gateway is a dedicated microservice in the macro-inc/macro repository that brokers real-time communication across the entire Macro platform. Rather than managing individual socket connections directly from application services, the gateway centralizes WebSocket lifecycle management and exposes a batch-send API that dramatically reduces network overhead.

Scalable AWS Infrastructure with Auto-Scaling

The gateway's deployment architecture eliminates connection bottlenecks by automatically adjusting capacity based on real-time demand.

Fargate and Application Load Balancer Setup

The ConnectionGateway class in [infra/stacks/connection-gateway/connection_gateway.ts](https://github.com/macro-inc/macro/blob/main/infra/stacks/connection-gateway/connection_gateway.ts#L9-L55) provisions:

• An AWS Fargate service for containerized, serverless compute
• An Application Load Balancer (ALB) for distributing WebSocket upgrade requests
• Security groups and target groups for network isolation

This configuration ensures new containers spin up quickly when connection volume increases, without manual intervention or pre-provisioned capacity.

Target-Tracking Auto-Scaling Policies

Lines 87-135 of the same infrastructure file implement two concurrent scaling policies:

1. ALB request rate tracking — Adds tasks when the rate of incoming requests exceeds thresholds
2. CPU utilization tracking — Scales based on container resource consumption

These policies work together to maintain sufficient Fargate tasks for accepting new WebSocket upgrades during traffic surges, then scale down during quiet periods to control costs.

Batch-Send API: One Request, Many Recipients

The gateway's most distinctive feature is its /message/batch_send endpoint, which eliminates the N+1 messaging problem that plagues naive WebSocket implementations.

How Batching Works

Instead of opening individual HTTP connections per recipient, the ConnectionGateway client serializes the payload once and transmits it once for all participants. The implementation in [crates/notification/src/outbound/websocket.rs](https://github.com/macro-inc/macro/blob/main/crates/notification/src/outbound/websocket.rs#L83-L98) shows this pattern:

// Build a ConnectionGatewayClient (internal auth key injected via Doppler)
let client = ConnectionGatewayClient::new(
    std::env::var("CG_INTERNAL_KEY").unwrap(),
    "https://connection-gateway.my-macro.com".to_string(),
);

// Prepare an arbitrary JSON payload
let payload = serde_json::json!({
    "event": "document_updated",
    "doc_id": "123e4567-e89b-12d3-a456-426614174000"
});

// List of user IDs (already parsed as MacroUserIdStr)
let recipients = vec![
    MacroUserIdStr::parse_from_str("user-1").unwrap(),
    MacroUserIdStr::parse_from_str("user-2").unwrap(),
    // … potentially thousands …
];

// Send once – the gateway will forward the payload to every live WebSocket
let delivered = client
    .batch_send_to_entities("document_update", &payload, recipients
        .iter()
        .map(|u| EntityType::User.with_entity_str(u.as_ref()))
        .collect())
    .await?;
println!("Delivered to {} users", delivered.len());

Entity-Based Routing Through DynamoDB

Behind the batch endpoint, the gateway maintains a mapping of Entity → WebSocket pairs in DynamoDB. As shown in lines 40-48 of the websocket module:

• Each Entity is a thin wrapper around a user ID
• The gateway queries active sockets for the provided entities
• It pushes the payload only to connections that are currently alive
• Returns a receipt indicating which users actually received the message

This lookup is fast and consistent, allowing any Fargate task to resolve its socket subset without shared in-memory state.

Fire-and-Forget Delivery for Low Latency

The Connection Gateway deliberately adopts a best-effort delivery semantic rather than guaranteed at-least-once delivery. The Rust client receives a HashSet<MacroUserIdStr> of successfully delivered users and does not retry per-socket failures.

This design choice, visible in lines 84-97 of the websocket module, provides two advantages:

• Low latency — No blocking wait for acknowledgments or retry timeouts
• Thundering-herd protection — When many connections drop simultaneously (common during network partitions or client app updates), the system doesn't attempt cascading retries that would overwhelm remaining resources

Stateless, Horizontally-Scalable Workers

Every architectural decision reinforces horizontal scalability:

Design Element	Scaling Benefit
Pre-serialized JSON payloads	Any task can process any request without transformation
DynamoDB for socket registry	Fast, consistent reads without inter-task communication
ALB traffic distribution	Even load across all healthy tasks
Independent request handling	No session affinity or sticky connections required

These properties mean the gateway can expand from a handful of tasks to dozens during peak usage, then contract automatically—all without configuration changes or deployment.

Higher-Level Abstractions in the Notification Service

The WebSocketGatewayAdapter provides a domain-specific wrapper for channel and notification events:

let gateway = ConnectionGatewayClient::new(key, url);
let adapter = WebSocketGatewayAdapter { gateway };

let users = vec![MacroUserIdStr::parse_from_str("alice")?];
let notif = NotificationStatusUpdate::new(vec![PatchDelete::Delete { id: "msg-99".into() }]);

// The adapter resolves which users actually have an open socket
let sent = adapter.send_notifications(&users, &notif).await?;
println!("Realtime sent to {} sockets", sent.len());

This abstraction lets application developers work with domain types (NotificationStatusUpdate, PatchDelete) while the adapter handles Entity conversion and batch construction.

Summary

The Macro Connection Gateway manages massive WebSocket connection volumes through four complementary strategies:

• Auto-scaling Fargate infrastructure that responds to request rate and CPU utilization
• Batch-send API that replaces N individual socket pushes with one HTTP request
• DynamoDB-backed entity routing for fast, stateless socket lookup
• Best-effort delivery that prioritizes low latency over guaranteed delivery

Together, these mechanisms let the platform accept thousands of concurrent WebSocket upgrades and broadcast updates to millions of users without proportional infrastructure overhead.

Frequently Asked Questions

What is the Connection Gateway in the Macro platform?

The Connection Gateway is a dedicated microservice that centralizes WebSocket connection management for real-time features. It runs as an auto-scaling AWS Fargate service behind an Application Load Balancer, exposing a batch-send API that lets other services push updates to many users through a single HTTP request.

How does the batch-send API reduce WebSocket overhead?

The batch_send_to_entities method in ConnectionGatewayClient accepts a list of user entities and one payload, serializes the payload once, and transmits it once. The gateway then fans this out to all active WebSocket connections internally. This replaces the common anti-pattern of opening separate HTTP connections or socket pushes per recipient, reducing network overhead from O(N) to O(1) for the calling service.

What happens if a WebSocket connection drops during message delivery?

The gateway implements fire-and-forget delivery. It returns a HashSet<MacroUserIdStr> indicating which users successfully received the message, but does not retry failed socket pushes. This keeps latency predictable and prevents retry storms when many clients disconnect simultaneously. Applications requiring stronger delivery guarantees must implement their own acknowledgment and retry logic.

Where is the Connection Gateway implemented in the Macro repository?

The implementation spans three areas: infrastructure definitions in [infra/stacks/connection-gateway/connection_gateway.ts](https://github.com/macro-inc/macro/blob/main/infra/stacks/connection-gateway/connection_gateway.ts), the Rust client library in [crates/notification/src/outbound/websocket.rs](https://github.com/macro-inc/macro/blob/main/crates/notification/src/outbound/websocket.rs), and higher-level adapters in [crates/channels/src/outbound/connection_gateway_realtime.rs](https://github.com/macro-inc/macro/blob/main/crates/channels/src/outbound/connection_gateway_realtime.rs) and [crates/connection/src/outbound/connection_gateway_client.rs](https://github.com/macro-inc/macro/blob/main/crates/connection/src/outbound/connection_gateway_client.rs).

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}