Static File Service and CDN Integration in the Macro Platform

The Macro static file service combines Amazon S3, CloudFront, and Lambda to provide a globally distributed CDN for binary assets with on-the-fly image optimization.

This repository (macro-inc/macro) implements a fully-managed storage layer for binary assets that exposes files through a fast, globally-distributed CDN. The architecture leverages AWS infrastructure to store files securely, serve them efficiently at the edge, and transform images dynamically without manual intervention.

Core Architecture Components

The static file service integrates multiple AWS services into a cohesive delivery pipeline. Each component is defined in TypeScript using Pulumi infrastructure-as-code, ensuring reproducible deployments across development, staging, and production environments.

S3 Bucket and DynamoDB Metadata Store

At the foundation lies the static-file-storage-<stack> S3 bucket, configured with versioning enabled and lifecycle rules for managing old versions. The bucket definition appears in [infra/stacks/static-file-service/static-file-service.ts](https://github.com/macro-inc/macro/blob/main/infra/stacks/static-file-service/static-file-service.ts) (lines 21‑49), where CORS policies allow cross-origin requests from any domain.

Metadata tracking occurs through a DynamoDB table created in the same stack (lines 95‑121). This table stores critical file attributes including owner_id, MIME type, and the S3 key mapping. The table name passes through the dynamoDbTableName parameter, utilizing a primary key schema of file_id with a Global Secondary Index (GSI) on owner_id for efficient owner-based queries.

ECS Fargate API Layer

The Axum-based HTTP API runs on ECS Fargate behind an Application Load Balancer. The service definition in [static-file-service.ts](https://github.com/macro-inc/macro/blob/main/infra/stacks/static-file-service/static-file-service.ts) (lines 42‑62) specifies container builds using docker/Dockerfile with the STATIC_FILE_SERVICE build argument. This architecture handles presigned URL generation and metadata management while delegating actual file delivery to the CDN layer.

Security groups segregate the ALB from the service containers, with explicit ingress and egress rules defined in the initializeSecurityGroups function. IAM roles aggregate granular policies for S3 access, DynamoDB operations, ECR pulls, and Secrets Manager retrieval.

CloudFront CDN and Origin Access Controls

The global distribution layer resides in [infra/stacks/static-file-service/distribution.ts](https://github.com/macro-inc/macro/blob/main/infra/stacks/static-file-service/distribution.ts). Unlike legacy implementations using bucket policies, this architecture employs Origin Access Controls (OAC) for both the S3 bucket and Lambda origins (lines 48‑231).

The CloudFront distribution implements a fail-over origin group named image-optimization-group, where S3 serves as the primary origin and the image optimizer Lambda functions as the secondary. Custom error handling routes 403 responses from S3 to the Lambda origin automatically. A response-headers policy adds security headers at the edge, while cache behaviors specify minTtl: 0 and defaultTtl: 31536000 for optimal performance.

Critical security enforcement occurs through bucket policies (lines 90‑107) that validate the AWS:SourceArn condition matches the distribution ARN explicitly. This ensures objects remain inaccessible outside the CDN pathway.

Image Optimization Lambda

Dynamic image transformation occurs through a dedicated Lambda function defined in [static-file-service.ts](https://github.com/macro-inc/macro/blob/main/infra/stacks/static-file-service/static-file-service.ts) (lines 151‑210). When triggered, the function fetches the original image from S3, performs resizing or format conversion, writes the transformed version back to a deterministic key pattern (file/<uuid>/<format>), and returns the optimized asset.

Route 53 provides friendly DNS resolution at static-file-service.<stack>.macro.com, creating a seamless endpoint for client applications (lines 82‑94).

CDN Integration Flow

The complete data flow follows four distinct phases that abstract storage complexity from client applications.

1. Presigned Upload Generation — Clients request upload capabilities via POST /api/file. The ECS service writes a DynamoDB entry using the helper logic in [crates/s3_key/src/static_file_key.rs](https://github.com/macro-inc/macro/blob/main/crates/s3_key/src/static_file_key.rs), generates an S3 presigned URL, and returns a PutFileResponse containing the upload_url and file_id.

2. Direct S3 Upload — The client uploads binary data directly to the presigned URL, bypassing the application servers to handle large payloads efficiently. URLs expire after 15 minutes by default.

3. Edge Caching and Delivery — File retrieval occurs via https://<cdn-domain>/file/<file_id>. CloudFront serves cached objects directly when available, pulling from the S3 origin only on cache misses.

4. Dynamic Image Transformation — Requests containing query parameters like ?size=300 trigger the viewer-request function (image-url-rewrite) which rewrites the URI to /size=<value> (lines 22‑33 in distribution.ts). CloudFront attempts S3 first; upon receiving a 403 (indicating the transformed version does not exist), it fails over to the image optimizer Lambda. The Lambda generates the resized asset, stores it in S3, and returns the optimized image. Subsequent requests hit the edge cache directly without invoking Lambda again.

Security Implementation

Security relies on modern AWS paradigms rather than public bucket configurations. Origin Access Controls (OAC) replace legacy origin access identities, establishing cryptographically secure relationships between CloudFront and S3. The bucket policy explicitly denies access unless the AWS:SourceArn matches the specific CloudFront distribution ARN, preventing direct S3 URL access.

Container security includes dedicated security groups for the ALB and ECS tasks, with the log configuration forwarding to Datadog via a log_router sidecar container. SQS event notifications fire when new objects arrive, enabling asynchronous downstream processing such as search indexing or document analysis.

Local Development with NGINX

The repository provides local parity through an NGINX configuration that mimics CloudFront behavior. The file [infra/local/nginx/static-file-cdn.conf](https://github.com/macro-inc/macro/blob/main/infra/local/nginx/static-file-cdn.conf) proxies /file/* routes to LocalStack's S3 endpoint while applying identical CORS and security headers to the production CDN.

# Start the local development stack

docker compose up -d

# Access files through the local CDN simulator

curl http://localhost/file/123e4567-e89b-12d3-a456-426614174000

This configuration enables rapid iteration without AWS round-trips, routing requests to either the local Axum service or LocalStack depending on the path pattern.

Client Integration Examples

Uploading Files via Rust SDK

The macro_sdk crate abstracts the HTTP API defined in [packages/sdk/specs/static-files.json](https://github.com/macro-inc/macro/blob/main/packages/sdk/specs/static-files.json) (lines 13‑60).

use macro_sdk::client::MacroClient;
use macro_sdk::models::static_files::{PutFileRequest, PutFileResponse};

let client = MacroClient::new("https://api.macro.com")?;
let req = PutFileRequest {
    file_name: "avatar.png".into(),
    content_type: Some("image/png".into()),
    extension_data: None,
};

let resp: PutFileResponse = client
    .post("/api/file")
    .json(&req)
    .send()
    .await?
    .json()
    .await?;

println!("Upload to: {}", resp.upload_url);

Direct S3 Upload

curl -X PUT \
  -H "Content-Type: image/png" \
  --upload-file ./avatar.png \
  "$UPLOAD_URL"

Retrieving Optimized Images

# Original file

curl https://static-file-service.dev.macro.com/file/123e4567-e89b-12d3-a456-426614174000 -o avatar.png

# Resized to 300px width (triggers Lambda optimization on first request)

curl "https://static-file-service.dev.macro.com/file/123e4567-e89b-12d3-a456-426614174000?size=300" -o avatar_small.png

Summary

• Hybrid Storage Architecture: The service combines S3 for durable storage, DynamoDB for metadata indexing, and ECS Fargate for API logic, creating a separation of concerns that optimizes cost and performance.
• Origin Access Controls: Modern OAC implementation ensures S3 objects remain private while allowing CloudFront edge locations to serve content securely.
• Intelligent Caching: The origin group pattern with CloudFront cache behaviors minimizes Lambda invocations by persisting transformed images back to S3 and serving subsequent requests from the edge.
• Developer Parity: Local NGINX configuration in infra/local/nginx/static-file-cdn.conf provides identical routing logic for local development, ensuring consistent behavior across environments.

Frequently Asked Questions

How does the image optimization handle concurrent requests for the same image size?

The architecture leverages CloudFront's cache behaviors and the origin group's fail-over mechanism. When multiple concurrent requests arrive for a non-existent transformed image, CloudFront forwards one request to the Lambda optimizer while queueing others. The Lambda writes the transformed image to a deterministic S3 key (file/<uuid>/<format>) before returning. Subsequent requests hit the S3 origin directly once the object propagates, avoiding redundant Lambda executions.

What prevents direct access to files via S3 URLs outside the CDN?

The bucket policy in distribution.ts (lines 90‑107) implements a strict AWS:SourceArn condition that validates requests originate from the specific CloudFront distribution ARN. Combined with Origin Access Controls (OAC), this ensures the S3 bucket rejects any direct HTTP requests lacking the cryptographic signatures provided by CloudFront's edge infrastructure.

How does the local NGINX configuration mirror production behavior?

The static-file-cdn.conf file replicates CloudFront's path routing logic by proxying /file/* requests to LocalStack's S3 implementation while injecting identical CORS headers and security policies. Other paths route to the local Axum service, creating a faithful simulation of the production request flow without requiring AWS credentials or network latency.

Where is the S3 key generation logic implemented?

Deterministic key construction resides in [crates/s3_key/src/static_file_key.rs](https://github.com/macro-inc/macro/blob/main/crates/s3_key/src/static_file_key.rs). This shared library ensures consistent key formats between the upload presigning logic in the ECS service and the transformation logic in the Lambda optimizer, preventing path mismatches that could break file retrieval.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}