Image Optimizer Service S3 Integration in Macro: CloudFront Lambda Origin Pattern

Macro implements on-the-fly image optimization using a CloudFront origin group where S3 serves as the primary origin and a custom Lambda function acts as a secondary origin, automatically caching transformed images back to S3 for subsequent requests.

The Macro repository utilizes a serverless architecture to deliver optimized images without pre-rendering millions of variants. By combining Amazon CloudFront, S3, and Lambda, the system generates image transformations—such as format conversion and resizing—on demand while storing results durably in S3. This pattern minimizes compute costs by ensuring that only the first request for a specific transformation invokes the Lambda function, with all subsequent requests served directly from the S3 bucket.

CloudFront Origin Group Fail-Over Architecture

The core of Macro's image optimizer relies on a CloudFront origin group configured in infra/stacks/static-file-service/distribution.ts. The group establishes S3 as the primary origin and the Lambda Function URL as the secondary, creating an automatic fail-over pattern.

When a client requests a transformed image using a URL pattern like /file/{uuid}/format=webp,width=300, CloudFront first attempts to fetch the object from S3. Since the transformed variant does not yet exist, S3 returns a 403 Forbidden response. CloudFront interprets this as a cache miss and automatically routes the request to the secondary origin—the Image Optimizer Lambda.

This origin group configuration ensures:

• Zero cold-start latency for cached images (S3 serves directly)
• Automatic processing for uncached variants (Lambda invoked only when needed)
• Deterministic cache keys based on the full URL path including transformation parameters

IAM Role and S3 Access Policies

The Lambda function operates under a dedicated IAM role defined in infra/stacks/static-file-service/static-file-service.ts. The role <service>-image-optimizer-role trusts the Lambda service principal and attaches the AWS-managed AWSLambdaBasicExecutionRole for basic CloudWatch logging.

An inline policy grants least-privilege access to the S3 bucket:

new aws.iam.RolePolicy(`${SERVICE_NAME}-image-optimizer-s3-policy`, {
  role: imageOptimizerRole.id,
  policy: staticFilesBucket.arn.apply(bucketArn =>
    JSON.stringify({
      Version: "2012-10-17",
      Statement: [
        { Effect: "Allow", Action: "s3:GetObject", Resource: `${bucketArn}/*` },
        { Effect: "Allow", Action: "s3:PutObject", Resource: `${bucketArn}/file/*/*` },
      ],
    })
  ),
});

The permissions are scoped specifically:

• s3:GetObject on the entire bucket allows reading original source images
• s3:PutObject restricted to the file/*/* prefix ensures transformed images are written only to designated cache locations

Lambda Function Configuration

The Image Optimizer Lambda is defined as a provided.al2023 runtime function, compiled as a custom binary and packaged at target/lambda/image_optimizer/bootstrap.zip. This Rust-based implementation (implied by the bootstrap naming convention) receives the target bucket name via the BUCKET environment variable.

const imageOptimizerLambda = new aws.lambda.Function(
  `${SERVICE_NAME}-image-optimizer`,
  {
    name: `${SERVICE_NAME}-image-optimizer-${stack}`,
    role: imageOptimizerRole.arn,
    handler: "bootstrap",
    runtime: "provided.al2023",
    code: new pulumi.asset.FileArchive(`${REPO_ROOT}/target/lambda/image_optimizer/bootstrap.zip`),
    timeout: 30,
    memorySize: 1536,
    environment: { variables: { BUCKET: STATIC_FILE_BUCKET } },
    tags: this.tags,
  },
  { parent: this }
);

A FunctionUrl with AWS_IAM authorization type exposes the Lambda to CloudFront without requiring an API Gateway:

const imageOptimizerUrl = new aws.lambda.FunctionUrl(
  `${SERVICE_NAME}-image-optimizer-url`,
  {
    functionName: imageOptimizerLambda.name,
    authorizationType: "AWS_IAM",
  },
  { parent: this }
);

CloudFront Distribution Wiring

The StaticFileCloudFront construct in infra/stacks/static-file-service/static-file-service.ts binds all components together. This construct receives both the S3 bucket reference and the imageOptimizerUrl, configuring the distribution with the origin group logic.

The distribution instantiation passes the Lambda's function name and URL:

const distribution = new StaticFileCloudFront(
  `static-files-${stack}`,
  {
    bucket: staticFilesBucket,
    imageOptimizerUrl: imageOptimizerUrl.functionUrl,
    imageOptimizerFunctionName: imageOptimizerLambda.name,
    // …custom domain, not-found page, and other settings…
  },
  { dependsOn: notFound }
);

Inside the StaticFileCloudFront construct (defined in distribution.ts), cache behaviors are configured to route traffic based on path patterns, ensuring that requests matching the image optimizer signature trigger the origin group logic.

End-to-End Request Flow

The complete lifecycle of an image optimization request follows this sequence:

1. Client Request: Browser requests https://static-files.macro.com/file/123e4567-e89b-12d3-a456-426614174000/format=webp,width=300
2. S3 Check: CloudFront forwards to S3 primary origin; object does not exist, returning 403
3. Lambda Invocation: CloudFront fails over to the Lambda origin via the Function URL
4. Transformation: Lambda downloads the original image from S3 using GetObject, performs the transformation (WebP conversion + resize), and streams the result
5. Cache Write: Lambda writes the transformed image to S3 at the exact key file/123e4567-e89b-12d3-a456-426614174000/format=webp,width=300 using PutObject
6. Client Response: Lambda returns the image bytes to CloudFront, which delivers to the client and caches at the edge
7. Subsequent Requests: Future requests for the same URL hit the now-existing S3 object directly, bypassing Lambda entirely

The runtime code in services/static_file_service/src/api/event/s3_create.rs specifically filters out optimizer-specific keys (file/{uuid}/format=webp,width=300) from S3 event processing, preventing the system from treating generated thumbnails as new uploads requiring further processing.

Summary

• Macro's image optimizer service uses a CloudFront origin group with S3 as primary and Lambda as secondary to minimize compute costs.
• IAM roles in static-file-service.ts grant the Lambda scoped permissions to read originals and write transformed images to specific S3 prefixes.
• Lambda configuration uses the provided.al2023 runtime with a compiled binary, exposed via a Function URL with IAM authorization.
• Deterministic caching ensures transformed images are stored at predictable S3 keys, allowing subsequent requests to bypass the Lambda entirely.
• Runtime filters prevent the system from processing generated thumbnails as new source files.

Frequently Asked Questions

How does Macro handle the first request for an image transformation?

When a client requests a transformed image that does not exist in S3, CloudFront receives a 403 response from the primary S3 origin and automatically fails over to the Image Optimizer Lambda. The Lambda downloads the original, applies the requested transformations (format, dimensions), writes the result to the same S3 path that was originally requested, and returns the image to the client. Subsequent requests hit the cached object directly in S3.

What permissions does the Image Optimizer Lambda need for S3 integration?

According to infra/stacks/static-file-service/static-file-service.ts, the Lambda requires s3:GetObject permission on the entire bucket to read source images, and s3:PutObject permission restricted to the file/*/* prefix to store transformed outputs. The role also attaches AWSLambdaBasicExecutionRole for CloudWatch logging.

Why does Macro use a Lambda Function URL instead of API Gateway?

The implementation uses a FunctionUrl with AWS_IAM authorization (defined in static-file-service.ts lines 123-130) because it provides a direct HTTPS endpoint for the Lambda without the cost or complexity of API Gateway. CloudFront integrates directly with this URL as a custom origin, leveraging the origin group pattern to handle the S3-Lambda fail-over logic natively.

How does the system prevent infinite processing loops for generated images?

The runtime code in services/static_file_service/src/api/event/s3_create.rs specifically filters S3 create events to exclude keys matching the optimizer pattern file/{uuid}/format=webp,width=300. This ensures that when the Lambda writes a transformed image back to S3, the system does not treat it as a new upload requiring further optimization.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}