How Macro Uses AWS Lambda Handlers for Async Processing: Docx Unzip & Email Suppression

Macro implements asynchronous document and email processing using AWS Lambda functions deployed as Pulumi ComponentResources, encapsulating VPC networking, typed environment variables, and fine-grained IAM policies within a reusable Lambda<TEnv> wrapper.

The macro-inc/macro repository deploys Rust-based async workers through a standardized infrastructure pattern that separates compute packaging from permission configuration. This approach enables the platform to process DOCX extractions and email suppressions serverlessly while maintaining strict security boundaries and comprehensive observability.

The Lambda Component Architecture

At the core of Macro's async processing strategy sits a generic Pulumi component defined in infra/packages/lambda/src/lambda.ts. The Lambda<TEnv> class extends ComponentResource and handles common concerns including code packaging, VPC attachment, and execution role binding.

Handler Packaging and Deployment

Each Lambda handler compiles from Rust source into a bootstrap.zip artifact. The Pulumi configuration points to these pre-built archives rather than embedding source code directly. For example, the Docx Unzip handler resides in services/docx_unzip_handler/, while the Email Suppression handler lives in services/email_suppression_handler/.

The infrastructure code references these artifacts through the zipLocation parameter, allowing the Lambda runtime to load the compiled Rust binary as a custom runtime.

VPC Integration and Network Access

Both handlers execute within Macro's private VPC, granting secure access to PostgreSQL databases, Redis clusters, and private S3 endpoints. The Lambda constructor accepts a vpc argument that supplies subnet IDs and security group configurations:

// Simplified configuration showing VPC attachment
new Lambda<DocxUnzipLambdaEnvVars>(`${BASE_NAME}-lambda`, {
  baseName: BASE_NAME,
  handlerBase: HANDLER_BASE,
  zipLocation: ZIP_LOCATION,
  vpc, // VPC configuration for private resource access
  envVars,
  role: this.role,
});

Docx Unzip Handler Implementation

The DOCX extraction worker, defined in infra/stacks/cloud-storage-service/docx-unzip-handler-lambda.ts, processes document uploads asynchronously. This handler requires specific permissions to interact with storage and queueing systems.

IAM Policies and Permissions

The Docx Unzip Lambda operates with a dedicated IAM role granting three specific capabilities:

• S3 Access: Read/write permissions on the document storage bucket for retrieving uploaded files and storing extracted content
• SQS Permissions: Access to send messages to the conversion SQS queue for downstream processing
• Cross-Lambda Invocation: Permission to invoke the job-update Lambda for status reporting

Resource Configuration

The handler runs with 256 MiB of memory and receives environment variables through the DocxUnzipLambdaEnvVars interface, ensuring type-safe access to database URLs and Redis connection strings:

// infra/stacks/cloud-storage-service/docx-unzip-handler-lambda.ts
const docxUnzipLambda = new Lambda<DocxUnzipLambdaEnvVars>(
  `${BASE_NAME}-lambda`,
  {
    baseName: BASE_NAME,
    handlerBase: HANDLER_BASE,
    zipLocation: ZIP_LOCATION,
    vpc,
    envVars,
    role: this.role,
    memorySize: 256,
    tags: this.tags,
  },
  { parent: this }
);

Email Suppression Handler Implementation

The email management worker, configured in infra/stacks/email-suppression/email-suppression.ts, handles list cleaning and bounce processing with stricter concurrency controls to protect downstream services.

Reserved Concurrency Configuration

Unlike the Docx handler, the Email Suppression Lambda implements reserved concurrent executions to limit throughput and prevent overwhelming the production database. The configuration applies environment-specific limits:

• Production: 50 concurrent executions
• Development/Staging: 10 concurrent executions

// infra/stacks/email-suppression/email-suppression.ts
const emailSuppressionLambda = new Lambda<EmailSuppressionLambdaEnvVars>(
  `${BASE_NAME}-lambda`,
  {
    baseName: BASE_NAME,
    handlerBase: HANDLER_BASE,
    zipLocation: ZIP_LOCATION,
    vpc,
    envVars,
    role: this.role,
    reservedConcurrentExecutions: stack === 'prod' ? 50 : 10,
    tags: this.tags,
  },
  { parent: this }
);

This Lambda requires only the default AWS Lambda execution policies, reflecting its narrower scope of operations compared to the document processing pipeline.

Monitoring and Alerting

Both handlers register CloudWatch alarms automatically through the Pulumi component. The infrastructure creates:

• Throttling alarms (throttle-alarm) triggered when invocation limits approach capacity
• Error rate alarms (error-alarm) detecting function failures and runtime exceptions

These alarms publish to Macro's central CloudTrail SNS topic, enabling immediate notification of processing delays or service degradation.

Summary

• Macro deploys async Rust handlers using a reusable Lambda<TEnv> Pulumi component located in infra/packages/lambda/src/lambda.ts, standardizing VPC attachment and IAM configuration.

• Handler code compiles to bootstrap.zip artifacts stored in service directories (services/docx_unzip_handler/ and services/email_suppression_handler/), deployed as custom runtime Lambda functions.

• Fine-grained IAM policies grant the Docx Unzip handler specific S3, SQS, and Lambda invocation permissions, while the Email Suppression handler operates with minimal default privileges.

• Reserved concurrency limits protect the production environment by capping the Email Suppression handler at 50 concurrent executions in production and 10 in other environments.

• CloudWatch alarms for throttling and errors wire into the central CloudTrail SNS topic, providing automated monitoring for both async processing pipelines.

Frequently Asked Questions

What is the Lambda<TEnv> wrapper in Macro?

The Lambda<TEnv> class is a generic Pulumi ComponentResource defined in infra/packages/lambda/src/lambda.ts that encapsulates AWS Lambda creation logic. It accepts typed environment variables through a generic parameter, ensuring that infrastructure code passes the correct configuration to Rust handlers while handling VPC networking, IAM role attachment, and monitoring setup consistently across all async workers.

How does Macro handle VPC networking for Lambda handlers?

Both the Docx Unzip and Email Suppression handlers execute inside Macro's private VPC by passing a vpc configuration object to the Lambda constructor. This placement grants the functions private network access to PostgreSQL databases, Redis clusters, and internal S3 endpoints without traversing the public internet, with subnet and security group definitions managed centrally in the Pulumi stack configuration.

Why does the Email Suppression handler use reserved concurrency?

The Email Suppression handler configures reservedConcurrentExecutions (set to 50 in production and 10 elsewhere) to prevent cascade failures when processing large suppression lists. This limit throttles the Lambda's maximum simultaneous executions, protecting downstream database connections and third-party email service APIs from being overwhelmed by sudden traffic spikes or batch processing operations.

What permissions does the Docx Unzip Lambda require?

According to the source code in infra/stacks/cloud-storage-service/docx-unzip-handler-lambda.ts, the Docx Unzip Lambda requires three specific IAM permissions beyond standard execution roles: S3 read/write access for document storage operations, SQS send permissions for the conversion queue, and Lambda invocation rights to trigger the job-update function for reporting extraction status.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}