How Lambda Triggers Power Async DOCX and PDF Processing in Macro

Macro uses a multi-stage AWS Lambda pipeline triggered by S3 events and SQS messages to process DOCX and PDF documents asynchronously, decoupling upload handling from compute-intensive conversion work.

The macro-inc/macro repository implements a serverless document processing architecture that handles Microsoft Word and PDF files without blocking user uploads. This article explains how Lambda triggers orchestrate the async pipeline using actual source code from the infrastructure stack.

The Async Processing Pipeline Overview

Macro's document processing flow spans four distinct trigger points across two buckets and two queues. Each stage uses Lambda triggers to hand off work without synchronous dependencies:

Stage	Trigger Source	Handler	Responsibility
1	S3 ObjectCreated (DOCX bucket)	DocxUnzipHandlerLambda	Unzip DOCX archives, enqueue conversion jobs
2	S3 ObjectCreated (upload bucket)	UploadExtractorLambdaHandler	Extract text, generate previews, route to convert queue
3	SQS message (convert queue)	ConvertService (Fargate)	Convert DOCX to PDF using pdfium
4	Lambda invoke (completion)	JobUpdateHandlerLambda	Update database, notify clients

The pipeline's asynchronous design ensures that slow conversion tasks never block fast upload responses.

Stage 1: DOCX Unzip Lambda Triggered by S3 Upload

When users upload .docx files to the dedicated DOCX bucket, S3 emits an ObjectCreated event configured via Pulumi in [infra/stacks/cloud-storage-service/docx-unzip-handler-lambda.ts](https://github.com/macro-inc/macro/blob/main/infra/stacks/cloud-storage-service/docx-unzip-handler-lambda.ts).

Lambda IAM Permissions

The DocxUnzipHandlerLambda requires S3 read access for the source bucket and SQS send permissions for the convert queue:

// S3 access policy (lines 58-85)
const s3Policy = new aws.iam.Policy(`${BASE_NAME}-s3-policy`, {
  policy: pulumi.output({
    Version: '2012-10-17',
    Statement: [{
      Effect: 'Allow',
      Action: ['s3:ListBucket', 's3:GetObject', 's3:PutObject'],
      Resource: [docxUploadBucketArn, `${docxUploadBucketArn}/*`],
    }],
  }),
  tags: this.tags,
}, { parent: this });

// SQS enqueue policy (lines 111-126)
const sqsPolicy = new aws.iam.Policy(`${BASE_NAME}-sqs-policy`, {
  policy: pulumi.output({
    Version: '2012-10-17',
    Statement: [{
      Effect: 'Allow',
      Action: ['sqs:SendMessage'],
      Resource: [convertQueueArn],
    }],
  }),
  tags: this.tags,
}, { parent: this });

After unzipping the DOCX archive (a ZIP format internally), the handler pushes a conversion job onto convertQueueArn via the SQS policy. This decouples the lightweight unzip work from the heavy PDF conversion.

Stage 2: General Upload Extractor with SQS Event Source Mapping

Files uploaded to the general upload bucket—including PDFs, images, and other DOCX files—trigger the UploadExtractorLambdaHandler defined in [infra/stacks/bulk-upload/upload-extractor-lambda-handler.ts](https://github.com/macro-inc/macro/blob/main/infra/stacks/bulk-upload/upload-extractor-lambda-handler.ts).

Unlike the DOCX-specific handler, this Lambda uses an SQS event source mapping for reliable, batched processing:

// Event source mapping connects Lambda to upload queue (lines 134-142)
new aws.lambda.EventSourceMapping(`${LAMBDA_BASE_NAME}-sqs-source-mapping`, {
  eventSourceArn: uploadExtractorQueueArn,
  functionName: this.lambda.name,
  batchSize: 1,
  enabled: true,
}, { parent: this });

The batchSize: 1 configuration ensures per-document processing isolation—failures in one file don't block others in the queue.

Handler Responsibilities

• Extract raw text from PDFs, DOCX files, and images
• Generate preview thumbnails
• Route DOCX files to the convert queue when PDF conversion is enabled
• Store extracted content in the document-storage bucket

Stage 3: Convert Service Consumes SQS Messages with Fargate

The ConvertService in [infra/stacks/convert-service/service.ts](https://github.com/macro-inc/macro/blob/main/infra/stacks/convert-service/service.ts) runs as an AWS Fargate task that polls the same convertQueueArn consumed by the upload handlers.

Service IAM Configuration

The Fargate task requires broad SQS access and the ability to invoke the completion handler:

// SQS policy for queue operations (lines 79-92)
const sqsPolicy = new aws.iam.Policy(`${BASE_NAME}-sqs-policy-${stack}`, {
  name: `${BASE_NAME}-sqs-policy-${stack}`,
  policy: {
    Version: '2012-10-17',
    Statement: [{
      Action: ['sqs:*'],
      Effect: 'Allow',
      Resource: '*',
    }],
  },
  tags: this.tags,
}, { parent: this });

// Lambda invoke policy for job completion callbacks (lines 98-108)
const lambdaInvokePolicy = new aws.iam.Policy(`${BASE_NAME}-lambda-invoke-policy-${stack}`, {
  name: `${BASE_NAME}-lambda-invoke-policy-${stack}`,
  policy: {
    Version: '2012-10-17',
    Statement: [{
      Action: ['lambda:InvokeFunction'],
      Resource: [jobUpdateHandlerLambdaArn],
      Effect: 'Allow',
    }],
  },
  tags: this.tags,
}, { parent: this });

PDF Conversion with pdfium

The conversion logic uses the pdfium-render Rust crate (see [services/search_processing_service/src/parsers/pdf.rs](https://github.com/macro-inc/macro/blob/main/services/search_processing_service/src/parsers/pdf.rs)) to render DOCX content to PDF pages. Running this in Fargate rather than Lambda avoids the 15-minute execution limit and provides consistent memory for large documents.

Stage 4: Job Update Handler Receives Completion Callbacks

After conversion finishes, the ConvertService invokes the JobUpdateHandlerLambda via the lambda:InvokeFunction permission granted above. This final handler:

• Updates document status in Macro's database
• Emits CloudWatch metrics for monitoring
• Pushes real-time updates through the WebSocket gateway to connected clients

The jobUpdateHandlerLambdaArn parameter flows through multiple stack layers, originating in the cloud-storage service configuration.

Summary

• S3 event triggers launch the DOCX unzip Lambda immediately upon upload, with IAM policies restricting access to specific buckets and the convert queue.
• SQS event source mappings enable reliable, asynchronous processing for the general upload extractor, with configurable batch sizes for failure isolation.
• Fargate services consume from the same SQS queue to perform compute-intensive PDF conversion, avoiding Lambda timeout constraints.
• Lambda invocation completes the pipeline by updating database records and notifying clients.

Each trigger type—S3 events, SQS mappings, and direct Lambda invokes—serves a specific purpose in Macro's async document processing architecture.

Frequently Asked Questions

What triggers the DOCX unzip Lambda in Macro?

The DocxUnzipHandlerLambda is triggered by S3 ObjectCreated events on the dedicated DOCX upload bucket. Pulumi configures this notification in [docx-unzip-handler-lambda.ts](https://github.com/macro-inc/macro/blob/main/infra/stacks/cloud-storage-service/docx-unzip-handler-lambda.ts), and the Lambda's IAM role includes s3:GetObject permissions for that specific bucket plus sqs:SendMessage for the convert queue.

Why does Macro use SQS instead of direct Lambda-to-Lambda calls?

SQS provides decoupling and durability: if the convert service is temporarily unavailable, messages remain in the queue rather than causing invocation failures. The EventSourceMapping in the upload extractor also enables batch control (batchSize: 1) and automatic retry with exponential backoff, which direct invocations lack.

How does the convert service handle large DOCX files that exceed Lambda limits?

The ConvertService runs as a Fargate task rather than Lambda, eliminating the 15-minute execution timeout and 10 GB memory ceiling. It consumes the same SQS queue as the Lambda handlers but processes messages with container resources suitable for pdfium rendering of large documents.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}