# How Lambda Triggers Power Async DOCX and PDF Processing in Macro

> Discover how Macro leverages Lambda triggers with S3 and SQS for efficient, asynchronous DOCX and PDF document processing, decoupling uploads from conversions.

- Repository: [Macro/macro](https://github.com/macro-inc/macro)
- Tags: how-to-guide
- Published: 2026-08-14

---

**Macro uses a multi-stage AWS Lambda pipeline triggered by S3 events and SQS messages to process DOCX and PDF documents asynchronously, decoupling upload handling from compute-intensive conversion work.**

The [macro-inc/macro](https://github.com/macro-inc/macro) repository implements a serverless document processing architecture that handles Microsoft Word and PDF files without blocking user uploads. This article explains how Lambda triggers orchestrate the async pipeline using actual source code from the infrastructure stack.

## The Async Processing Pipeline Overview

Macro's document processing flow spans four distinct trigger points across two buckets and two queues. Each stage uses **Lambda triggers** to hand off work without synchronous dependencies:

| Stage | Trigger Source | Handler | Responsibility |
|-------|---------------|---------|--------------|
| 1 | S3 `ObjectCreated` (DOCX bucket) | `DocxUnzipHandlerLambda` | Unzip DOCX archives, enqueue conversion jobs |
| 2 | S3 `ObjectCreated` (upload bucket) | `UploadExtractorLambdaHandler` | Extract text, generate previews, route to convert queue |
| 3 | SQS message (convert queue) | `ConvertService` (Fargate) | Convert DOCX to PDF using `pdfium` |
| 4 | Lambda invoke (completion) | `JobUpdateHandlerLambda` | Update database, notify clients |

The pipeline's **asynchronous design** ensures that slow conversion tasks never block fast upload responses.

## Stage 1: DOCX Unzip Lambda Triggered by S3 Upload

When users upload `.docx` files to the dedicated DOCX bucket, S3 emits an `ObjectCreated` event configured via Pulumi in [[`infra/stacks/cloud-storage-service/docx-unzip-handler-lambda.ts`](https://github.com/macro-inc/macro/blob/main/infra/stacks/cloud-storage-service/docx-unzip-handler-lambda.ts)](https://github.com/macro-inc/macro/blob/main/infra/stacks/cloud-storage-service/docx-unzip-handler-lambda.ts).

### Lambda IAM Permissions

The `DocxUnzipHandlerLambda` requires **S3 read access** for the source bucket and **SQS send permissions** for the convert queue:

```ts
// S3 access policy (lines 58-85)
const s3Policy = new aws.iam.Policy(`${BASE_NAME}-s3-policy`, {
  policy: pulumi.output({
    Version: '2012-10-17',
    Statement: [{
      Effect: 'Allow',
      Action: ['s3:ListBucket', 's3:GetObject', 's3:PutObject'],
      Resource: [docxUploadBucketArn, `${docxUploadBucketArn}/*`],
    }],
  }),
  tags: this.tags,
}, { parent: this });

// SQS enqueue policy (lines 111-126)
const sqsPolicy = new aws.iam.Policy(`${BASE_NAME}-sqs-policy`, {
  policy: pulumi.output({
    Version: '2012-10-17',
    Statement: [{
      Effect: 'Allow',
      Action: ['sqs:SendMessage'],
      Resource: [convertQueueArn],
    }],
  }),
  tags: this.tags,
}, { parent: this });

```

After unzipping the DOCX archive (a ZIP format internally), the handler pushes a conversion job onto `convertQueueArn` via the SQS policy. This **decouples the lightweight unzip work from the heavy PDF conversion**.

## Stage 2: General Upload Extractor with SQS Event Source Mapping

Files uploaded to the general upload bucket—including PDFs, images, and other DOCX files—trigger the `UploadExtractorLambdaHandler` defined in [[`infra/stacks/bulk-upload/upload-extractor-lambda-handler.ts`](https://github.com/macro-inc/macro/blob/main/infra/stacks/bulk-upload/upload-extractor-lambda-handler.ts)](https://github.com/macro-inc/macro/blob/main/infra/stacks/bulk-upload/upload-extractor-lambda-handler.ts).

Unlike the DOCX-specific handler, this Lambda uses an **SQS event source mapping** for reliable, batched processing:

```ts
// Event source mapping connects Lambda to upload queue (lines 134-142)
new aws.lambda.EventSourceMapping(`${LAMBDA_BASE_NAME}-sqs-source-mapping`, {
  eventSourceArn: uploadExtractorQueueArn,
  functionName: this.lambda.name,
  batchSize: 1,
  enabled: true,
}, { parent: this });

```

The `batchSize: 1` configuration ensures **per-document processing isolation**—failures in one file don't block others in the queue.

### Handler Responsibilities

- Extract raw text from PDFs, DOCX files, and images
- Generate preview thumbnails
- Route DOCX files to the convert queue when PDF conversion is enabled
- Store extracted content in the document-storage bucket

## Stage 3: Convert Service Consumes SQS Messages with Fargate

The `ConvertService` in [[`infra/stacks/convert-service/service.ts`](https://github.com/macro-inc/macro/blob/main/infra/stacks/convert-service/service.ts)](https://github.com/macro-inc/macro/blob/main/infra/stacks/convert-service/service.ts) runs as an **AWS Fargate task** that polls the same `convertQueueArn` consumed by the upload handlers.

### Service IAM Configuration

The Fargate task requires broad SQS access and the ability to invoke the completion handler:

```ts
// SQS policy for queue operations (lines 79-92)
const sqsPolicy = new aws.iam.Policy(`${BASE_NAME}-sqs-policy-${stack}`, {
  name: `${BASE_NAME}-sqs-policy-${stack}`,
  policy: {
    Version: '2012-10-17',
    Statement: [{
      Action: ['sqs:*'],
      Effect: 'Allow',
      Resource: '*',
    }],
  },
  tags: this.tags,
}, { parent: this });

// Lambda invoke policy for job completion callbacks (lines 98-108)
const lambdaInvokePolicy = new aws.iam.Policy(`${BASE_NAME}-lambda-invoke-policy-${stack}`, {
  name: `${BASE_NAME}-lambda-invoke-policy-${stack}`,
  policy: {
    Version: '2012-10-17',
    Statement: [{
      Action: ['lambda:InvokeFunction'],
      Resource: [jobUpdateHandlerLambdaArn],
      Effect: 'Allow',
    }],
  },
  tags: this.tags,
}, { parent: this });

```

### PDF Conversion with pdfium

The conversion logic uses the `pdfium-render` Rust crate (see [[`services/search_processing_service/src/parsers/pdf.rs`](https://github.com/macro-inc/macro/blob/main/services/search_processing_service/src/parsers/pdf.rs)](https://github.com/macro-inc/macro/blob/main/services/search_processing_service/src/parsers/pdf.rs)) to render DOCX content to PDF pages. Running this in **Fargate rather than Lambda** avoids the 15-minute execution limit and provides consistent memory for large documents.

## Stage 4: Job Update Handler Receives Completion Callbacks

After conversion finishes, the `ConvertService` invokes the **`JobUpdateHandlerLambda`** via the `lambda:InvokeFunction` permission granted above. This final handler:

- Updates document status in Macro's database
- Emits CloudWatch metrics for monitoring
- Pushes real-time updates through the WebSocket gateway to connected clients

The `jobUpdateHandlerLambdaArn` parameter flows through multiple stack layers, originating in the cloud-storage service configuration.

## Summary

- **S3 event triggers** launch the DOCX unzip Lambda immediately upon upload, with IAM policies restricting access to specific buckets and the convert queue.
- **SQS event source mappings** enable reliable, asynchronous processing for the general upload extractor, with configurable batch sizes for failure isolation.
- **Fargate services** consume from the same SQS queue to perform compute-intensive PDF conversion, avoiding Lambda timeout constraints.
- **Lambda invocation** completes the pipeline by updating database records and notifying clients.

Each trigger type—S3 events, SQS mappings, and direct Lambda invokes—serves a specific purpose in Macro's async document processing architecture.

## Frequently Asked Questions

### What triggers the DOCX unzip Lambda in Macro?

The `DocxUnzipHandlerLambda` is triggered by **S3 `ObjectCreated` events** on the dedicated DOCX upload bucket. Pulumi configures this notification in [[`docx-unzip-handler-lambda.ts`](https://github.com/macro-inc/macro/blob/main/docx-unzip-handler-lambda.ts)](https://github.com/macro-inc/macro/blob/main/infra/stacks/cloud-storage-service/docx-unzip-handler-lambda.ts), and the Lambda's IAM role includes `s3:GetObject` permissions for that specific bucket plus `sqs:SendMessage` for the convert queue.

### Why does Macro use SQS instead of direct Lambda-to-Lambda calls?

SQS provides **decoupling and durability**: if the convert service is temporarily unavailable, messages remain in the queue rather than causing invocation failures. The `EventSourceMapping` in the upload extractor also enables **batch control** (`batchSize: 1`) and automatic retry with exponential backoff, which direct invocations lack.

### How does the convert service handle large DOCX files that exceed Lambda limits?

The `ConvertService` runs as a **Fargate task** rather than Lambda, eliminating the 15-minute execution timeout and 10 GB memory ceiling. It consumes the same SQS queue as the Lambda handlers but processes messages with container resources suitable for `pdfium` rendering of large documents.