# How Macro Handles Data Retention and Cleanup with SFS Deletion Handlers

> Discover how Macro manages data retention and cleanup using SFS deletion handlers. Learn about its two-stage pipeline for efficient file deletion and database synchronization.

- Repository: [Macro/macro](https://github.com/macro-inc/macro)
- Tags: internals
- Published: 2026-08-15

---

**Data retention and cleanup in Macro's Static File Service is handled by a two-stage pipeline: an AWS Lambda handler scans for orphaned files and enqueues delete requests to SQS, while a separate worker process consumes those messages to remove files from storage and synchronize the database.**

The Macro codebase implements a robust garbage collection system for binary email attachments. Rather than storing large files directly in PostgreSQL, the system uses the **Static File Service (SFS)** for object storage. This architecture requires careful coordination between database state and external storage to prevent orphaned files from accumulating indefinitely.

## Architecture Overview

The cleanup system spans two distinct services that communicate through **Amazon SQS**:

1. **Scheduled Lambda (`email_sfs_delete_handler`)** — runs periodically to detect orphaned records
2. **Persistent worker (`email_service`)** — continuously processes delete requests

This decoupled design ensures reliable delivery of deletion commands even when the SFS API experiences transient failures.

## How the Lambda Handler Detects Orphaned Files

The entry point resides in [[`services/email_sfs_delete_handler/src/handler.rs`](https://github.com/macro-inc/macro/blob/main/services/email_sfs_delete_handler/src/handler.rs)](https://github.com/macro-inc/macro/blob/main/services/email_sfs_delete_handler/src/handler.rs). The `handler()` function queries the `email_attachments_sfs` table for rows where `attachment_id IS NULL` — indicating the parent email no longer references this file.

```rust
#[tracing::instrument(skip_all, err)]
pub async fn handler(
    ctx: context::Context,
    _event: LambdaEvent<EventBridgeEvent>,
) -> Result<(), Error> {
    let messages = sqlx::query_as!(
        SFSDeleteMessage,
        r#"
        SELECT
            id as "db_id", sfs_id
        FROM email_attachments_sfs
        WHERE attachment_id IS NULL
        "#,
    )
    .fetch_all(&ctx.db)
    .await
    .unwrap_or_else(|e| {
        tracing::error!(error=?e, "Error fetching orphaned sfs attachments");
        Vec::new()
    });

    // Log results for observability
    if !messages.is_empty() {
        tracing::info!(count = messages.len(), "Sending sfs delete messages");
    } else {
        tracing::info!("No orphaned sfs attachments found");
    }

    Ok(())
}

```

The query operates on the following schema logic: when an email attachment is created, a row in `email_attachments_sfs` links the database record to the SFS object via `sfs_id`. If the email or attachment is deleted, `attachment_id` becomes `NULL` through foreign key constraints or explicit cleanup — flagging the row for the deletion handler.

## Enqueuing Delete Messages to SQS

After identifying orphans, the handler pushes **delete messages** to a dedicated SQS queue. The [[`sqs_client/src/email.rs`](https://github.com/macro-inc/macro/blob/main/sqs_client/src/email.rs)](https://github.com/macro-inc/macro/blob/main/sqs_client/src/email.rs) module defines `SFSDeleteMessage`:

```rust
pub struct SFSDeleteMessage {
    pub db_id: Uuid,  // Row ID in email_attachments_sfs
    pub sfs_id: String,  // External SFS object identifier
}

```

The enqueue loop in [`handler.rs`](https://github.com/macro-inc/macro/blob/main/handler.rs) logs per-message failures without halting the batch:

```rust
for message in messages.into_iter() {
    let db_id = message.db_id;
    let sfs_id = message.sfs_id;
    if let Err(e) = ctx.sqs_client.enqueue_sfs_delete_message(message).await {
        tracing::error!(
            error = ?e,
            db_id = %db_id,
            sfs_id = %sfs_id,
            "Error enqueueing sfs delete message",
        );
    };
}

```

Error isolation here is critical — a single malformed `sfs_id` must not prevent cleanup of other orphaned files.

## The SFS Delete Worker: Processing Queue Messages

The second stage runs in [[`services/email_service/src/pubsub/sfs_deleter/worker.rs`](https://github.com/macro-inc/macro/blob/main/services/email_service/src/pubsub/sfs_deleter/worker.rs)](https://github.com/macro-inc/macro/blob/main/services/email_service/src/pubsub/sfs_deleter/worker.rs). This long-running task polls SQS and delegates to [[`process.rs`](https://github.com/macro-inc/macro/blob/main/process.rs)](https://github.com/macro-inc/macro/blob/main/services/email_service/src/pubsub/sfs_deleter/process.rs) for actual deletion.

```rust
pub async fn run_worker(ctx: SFSDeleteContext, mut shutdown: watch::Receiver<bool>) {
    while !*shutdown.borrow() {
        let msgs = ctx.sqs_client.receive_sfs_delete_messages().await;
        for msg in msgs.unwrap_or_default() {
            match process_delete(&ctx, msg).await {
                Ok(_) => {}
                Err(e) => tracing::error!(error=?e, "SFS delete failed"),
            }
        }
    }
}

```

The processing logic in [`process.rs`](https://github.com/macro-inc/macro/blob/main/process.rs) executes two operations atomically in sequence:

1. **Call SFS API** to delete the physical file using `sfs_id`
2. **Delete database row** to remove the tracking record

```rust
pub async fn process_delete(ctx: &SFSDeleteContext, msg: SFSDeleteMessage) -> Result<()> {
    // Step 1: Remove from external storage
    ctx.sfs_client.delete_file(&msg.sfs_id).await?;
    
    // Step 2: Synchronize database state
    sqlx::query!("DELETE FROM email_attachments_sfs WHERE id = $1", msg.db_id)
        .execute(&ctx.db)
        .await?;
    
    Ok(())
}

```

If the SFS deletion succeeds but the database deletion fails, the message remains in SQS for retry (until the visibility timeout expires). Conversely, if SFS deletion fails, the database row persists — maintaining referential integrity until the storage layer confirms removal.

## Data Retention Guarantees

The **SFS deletion handler** system provides these retention properties:

- **Automatic cleanup** — no manual intervention required for orphaned files
- **Eventual consistency** — files may persist briefly after becoming orphaned, bounded by the Lambda schedule frequency and SQS processing lag
- **Idempotent operations** — duplicate delete requests for the same `sfs_id` are safe; SFS delete operations are typically idempotent
- **Audit trail** — structured logging via `tracing` captures all deletion attempts with correlation IDs

## Comparison: Lambda vs. Worker Responsibilities

| Aspect | Lambda Handler | SFS Delete Worker |
|--------|--------------|-------------------|
| **Trigger** | EventBridge schedule (e.g., every 5 minutes) | Continuous SQS poll |
| **Primary task** | Detection (database query) | Execution (SFS API + DB cleanup) |
| **Failure handling** | Log and continue; failed enqueues reported | SQS retry with backoff; dead-letter queue support |
| **Scaling** | Provisioned concurrency for cold-start latency | Auto-scaled based on queue depth |
| **Source file** | [`handler.rs`](https://github.com/macro-inc/macro/blob/main/handler.rs) | [`worker.rs`](https://github.com/macro-inc/macro/blob/main/worker.rs), [`process.rs`](https://github.com/macro-inc/macro/blob/main/process.rs) |

## Configuration and Deployment

The Lambda handler requires:

- **Database pool** — via `context::Context` in [`services/email_sfs_delete_handler/src/context.rs`](https://github.com/macro-inc/macro/blob/main/services/email_sfs_delete_handler/src/context.rs)
- **SQS client** — shared client for enqueue operations
- **IAM permissions** — read access to `email_attachments_sfs`, send access to the delete queue

The worker service runs as part of the main `email_service` binary, requiring:

- **SQS receive/delete permissions** for the delete queue
- **SFS API credentials** for object deletion
- **Database write access** for row removal

## Summary

- The **SFS deletion handler** in Macro implements a **two-phase cleanup pipeline** separating detection from execution
- **Phase 1**: Scheduled Lambda scans `email_attachments_sfs` for `attachment_id IS NULL` and enqueues `SFSDeleteMessage` structs to SQS
- **Phase 2**: Persistent worker consumes messages, deletes files from SFS, and removes database rows to maintain consistency
- **Error isolation** at both stages prevents cascading failures and ensures partial progress
- **Structured logging** throughout enables observability of retention policy enforcement

## Frequently Asked Questions

### How does Macro prevent orphaned files when emails are deleted?

The database schema uses a foreign key relationship where `email_attachments_sfs.attachment_id` references the parent attachment. When an email or attachment is deleted, this column becomes `NULL` — either through `ON DELETE SET NULL` or application logic. The Lambda handler detects these rows and triggers cleanup.

### What happens if the SFS delete API fails?

The SQS message remains visible and returns to the queue after the visibility timeout expires. The worker retries deletion according to the queue's redrive policy. Repeated failures eventually route the message to a dead-letter queue for manual investigation.

### Can the cleanup process be monitored?

Yes. Both the Lambda handler and the worker emit structured logs via `tracing` with consistent field names: `db_id`, `sfs_id`, `count`, and `error`. These integrate with OpenTelemetry-compatible observability platforms for alerting on deletion backlogs or failure rates.

### Why use SQS instead of direct Lambda-to-worker invocation?

Decoupling via SQS provides **resilience** and **rate limiting**. The SFS API may have throughput constraints; the SQS buffer absorbs bursts of orphaned files from bulk email deletions. It also allows the worker to scale independently of Lambda concurrency limits.