How Macro Handles Data Retention and Cleanup with SFS Deletion Handlers

Data retention and cleanup in Macro's Static File Service is handled by a two-stage pipeline: an AWS Lambda handler scans for orphaned files and enqueues delete requests to SQS, while a separate worker process consumes those messages to remove files from storage and synchronize the database.

The Macro codebase implements a robust garbage collection system for binary email attachments. Rather than storing large files directly in PostgreSQL, the system uses the Static File Service (SFS) for object storage. This architecture requires careful coordination between database state and external storage to prevent orphaned files from accumulating indefinitely.

Architecture Overview

The cleanup system spans two distinct services that communicate through Amazon SQS:

1. Scheduled Lambda (email_sfs_delete_handler) — runs periodically to detect orphaned records
2. Persistent worker (email_service) — continuously processes delete requests

This decoupled design ensures reliable delivery of deletion commands even when the SFS API experiences transient failures.

How the Lambda Handler Detects Orphaned Files

The entry point resides in [services/email_sfs_delete_handler/src/handler.rs](https://github.com/macro-inc/macro/blob/main/services/email_sfs_delete_handler/src/handler.rs). The handler() function queries the email_attachments_sfs table for rows where attachment_id IS NULL — indicating the parent email no longer references this file.

#[tracing::instrument(skip_all, err)]
pub async fn handler(
    ctx: context::Context,
    _event: LambdaEvent<EventBridgeEvent>,
) -> Result<(), Error> {
    let messages = sqlx::query_as!(
        SFSDeleteMessage,
        r#"
        SELECT
            id as "db_id", sfs_id
        FROM email_attachments_sfs
        WHERE attachment_id IS NULL
        "#,
    )
    .fetch_all(&ctx.db)
    .await
    .unwrap_or_else(|e| {
        tracing::error!(error=?e, "Error fetching orphaned sfs attachments");
        Vec::new()
    });

    // Log results for observability
    if !messages.is_empty() {
        tracing::info!(count = messages.len(), "Sending sfs delete messages");
    } else {
        tracing::info!("No orphaned sfs attachments found");
    }

    Ok(())
}

The query operates on the following schema logic: when an email attachment is created, a row in email_attachments_sfs links the database record to the SFS object via sfs_id. If the email or attachment is deleted, attachment_id becomes NULL through foreign key constraints or explicit cleanup — flagging the row for the deletion handler.

Enqueuing Delete Messages to SQS

After identifying orphans, the handler pushes delete messages to a dedicated SQS queue. The [sqs_client/src/email.rs](https://github.com/macro-inc/macro/blob/main/sqs_client/src/email.rs) module defines SFSDeleteMessage:

pub struct SFSDeleteMessage {
    pub db_id: Uuid,  // Row ID in email_attachments_sfs
    pub sfs_id: String,  // External SFS object identifier
}

The enqueue loop in handler.rs logs per-message failures without halting the batch:

for message in messages.into_iter() {
    let db_id = message.db_id;
    let sfs_id = message.sfs_id;
    if let Err(e) = ctx.sqs_client.enqueue_sfs_delete_message(message).await {
        tracing::error!(
            error = ?e,
            db_id = %db_id,
            sfs_id = %sfs_id,
            "Error enqueueing sfs delete message",
        );
    };
}

Error isolation here is critical — a single malformed sfs_id must not prevent cleanup of other orphaned files.

The SFS Delete Worker: Processing Queue Messages

The second stage runs in [services/email_service/src/pubsub/sfs_deleter/worker.rs](https://github.com/macro-inc/macro/blob/main/services/email_service/src/pubsub/sfs_deleter/worker.rs). This long-running task polls SQS and delegates to [process.rs](https://github.com/macro-inc/macro/blob/main/services/email_service/src/pubsub/sfs_deleter/process.rs) for actual deletion.

pub async fn run_worker(ctx: SFSDeleteContext, mut shutdown: watch::Receiver<bool>) {
    while !*shutdown.borrow() {
        let msgs = ctx.sqs_client.receive_sfs_delete_messages().await;
        for msg in msgs.unwrap_or_default() {
            match process_delete(&ctx, msg).await {
                Ok(_) => {}
                Err(e) => tracing::error!(error=?e, "SFS delete failed"),
            }
        }
    }
}

The processing logic in process.rs executes two operations atomically in sequence:

1. Call SFS API to delete the physical file using sfs_id
2. Delete database row to remove the tracking record

pub async fn process_delete(ctx: &SFSDeleteContext, msg: SFSDeleteMessage) -> Result<()> {
    // Step 1: Remove from external storage
    ctx.sfs_client.delete_file(&msg.sfs_id).await?;
    
    // Step 2: Synchronize database state
    sqlx::query!("DELETE FROM email_attachments_sfs WHERE id = $1", msg.db_id)
        .execute(&ctx.db)
        .await?;
    
    Ok(())
}

If the SFS deletion succeeds but the database deletion fails, the message remains in SQS for retry (until the visibility timeout expires). Conversely, if SFS deletion fails, the database row persists — maintaining referential integrity until the storage layer confirms removal.

Data Retention Guarantees

The SFS deletion handler system provides these retention properties:

• Automatic cleanup — no manual intervention required for orphaned files
• Eventual consistency — files may persist briefly after becoming orphaned, bounded by the Lambda schedule frequency and SQS processing lag
• Idempotent operations — duplicate delete requests for the same sfs_id are safe; SFS delete operations are typically idempotent
• Audit trail — structured logging via tracing captures all deletion attempts with correlation IDs

Comparison: Lambda vs. Worker Responsibilities

Aspect	Lambda Handler	SFS Delete Worker
Trigger	EventBridge schedule (e.g., every 5 minutes)	Continuous SQS poll
Primary task	Detection (database query)	Execution (SFS API + DB cleanup)
Failure handling	Log and continue; failed enqueues reported	SQS retry with backoff; dead-letter queue support
Scaling	Provisioned concurrency for cold-start latency	Auto-scaled based on queue depth
Source file	handler.rs	worker.rs, process.rs

Configuration and Deployment

The Lambda handler requires:

• Database pool — via context::Context in services/email_sfs_delete_handler/src/context.rs
• SQS client — shared client for enqueue operations
• IAM permissions — read access to email_attachments_sfs, send access to the delete queue

The worker service runs as part of the main email_service binary, requiring:

• SQS receive/delete permissions for the delete queue
• SFS API credentials for object deletion
• Database write access for row removal

Summary

• The SFS deletion handler in Macro implements a two-phase cleanup pipeline separating detection from execution
• Phase 1: Scheduled Lambda scans email_attachments_sfs for attachment_id IS NULL and enqueues SFSDeleteMessage structs to SQS
• Phase 2: Persistent worker consumes messages, deletes files from SFS, and removes database rows to maintain consistency
• Error isolation at both stages prevents cascading failures and ensures partial progress
• Structured logging throughout enables observability of retention policy enforcement

Frequently Asked Questions

How does Macro prevent orphaned files when emails are deleted?

The database schema uses a foreign key relationship where email_attachments_sfs.attachment_id references the parent attachment. When an email or attachment is deleted, this column becomes NULL — either through ON DELETE SET NULL or application logic. The Lambda handler detects these rows and triggers cleanup.

What happens if the SFS delete API fails?

The SQS message remains visible and returns to the queue after the visibility timeout expires. The worker retries deletion according to the queue's redrive policy. Repeated failures eventually route the message to a dead-letter queue for manual investigation.

Can the cleanup process be monitored?

Yes. Both the Lambda handler and the worker emit structured logs via tracing with consistent field names: db_id, sfs_id, count, and error. These integrate with OpenTelemetry-compatible observability platforms for alerting on deletion backlogs or failure rates.

Why use SQS instead of direct Lambda-to-worker invocation?

Decoupling via SQS provides resilience and rate limiting. The SFS API may have throughput constraints; the SQS buffer absorbs bursts of orphaned files from bulk email deletions. It also allows the worker to scale independently of Lambda concurrency limits.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}