How Macro Configures S3 Buckets for Document Storage with Lifecycle Policies

Macro uses a dedicated S3 bucket for document storage with a 30-day Glacier transition and 90-day expiration policy applied via put_bucket_lifecycle_configuration, while runtime services interact through a thin S3Client wrapper that abstracts all bucket operations.

The Macro codebase organizes document persistence around a single document storage bucket concept. Rather than embedding lifecycle logic in application code, the platform relies on AWS S3's native lifecycle management, configured at provisioning time through the AWS SDK. This separation keeps services lightweight while ensuring automatic cost optimization and data retention compliance.

Architecture Overview

Document storage in Macro follows a three-layer pattern: configuration-driven bucket naming, SDK-based lifecycle enforcement, and a client abstraction that hides S3 implementation details from business logic.

Configuration-Driven Bucket Selection

Every service that handles documents receives its bucket name through the shared configuration system. The key parameter document_storage_bucket appears across multiple service entry points.

In services/static_file_service/src/service/s3/client.rs, the S3Client struct stores this bucket name for all subsequent operations:

pub struct S3Client {
    client: aws_sdk_s3::Client,
    storage_bucket: String,
}

impl S3Client {
    pub fn new(client: aws_sdk_s3::Client, storage_bucket: String) -> Self {
        Self { client, storage_bucket }
    }
}

This pattern repeats in services/document_storage_service/src/main.rs, where the bucket name from config.document_storage_bucket initializes the service's storage backend.

Local Development: LocalStack Bucket Provisioning

For local development, Macro spins up LocalStack and creates buckets programmatically. The setup routine in tooling/xtask/crates/xtask_local/src/local/localstack.rs handles bucket creation, CORS configuration, and event notifications.

Bucket Creation Loop

for bucket in resources::BUCKETS {
    let name = bucket.name;
    s3.create_bucket()
        .bucket(name)
        .send()
        .await
        .map(|_| ())?;
}

The resources module defines DOC_STORAGE_BUCKET as one of the buckets created during this startup phase.

CORS Configuration

After creation, the code immediately applies CORS rules to allow browser-direct access:

s3.put_bucket_cors()
    .bucket(name)
    .cors_configuration(cors_cfg)
    .send()
    .await?;

This enables the frontend to generate presigned URLs and fetch objects without proxying through Macro's API servers.

S3 Event Notification

The setup registers an ObjectCreated notification that forwards to the internal event queue:

s3.put_bucket_notification_configuration()
    .bucket(resources::DOC_STORAGE_BUCKET)
    .notification_configuration(notif_cfg)
    .send()
    .await?;

This triggers downstream processing (thumbnail generation, text extraction) when new documents arrive.

Production Lifecycle Policy Configuration

While the repository source code for lifecycle management isn't directly exposed—likely managed through infrastructure-as-code outside the visible paths—the intended policy follows Macro's documented SDK specification. The lifecycle configuration uses put_bucket_lifecycle_configuration from the AWS SDK, matching the same client patterns seen in local setup.

Standard Lifecycle Rules

Rule	Trigger	Action
Glacier Transition	30 days after object creation	Move to GLACIER storage class
Permanent Deletion	90 days after object creation	Delete object entirely

This two-tier approach balances accessibility with cost: documents remain in standard storage for a month of active use, shift to cold storage for archival, then disappear completely to satisfy retention limits.

Lifecycle Configuration Implementation

The AWS SDK call follows this structure (based on Macro's SDK patterns and AWS best practices):

let lifecycle = LifecycleConfiguration {
    rules: vec![LifecycleRule {
        id: Some("auto-expire".into()),
        filter: LifecycleFilter::Prefix("documents/".into()),
        status: LifecycleRuleStatus::Enabled,
        expiration: Some(LifecycleExpiration::Days(90)),
        transitions: Some(vec![Transition {
            days: Some(30),
            storage_class: StorageClass::Glacier,
        }]),
        ..Default::default()
    }],
};

s3.put_bucket_lifecycle_configuration()
    .bucket(bucket_name)
    .lifecycle_configuration(lifecycle)
    .send()
    .await?;

The documents/ prefix filter ensures lifecycle rules apply only to content uploads, excluding system metadata or configuration objects that might share the bucket.

Runtime Document Operations

Services interact with S3 exclusively through the S3Client wrapper, which encapsulates bucket naming and error handling.

Client Initialization

let s3_client = S3Client::new(
    aws_sdk_s3::Client::new(&aws_config),
    config.document_storage_bucket.clone()
);

The S3Client::new constructor accepts the preconfigured AWS SDK client and the bucket name from service configuration.

Upload with Streaming

let key = format!("documents/{}/content.md", document_id);
S3Client::put_object(&s3_client, &key, ByteStream::from(file_bytes))
    .await?;

The put_object method (implementation in services/static_file_service/src/service/s3/client.rs) automatically prepends the correct bucket from the stored storage_bucket field.

Presigned URL Generation

For frontend downloads, the client generates time-limited presigned URLs:

let url = s3_client
    .put_document_storage_presigned_url(&key, PresignConfig::default())
    .await?;

This delegates to aws_sdk_s3::presigning::PresigningConfig with the stored bucket context.

Key Source Files

Path	Responsibility
services/static_file_service/src/service/s3/client.rs	S3Client wrapper with bucket-scoped operations
tooling/xtask/crates/xtask_local/src/local/localstack.rs	LocalStack orchestration: bucket creation, CORS, notifications
services/document_storage_service/src/main.rs	Service bootstrap reading document_storage_bucket config
packages/sdk/specs/storage.json	SDK contract documenting lifecycle metadata in API responses

Summary

• Configuration centralization: The document_storage_bucket parameter feeds all services from a single configuration source.
• Provisioning-time lifecycle: Glacier transition (30 days) and expiration (90 days) are enforced through put_bucket_lifecycle_configuration, not application logic.
• LocalStack parity: Development environments recreate production bucket behavior including CORS and event notifications.
• Client abstraction: The S3Client struct eliminates bucket name bookkeeping from business code.
• Cost optimization: Automated lifecycle rules reduce storage spend without service-level intervention.

Frequently Asked Questions

How does Macro handle S3 bucket naming across environments?

Each environment sets document_storage_bucket in its configuration, and S3Client::new receives this value at service startup. Local development uses LocalStack with hardcoded bucket names in resources::BUCKETS, while production uses externally provisioned bucket names injected through environment configuration.

Where is the lifecycle policy actually applied if not in the visible source code?

The lifecycle configuration likely executes through infrastructure-as-code (Terraform, CloudFormation, or CDK) outside the application repository, consistent with Macro's pattern of provisioning AWS resources separately from runtime services. The policy matches the SDK-documented behavior: 30-day Glacier transition, 90-day expiration.

Why does the local setup skip lifecycle configuration?

LocalStack's community edition has limited lifecycle support, and Macro's local development prioritizes functional parity for upload/download flows over cost-optimization features. The lifecycle rules are integration-tested in staging environments with real S3.

Can lifecycle rules target specific document types?

Yes. The LifecycleFilter::Prefix("documents/") pattern shown in the configuration allows prefix-based filtering. Macro could extend this with multiple rules for different documents/{type}/ prefixes if retention requirements vary by content category.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}