How Macro's Document Storage Service Handles S3 Uploads and Retrieval

Macro's document storage service abstracts all S3 interactions behind a thin, testable S3Client wrapper that supports direct uploads, presigned URL generation, and concurrent batch operations across three distinct buckets.

The macro-inc/macro repository implements a dedicated document-storage-service that centralizes every S3 operation behind a single Rust crate. According to the source code in services/document_storage_service/src/service/s3/mod.rs, the service uses a custom S3Client wrapper around the AWS SDK to manage three separate buckets—document_storage_bucket, docx_upload_bucket, and upload_staging_bucket—enabling secure client-side uploads and efficient document retrieval for the entire Macro ecosystem.

S3Client Architecture and Bucket Configuration

The service injects AWS configuration through a thin abstraction that separates production S3 calls from test mocks.

In services/document_storage_service/src/main.rs, the application constructs the S3Client with bucket names loaded from environment configuration. The wrapper accepts three distinct bucket strings to isolate different document lifecycle stages:

• document_storage_bucket – Stores every persisted document permanently.
• docx_upload_bucket – Holds raw .docx files awaiting processing.
• upload_staging_bucket – Temporary storage for files staged before validation.

use aws_sdk_s3::Client as AwsS3Client;
use document_storage_service::service::s3::S3Client;

let s3 = AwsS3Client::new(&aws_config).await;
let s3_client = S3Client::new(
    s3,
    &config.document_storage_bucket,
    &config.docx_upload_bucket,
    &config.upload_staging_bucket,
);

This design allows other services—such as the email_service, search_processing_service, and sync-service—to interact with S3 indirectly through the DocumentStorageServiceClient defined in crates/document_storage_service_client/src/lib.rs, rather than managing AWS credentials themselves.

Direct Upload and Download Operations

For server-side document processing, the S3Client provides low-latency streaming methods that bypass presigned URLs.

Uploading documents uses upload_document in services/document_storage_service/src/service/s3/upload_document.rs, which streams a Vec<u8> buffer to S3 using ByteStream:

let key = "user/123/doc/456/file.pdf";
let content = std::fs::read("local/file.pdf")?;
s3_client.upload_document(key, content).await?;

Retrieving documents calls get_document (implemented in services/document_storage_service/src/service/s3/get.rs), returning raw bytes for text extraction or search indexing:

let key = "user/123/doc/456/file.pdf";
let bytes = s3_client.get_document(key).await?;

Presigned URL Generation for Secure Client Uploads

Rather than proxying large files through the service, Macro generates time-limited presigned URLs that allow clients to upload directly to S3 while maintaining security metadata.

The central logic resides in services/document_storage_service/src/service/s3/internal_presigned_helpers.rs, with specializations in put_presigned_url.rs. The service supports three distinct presigned upload endpoints:

• put_document_storage_presigned_url – Primary documents requiring SHA validation and content-type metadata.
• put_docx_upload_presigned_url – Word documents routed to the dedicated DOCX processing bucket.
• put_upload_zip_staging_presigned_url – ZIP archives staged for temporary processing.

let key = format!("user/{}/doc/{}", user_id, document_id);
let sha = sha256(&file_bytes);
let content_type = ContentType::Pdf; // model::document::ContentType
let upload_url = s3_client
    .put_document_storage_presigned_url(&key, &sha, content_type)
    .await?;

For retrieval, get_snapshot_presigned_url generates read-only URLs that expire after a configured duration, allowing frontend clients to download documents without exposing bucket permissions.

Advanced Lifecycle Operations

The S3Client exposes additional methods in services/document_storage_service/src/service/s3/mod.rs for document management at scale:

Method	Purpose	Implementation File
copy_document	Duplicates objects within the primary bucket	copy_document.rs
delete_document	Removes all objects matching a user-document pair	delete.rs
exists	Probes whether a specific S3 key exists	exists.rs
get_folder_content_names	Lists file names and types under a prefix	get_folder.rs
shas_exist	Concurrently validates multiple SHA checksums	mod.rs (uses semaphore-bounded concurrency)

The shas_exist method deserves particular attention for batch import workflows. It accepts a Vec<String> of SHA values and checks existence concurrently, using a semaphore to bound parallelism and prevent AWS rate limiting:

let shas = vec!["abc123".to_string(), "def456".to_string()];
let all_present = s3_client.shas_exist(&shas).await?;

Testing Abstraction with Mockall

The crate uses conditional compilation to enable unit testing without AWS dependencies. In services/document_storage_service/src/service/s3/mod.rs, the code exposes MockS3Client as S3 when the #[cfg(test)] flag is active:

#[cfg(test)]
pub use MockS3Client as S3;

This allows downstream services to test their integration with the document storage service without network calls, verifying that DocumentStorageServiceClient correctly forwards parameters to the underlying S3 operations.

Summary

• Macro's document storage service centralizes S3 operations through a single S3Client wrapper located in services/document_storage_service/src/service/s3/mod.rs.
• The architecture segregates storage into three buckets: primary document storage, DOCX processing, and temporary staging.
• Client uploads flow through presigned URLs generated by put_document_storage_presigned_url, which embed SHA checksums and content-type metadata for integrity validation.
• Server-side operations like upload_document and get_document provide direct byte streaming for internal services that require raw document processing.
• Batch operations such as shas_exist use semaphore-bounded concurrency to check multiple document hashes efficiently.
• The mockall integration allows the entire ecosystem to test S3 interactions without AWS credentials or network calls.

Frequently Asked Questions

How does Macro handle large file uploads without overloading the API servers?

Macro offloads large file transfers to S3 directly using presigned PUT URLs generated by put_document_storage_presigned_url. The API server only generates and signs the URL metadata—including the SHA256 hash and MIME type—allowing clients to stream multi-gigabyte files straight to the document_storage_bucket while the service instance remains unburdened by proxy traffic.

What is the purpose of the three separate S3 buckets in Macro's architecture?

The three buckets isolate documents by processing stage and security requirements. document_storage_bucket stores finalized documents permanently, docx_upload_bucket receives raw Microsoft Word files for conversion pipelines, and upload_staging_bucket holds temporary ZIP archives during import validation. This separation prevents naming collisions and allows distinct lifecycle policies—for example, automatically expiring staging files after 24 hours.

How does the service verify document integrity during upload?

When generating presigned URLs via put_document_storage_presigned_url, the service requires a SHA256 checksum as a parameter. This hash becomes part of the signed request metadata, forcing S3 to validate the uploaded bytes against the expected digest. The shas_exist method later allows batch verification that identical documents already exist in storage, enabling deduplication before expensive processing begins.

Can other Macro services test S3 interactions without AWS credentials?

Yes. The S3Client implementation uses the mockall crate to generate MockS3Client under #[cfg(test)] conditions. Other services import DocumentStorageServiceClient from crates/document_storage_service_client/src/lib.rs and can inject the mock during unit tests, verifying integration logic without network calls or valid AWS tokens.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}