# How Macro's Document Storage Service Handles S3 Uploads and Retrieval

> Discover how Macro's document storage service simplifies S3 uploads and retrieval with a testable S3Client wrapper enabling direct uploads, presigned URLs, and batch operations.

- Repository: [Macro/macro](https://github.com/macro-inc/macro)
- Tags: how-to-guide
- Published: 2026-08-18

---

**Macro's document storage service abstracts all S3 interactions behind a thin, testable `S3Client` wrapper that supports direct uploads, presigned URL generation, and concurrent batch operations across three distinct buckets.**

The `macro-inc/macro` repository implements a dedicated **document-storage-service** that centralizes every S3 operation behind a single Rust crate. According to the source code in [`services/document_storage_service/src/service/s3/mod.rs`](https://github.com/macro-inc/macro/blob/main/services/document_storage_service/src/service/s3/mod.rs), the service uses a custom `S3Client` wrapper around the AWS SDK to manage three separate buckets—**`document_storage_bucket`**, **`docx_upload_bucket`**, and **`upload_staging_bucket`**—enabling secure client-side uploads and efficient document retrieval for the entire Macro ecosystem.

## S3Client Architecture and Bucket Configuration

The service injects AWS configuration through a thin abstraction that separates production S3 calls from test mocks.

In [`services/document_storage_service/src/main.rs`](https://github.com/macro-inc/macro/blob/main/services/document_storage_service/src/main.rs), the application constructs the `S3Client` with bucket names loaded from environment configuration. The wrapper accepts three distinct bucket strings to isolate different document lifecycle stages:

- **`document_storage_bucket`** – Stores every persisted document permanently.
- **`docx_upload_bucket`** – Holds raw *.docx* files awaiting processing.
- **`upload_staging_bucket`** – Temporary storage for files staged before validation.

```rust
use aws_sdk_s3::Client as AwsS3Client;
use document_storage_service::service::s3::S3Client;

let s3 = AwsS3Client::new(&aws_config).await;
let s3_client = S3Client::new(
    s3,
    &config.document_storage_bucket,
    &config.docx_upload_bucket,
    &config.upload_staging_bucket,
);

```

This design allows other services—such as the *email_service*, *search_processing_service*, and *sync-service*—to interact with S3 indirectly through the `DocumentStorageServiceClient` defined in [`crates/document_storage_service_client/src/lib.rs`](https://github.com/macro-inc/macro/blob/main/crates/document_storage_service_client/src/lib.rs), rather than managing AWS credentials themselves.

## Direct Upload and Download Operations

For server-side document processing, the `S3Client` provides low-latency streaming methods that bypass presigned URLs.

**Uploading documents** uses `upload_document` in [`services/document_storage_service/src/service/s3/upload_document.rs`](https://github.com/macro-inc/macro/blob/main/services/document_storage_service/src/service/s3/upload_document.rs), which streams a `Vec<u8>` buffer to S3 using `ByteStream`:

```rust
let key = "user/123/doc/456/file.pdf";
let content = std::fs::read("local/file.pdf")?;
s3_client.upload_document(key, content).await?;

```

**Retrieving documents** calls `get_document` (implemented in [`services/document_storage_service/src/service/s3/get.rs`](https://github.com/macro-inc/macro/blob/main/services/document_storage_service/src/service/s3/get.rs)), returning raw bytes for text extraction or search indexing:

```rust
let key = "user/123/doc/456/file.pdf";
let bytes = s3_client.get_document(key).await?;

```

## Presigned URL Generation for Secure Client Uploads

Rather than proxying large files through the service, Macro generates time-limited presigned URLs that allow clients to upload directly to S3 while maintaining security metadata.

The central logic resides in [`services/document_storage_service/src/service/s3/internal_presigned_helpers.rs`](https://github.com/macro-inc/macro/blob/main/services/document_storage_service/src/service/s3/internal_presigned_helpers.rs), with specializations in [`put_presigned_url.rs`](https://github.com/macro-inc/macro/blob/main/put_presigned_url.rs). The service supports three distinct presigned upload endpoints:

- **`put_document_storage_presigned_url`** – Primary documents requiring SHA validation and content-type metadata.
- **`put_docx_upload_presigned_url`** – Word documents routed to the dedicated DOCX processing bucket.
- **`put_upload_zip_staging_presigned_url`** – ZIP archives staged for temporary processing.

```rust
let key = format!("user/{}/doc/{}", user_id, document_id);
let sha = sha256(&file_bytes);
let content_type = ContentType::Pdf; // model::document::ContentType
let upload_url = s3_client
    .put_document_storage_presigned_url(&key, &sha, content_type)
    .await?;

```

For retrieval, `get_snapshot_presigned_url` generates read-only URLs that expire after a configured duration, allowing frontend clients to download documents without exposing bucket permissions.

## Advanced Lifecycle Operations

The `S3Client` exposes additional methods in [`services/document_storage_service/src/service/s3/mod.rs`](https://github.com/macro-inc/macro/blob/main/services/document_storage_service/src/service/s3/mod.rs) for document management at scale:

| Method | Purpose | Implementation File |
|--------|---------|---------------------|
| **`copy_document`** | Duplicates objects within the primary bucket | [`copy_document.rs`](https://github.com/macro-inc/macro/blob/main/copy_document.rs) |
| **`delete_document`** | Removes all objects matching a user-document pair | [`delete.rs`](https://github.com/macro-inc/macro/blob/main/delete.rs) |
| **`exists`** | Probes whether a specific S3 key exists | [`exists.rs`](https://github.com/macro-inc/macro/blob/main/exists.rs) |
| **`get_folder_content_names`** | Lists file names and types under a prefix | [`get_folder.rs`](https://github.com/macro-inc/macro/blob/main/get_folder.rs) |
| **`shas_exist`** | Concurrently validates multiple SHA checksums | [`mod.rs`](https://github.com/macro-inc/macro/blob/main/mod.rs) (uses semaphore-bounded concurrency) |

The **`shas_exist`** method deserves particular attention for batch import workflows. It accepts a `Vec<String>` of SHA values and checks existence concurrently, using a semaphore to bound parallelism and prevent AWS rate limiting:

```rust
let shas = vec!["abc123".to_string(), "def456".to_string()];
let all_present = s3_client.shas_exist(&shas).await?;

```

## Testing Abstraction with Mockall

The crate uses conditional compilation to enable unit testing without AWS dependencies. In [`services/document_storage_service/src/service/s3/mod.rs`](https://github.com/macro-inc/macro/blob/main/services/document_storage_service/src/service/s3/mod.rs), the code exposes `MockS3Client` as `S3` when the `#[cfg(test)]` flag is active:

```rust
#[cfg(test)]
pub use MockS3Client as S3;

```

This allows downstream services to test their integration with the document storage service without network calls, verifying that `DocumentStorageServiceClient` correctly forwards parameters to the underlying S3 operations.

## Summary

- Macro's document storage service centralizes S3 operations through a single `S3Client` wrapper located in [`services/document_storage_service/src/service/s3/mod.rs`](https://github.com/macro-inc/macro/blob/main/services/document_storage_service/src/service/s3/mod.rs).
- The architecture segregates storage into three buckets: primary document storage, DOCX processing, and temporary staging.
- Client uploads flow through presigned URLs generated by `put_document_storage_presigned_url`, which embed SHA checksums and content-type metadata for integrity validation.
- Server-side operations like `upload_document` and `get_document` provide direct byte streaming for internal services that require raw document processing.
- Batch operations such as `shas_exist` use semaphore-bounded concurrency to check multiple document hashes efficiently.
- The `mockall` integration allows the entire ecosystem to test S3 interactions without AWS credentials or network calls.

## Frequently Asked Questions

### How does Macro handle large file uploads without overloading the API servers?

Macro offloads large file transfers to S3 directly using **presigned PUT URLs** generated by `put_document_storage_presigned_url`. The API server only generates and signs the URL metadata—including the SHA256 hash and MIME type—allowing clients to stream multi-gigabyte files straight to the `document_storage_bucket` while the service instance remains unburdened by proxy traffic.

### What is the purpose of the three separate S3 buckets in Macro's architecture?

The three buckets isolate documents by processing stage and security requirements. **`document_storage_bucket`** stores finalized documents permanently, **`docx_upload_bucket`** receives raw Microsoft Word files for conversion pipelines, and **`upload_staging_bucket`** holds temporary ZIP archives during import validation. This separation prevents naming collisions and allows distinct lifecycle policies—for example, automatically expiring staging files after 24 hours.

### How does the service verify document integrity during upload?

When generating presigned URLs via `put_document_storage_presigned_url`, the service requires a SHA256 checksum as a parameter. This hash becomes part of the signed request metadata, forcing S3 to validate the uploaded bytes against the expected digest. The `shas_exist` method later allows batch verification that identical documents already exist in storage, enabling deduplication before expensive processing begins.

### Can other Macro services test S3 interactions without AWS credentials?

Yes. The `S3Client` implementation uses the `mockall` crate to generate `MockS3Client` under `#[cfg(test)]` conditions. Other services import `DocumentStorageServiceClient` from [`crates/document_storage_service_client/src/lib.rs`](https://github.com/macro-inc/macro/blob/main/crates/document_storage_service_client/src/lib.rs) and can inject the mock during unit tests, verifying integration logic without network calls or valid AWS tokens.