How Buzz Stores Media Files: S3-Compatible Object Storage with Content Hashing

Buzz stores media files in an S3-compatible object store using SHA-256 content-addressed keys while maintaining per-community metadata in PostgreSQL, enabling deduplication across communities with granular access control.

The block/buzz repository implements a community-scoped media storage system that separates raw binary storage from relational metadata. This architecture allows the Nostr relay to enforce community-specific quotas and visibility rules while leveraging scalable object storage backends like MinIO or AWS S3.

Upload Flow and Blossom Authentication

Media uploads in Buzz require cryptographically signed authorization scoped to individual communities. When a user initiates an upload, the client must provide a valid NIP-98 Blossom authentication event that identifies the target community.

In crates/buzz-relay/src/api/media.rs, the upload_media handler validates this auth event before processing bytes. The desktop application exposes this functionality through desktop/src-tauri/src/commands/media.rs, which forwards the file stream to the relay's API endpoint.

// crates/buzz-relay/src/api/media.rs
pub async fn upload_media(
    auth_event: SignedEvent,
    bytes: Vec<u8>,
    mime: Mime,
) -> Result<String, Error> {
    // Validate NIP-98 Blossom auth against community scope
    validate_blossom_auth(&auth_event)?;
    
    // Generate SHA-256 hash for content addressing
    let hash = sha256(&bytes);
    let ext = extension_from_mime(&mime);
    let key = format!("{}/{}.{}", &hash[..2], &hash[2..], ext);
    
    // Upload to S3-compatible store
    s3_client.put_object(&key, bytes).await?;
    
    // Store metadata in PostgreSQL
    db::media::insert(hash, mime, bytes.len(), community_id).await?;
    
    Ok(format!("/media/{}.{}", hash, ext))
}

Object Storage Configuration

Buzz uses standard S3 protocol semantics for durability and broad compatibility. By default, the development stack runs a local MinIO container, but production deployments target AWS S3, Cloudflare R2, Wasabi, or any S3-compatible provider.

Configuration is controlled via environment variables defined in deploy/compose/.env.example:

# S3-compatible storage configuration

BUZZ_S3_ENDPOINT=https://s3.example.com
BUZZ_S3_BUCKET=buzz-media
BUZZ_S3_ACCESS_KEY_ID=AKIA...
BUZZ_S3_SECRET_ACCESS_KEY=...
BUZZ_S3_ADDRESSING_STYLE=path  # or "virtual"

The BUZZ_S3_ADDRESSING_STYLE parameter determines whether the client uses path-style (https://s3.example.com/bucket-name/key) or virtual-hosted-style (https://bucket-name.s3.example.com/key) requests. The relay implementation uses the AWS SDK for Rust to handle these addressing modes transparently.

Content-Addressed Storage and Deduplication

Buzz implements content-addressed storage where the SHA-256 hash of the file content determines the S3 object key. This guarantees that identical files reference the same underlying storage object regardless of filename or upload context.

The storage key format follows {first-two-hash-chars}/{remaining-hash}.{ext}. For example, a PNG file with hash a3f1c2e4b9... stores at a3/f1c2e4b9...png. This prefixing prevents S3 bucket listing hot spots when storing millions of objects.

Because storage is content-addressed, uploading the same image to multiple communities creates only one S3 object while maintaining separate metadata rows in PostgreSQL for each community. This deduplication reduces storage costs while preserving per-community audit trails and quota enforcement.

Metadata Management in PostgreSQL

While raw bytes live in object storage, relational metadata resides in the media table managed by the buzz-db crate. The schema tracks community scoping, file characteristics, and audit information:

-- crates/buzz-db/src/schema/media.sql
CREATE TABLE media (
    sha256_hash BYTEA PRIMARY KEY,
    community_id UUID NOT NULL REFERENCES communities(id),
    mime_type TEXT NOT NULL,
    file_size BIGINT NOT NULL,
    original_filename TEXT,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE INDEX idx_media_community ON media(community_id);

This separation allows Buzz to enforce community-specific policies—such as visibility restrictions or storage quotas—without duplicating binary storage or managing complex S3 ACLs.

Retrieval and Client Integration

Clients fetch media via GET /media/{sha256_hash}.{ext}, with requests authenticated using the same NIP-98 Blossom scheme. The TypeScript helper in desktop/src/shared/lib/mediaUrl.ts constructs these endpoints:

// desktop/src/shared/lib/mediaUrl.ts
export function mediaUrl(hash: string, ext: string): string {
  return `${process.env.BUZZ_RELAY_URL}/media/${hash}.${ext}`;
}

// Usage in UI components
const img = document.createElement("img");
img.src = mediaUrl("9f2c...c4", "png");

The mobile client implementation in mobile/lib/shared/relay/media_upload.dart demonstrates the multipart upload flow:

// mobile/lib/shared/relay/media_upload.dart
Future<String> uploadMedia(Uint8List bytes, String mimeType) async {
  final authEvent = await createBlossomAuthEvent(
    method: 'POST',
    url: '$relayUrl/media',
  );
  
  final request = http.MultipartRequest('POST', Uri.parse('$relayUrl/media'))
    ..files.add(http.MultipartFile.fromBytes(
      'file', bytes, 
      contentType: MediaType.parse(mimeType)
    ))
    ..headers['Authorization'] = 'Nostr ${authEvent.serialize()}';
  
  final response = await request.send();
  return response.headers['location']!; // Returns /media/{hash}.{ext}
}

CLI and Programmatic Usage

The buzz CLI provides a convenient interface for shell-based uploads:

export BUZZ_RELAY_URL=https://relay.example.com
export BUZZ_PRIVATE_KEY=nsec...

buzz upload ./presentation.pdf

# Output: https://relay.example.com/media/7d8e9f...a2b.pdf

For Rust applications, direct SDK usage follows this pattern:

use buzz_relay::api::media::upload_media;
use buzz_core::event::SignedEvent;

let auth_event: SignedEvent = create_blossom_auth("POST", "/media").await?;
let bytes = std::fs::read("photo.jpg")?;
let mime = mime_guess::from_path("photo.jpg").first_or_octet_stream();

let media_url = upload_media(&auth_event, bytes, mime).await?;
println!("Stored at: {}", media_url);

Summary

• Buzz uses S3-compatible object storage (MinIO in development, AWS S3/R2 in production) configured via BUZZ_S3_BUCKET and related environment variables in deploy/compose/.env.example.
• Files are stored by SHA-256 hash in crates/buzz-relay/src/api/media.rs to enable automatic deduplication across communities while keeping metadata separate.
• Uploads and downloads require NIP-98 Blossom authentication scoped to specific communities, enforced in the relay's media handler.
• PostgreSQL tracks per-community metadata in the media table defined in crates/buzz-db/src/schema/media.sql, allowing granular access control without S3 ACL complexity.
• Content-addressed keys use hash-prefixing ({hash[0..2]}/{hash[2..]}.{ext}) to optimize S3 performance at scale.

Frequently Asked Questions

How does Buzz handle duplicate media uploads across different communities?

Buzz stores only one copy of the binary data in S3 because the object key is derived from the file's SHA-256 hash. However, each community that uploads the file gets a distinct row in the PostgreSQL media table with its own community_id. This allows Community A and Community B to share storage costs while maintaining independent visibility rules and audit logs.

What S3 storage backends are compatible with Buzz?

Any S3-compatible object store works, including MinIO (default for local development), AWS S3, Cloudflare R2, Wasabi, and DigitalOcean Spaces. Configuration is handled through standard environment variables like BUZZ_S3_ENDPOINT, BUZZ_S3_ACCESS_KEY_ID, and BUZZ_S3_ADDRESSING_STYLE defined in deploy/compose/.env.example.

How does Buzz secure media access between communities?

Access control relies on NIP-98 Blossom authentication events rather than S3 ACLs. When requesting GET /media/{hash}.{ext}, the client must provide a signed auth event proving membership in the community that owns the metadata row. The relay validates this in crates/buzz-relay/src/api/media.rs before serving or proxying the S3 object, ensuring cross-community isolation even though the underlying storage is shared.

Where is the media metadata stored and what does it track?

Metadata lives in PostgreSQL in the media table defined by crates/buzz-db/src/schema/media.sql. It stores the SHA-256 hash (linking to the S3 object), MIME type, file size, original filename, community ID for scoping, and creation timestamps. This relational data enables quota enforcement, search, and community-specific policies without modifying the immutable object storage layer.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}