How celld Supports Both S3 and Google Cloud Storage Buckets

celld abstracts storage behind a unified bucket specification using s3:// or gs:// prefixes, automatically selecting the appropriate backend while exposing a consistent API for reads and writes.

The celld project from Deno Land Inc. provides a distributed SQLite replication system that delegates persistent storage to external object stores. Rather than maintaining separate code paths for different providers, celld implements a single bucket abstraction in crates/celld/bucket.rs that transparently handles both S3-compatible services and Google Cloud Storage.

How Bucket Specifications Work

celld uses URL-style prefixes to determine which storage backend to initialize. This design keeps configuration explicit and user-facing interfaces minimal.

Syntax and Parsing

The entry point is Bucket::parse_spec in crates/celld/bucket.rs. It accepts strings in two forms:

• s3://BUCKET_NAME[/PREFIX] — S3-compatible endpoints (AWS S3, MinIO, Cloudflare R2, etc.)
• gs://BUCKET_NAME[/PREFIX] — Google Cloud Storage buckets

// From crates/celld/bucket.rs
let bucket = Bucket::open("s3://my-bucket/backups").await?;
// or
let bucket = Bucket::open("gs://project-bucket/data").await?;

The parser extracts the scheme, validates mutual exclusivity of options, and dispatches to the corresponding builder.

Backend Selection and Implementation

celld defines a StorageBackend enum with two variants: S3 and Gcs. Each wraps provider-specific authentication and transport logic while presenting identical operation semantics.

S3 Backend: StorageBackend::S3

When the s3:// prefix is detected, celld constructs an object_store::aws::AmazonS3Builder via ltx/src/client/object_store.rs. The implementation respects:

• Standard AWS credential chain (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN)
• Optional S3_ENDPOINT for custom endpoints (MinIO, R2, etc.)
• Conditional PUT semantics using ETag verification for safe concurrent writes

// S3 configuration flow
let backend = StorageBackend::S3 {
    endpoint: std::env::var("S3_ENDPOINT").ok(),
    // ... other AWS-specific options
};

GCS Backend: StorageBackend::Gcs

For gs:// prefixes, celld bypasses all S3-specific configuration. The GCS backend:

• Authenticates via Google Application Default Credentials (ADC)
• Accepts GOOGLE_APPLICATION_CREDENTIALS for service account keys
• Ignores S3_ENDPOINT, AWS_* environment variables, and path-style endpoint logic

This separation is documented in docs/limitations.md to prevent configuration errors.

Unified API Across Backends

Both backends implement the same Bucket trait methods, enabling celld's replication, fencing, and restore operations to remain storage-agnostic. The unified interface includes:

• get(key) — streaming reads
• put(key, data, options) — conditional writes with ETag / generation preconditions
• delete(key) — removal with verification

// Same code works regardless of s3:// or gs://
let data = bucket.get("snapshot/0001.ltx").await?;
bucket.put("snapshot/0002.ltx", &new_data, PutOptions {
    if_match: Some(current_etag),
}).await?;

CLI Validation and User Experience

The deploy subcommand in crates/celld/deploy.rs enforces valid flag combinations at parse time:

• --bucket gs://... rejects --endpoint (S3-specific)
• --bucket gs://... rejects static AWS credential flags
• Help text documents required environment variables for each backend

This prevents runtime authentication failures from misconfigured credential mixing.

Configuration Examples

Provider	Bucket Spec	Required Environment
AWS S3	s3://my-bucket/prefix	AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
MinIO	s3://my-bucket/prefix	S3_ENDPOINT=http://localhost:9000, AWS credentials
Cloudflare R2	s3://my-bucket/prefix	S3_ENDPOINT=https://<account>.r2.cloudflarestorage.com, R2 tokens
GCS	gs://my-bucket/prefix	GOOGLE_APPLICATION_CREDENTIALS or ADC

Key Source Files

• crates/celld/bucket.rs — Bucket::parse_spec, StorageBackend enum, backend dispatch
• crates/ltx/src/client/object_store.rs — S3 client construction via object_store crate
• crates/celld/deploy.rs — CLI argument parsing and validation
• docs/README.md — User-facing bucket configuration documentation
• docs/limitations.md — Authentication requirements and unsupported combinations

Summary

• Single specification format — s3:// or gs:// prefixes trigger automatic backend selection
• S3 backend — Uses object_store crate with full AWS credential chain and custom endpoint support via S3_ENDPOINT
• GCS backend — Relies on Google Application Default Credentials, explicitly ignoring S3 configuration
• Unified interface — Bucket trait methods abstract storage details from replication logic
• CLI safeguards — deploy.rs validates that GCS buckets cannot combine with S3-specific flags

Frequently Asked Questions

Can I use GCS with S3-compatible interoperability mode?

No. celld's GCS backend (gs:// prefix) uses native Google Cloud Storage authentication and APIs. It does not support GCS's S3 interoperability credentials. For S3-compatible access to GCS, use a third-party gateway or choose the S3 backend with a compatible endpoint.

What happens if I set both S3_ENDPOINT and use gs://?

The CLI parser in crates/celld/deploy.rs rejects this combination at startup. If bypassed, the GCS backend explicitly ignores S3_ENDPOINT per docs/limitations.md, so the setting has no effect.

Does celld support other object storage providers?

celld officially supports S3-compatible endpoints and Google Cloud Storage. Other providers (Azure Blob, etc.) are not implemented in the current codebase. Some may work via S3-compatible gateways if they implement the standard S3 API.

How does celld handle credentials rotation?

For S3, celld relies on the standard AWS SDK credential chain, which automatically handles IAM role assumption and credential refresh. For GCS, the object_store crate manages OAuth2 token refresh using ADC. No explicit credential reload code exists in celld itself.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}