Understanding the croc Stored Transfer Protocol and Client-Side Encryption

The croc stored transfer protocol enables secure file uploads to remote storage services (S3, GCS, etc.) using client-side AES-GCM encryption, ensuring the storage provider never sees file names, sizes, hashes, or encryption keys.

The croc file transfer tool from schollz/croc supports a stored-transfer mode that uploads encrypted files to cloud storage rather than using direct peer-to-peer connections. This architecture moves all cryptographic operations—including key generation, derivation, and authenticated encryption—to the client, maintaining zero-knowledge privacy guarantees even when using third-party storage backends.

Protocol Identity and Architecture

The stored-transfer protocol is formally identified by the constant croc-store-v1 defined in src/storecrypto/storecrypto.go:

const Protocol = "croc-store-v1"

When operating in stored mode, croc splits files into 4 MiB chunks (ChunkSize), encrypts them with AES-256-GCM, and uploads the ciphertext to a configurable storage backend. The storage service only handles opaque encrypted blobs and never receives the master encryption key or the transfer identifier required for decryption.

Key Generation and Derivation

All cryptographic material originates from two client-generated values:

• Master Key – A fresh 256-bit random key created via GenerateKey() for each transfer.
• Transfer ID – A public, URL-safe 128-bit identifier generated by GenerateTransferID().

Neither value leaves the client unencrypted. From the master key, croc derives specialized keys using HKDF-SHA256 via the internal derive function:

key, err := hkdf.Key(sha256.New, master, nil,
    Protocol+"/"+label, KeySize)

In src/storecrypto/storecrypto.go, two labels determine the key purpose:

• "manifest" – Encrypts the JSON manifest describing the transfer.
• "data" – Encrypts individual file chunks.

Authenticated Encryption with Associated Data

croc implements AES-GCM through the aeadFor helper, using randomly generated nonces for each encryption operation. Critical to the security model is the Associated Data (AAD) that binds ciphertext to the specific transfer and data type, preventing substitution attacks.

The AAD format varies by content type as defined in storecrypto.go:

// Manifest AAD
[]byte(Protocol + "\x00" + id + "\x00manifest")

// Chunk AAD  
[]byte(Protocol + "\x00" + id + "\x00chunk\x00" + …)

This construction ensures that chunks cannot be swapped between transfers or confused with manifest data.

Manifest Encryption and Validation

Before upload, croc constructs a Manifest structure describing the entire transfer—version, chunk size, and a slice of ManifestFile entries containing file names, sizes, and SHA-256 hashes. The sender validates the manifest with ValidateManifest(), then encrypts it using SealManifest() (which chains to SealManifestJSON and the internal seal function).

Receivers download the encrypted manifest and decrypt it via OpenManifest() → OpenManifestJSON → open, which verifies authenticity before returning the plaintext JSON description of the file collection.

Chunk Encryption and File Splitting

Files are divided into 4 MiB chunks. For each chunk, croc creates a ChunkRef recording its object index, file index, chunk number, and plaintext size. The SealChunk function encrypts chunk data using the "data" label and chunk-specific AAD, while OpenChunk performs decryption and integrity verification, confirming the output length matches PlainSize.

This chunking strategy allows resumable downloads and streaming decryption without requiring the receiver to download entire files before verification.

Share Generation and Transfer Tokens

To initiate a download, the sender creates a Share structure bundling three fields:

• Origin – Base URL of the storage service (e.g., https://s3.amazonaws.com)
• ID – Public transfer identifier
• MasterKey – 32-byte secret for key derivation

croc encodes shares in two formats via methods in storecrypto.go:

• Browser URL: https://<origin>/s/<id>#v1.<key> (BrowserURL())
• CLI Token: croc-store-v1.<origin-base64>.<id>.<key> (CLIToken())

Receivers parse either format using ParseShare(), which validates and normalizes the components before returning a Share struct for decryption operations.

End-to-End Implementation Example

The following demonstrates the complete sender and receiver flow using the storecrypto and storeclient packages:

// Sender implementation
import (
    "github.com/schollz/croc/src/storecrypto"
    "github.com/schollz/croc/src/storeclient"
)

// Generate fresh cryptographic material
masterKey, _ := storecrypto.GenerateKey()
transferID, _ := storecrypto.GenerateTransferID()

// Build the manifest describing files
manifest := storecrypto.Manifest{
    Version:   storecrypto.Version,
    ChunkSize: storecrypto.ChunkSize,
    Files: []storecrypto.ManifestFile{
        {
            Name:       "document.pdf",
            Size:       1048576,
            Modified:   time.Now(),
            SHA256:     storecrypto.EncodedSHA256(fileBytes),
            FirstChunk: 0,
            ChunkCount: 1,
        },
    },
}

// Encrypt and upload
encManifest, _ := storecrypto.SealManifest(masterKey, transferID, manifest)
ref := storecrypto.ChunkRef{ObjectIndex: 0, FileIndex: 0, FileChunk: 0, PlainSize: len(fileBytes)}
encChunk, _ := storecrypto.SealChunk(masterKey, transferID, ref, fileBytes)

client := storeclient.New("https://my-storage.example")
client.PutObject(transferID, encManifest)
client.PutObject(transferID, encChunk)

// Generate share for receiver
share := storecrypto.Share{Origin: "https://my-storage.example", ID: transferID, MasterKey: masterKey}
url, _ := share.BrowserURL()   // https://my-storage.example/s/<id>#v1.<key>
token, _ := share.CLIToken()     // croc-store-v1.<b64origin>.<id>.<key>

// Receiver implementation
import "github.com/schollz/croc/src/storecrypto"

// Parse CLI token or browser URL
share, err := storecrypto.ParseShare("croc-store-v1.aHR0cHM6Ly9teS1zdG9yYWdlLmV4YW1wbGU=.transferID.key")
if err != nil {
    log.Fatal(err)
}

// Download and decrypt manifest
encManifest := downloadObject(share.ID) // platform-specific download
manifest, err := storecrypto.OpenManifest(share.MasterKey, share.ID, encManifest, 1<<40)

// Stream and decrypt chunks
for _, ref := range storecrypto.ChunkRefs(manifest) {
    encChunk := downloadChunk(ref.ObjectIndex)
    plain, err := storecrypto.OpenChunk(share.MasterKey, share.ID, ref, encChunk)
    // Write plain to manifest.Files[ref.FileIndex].Name
}

Summary

• The croc stored transfer protocol (croc-store-v1) enables secure file hosting on untrusted storage by performing all encryption client-side in src/storecrypto/storecrypto.go.
• AES-256-GCM protects both manifests and file chunks, with HKDF-SHA256 deriving separate keys for each data type from a client-generated master key.
• Associated Data (AAD) binds ciphertext to specific transfers, preventing cross-transfer substitution attacks.
• Share tokens encode the storage origin, public transfer ID, and master key in URL-safe formats parseable by ParseShare().
• The storage backend only handles opaque encrypted blobs and cannot determine file names, sizes, content hashes, or encryption keys.

Frequently Asked Questions

How is the encryption key generated and managed in croc stored transfers?

Each transfer generates a unique 256-bit master key via storecrypto.GenerateKey() and a 128-bit transfer ID via GenerateTransferID(). Both are created on the sender's client using cryptographically secure random number generation. The master key never leaves the client unencrypted—it is shared with receivers only through the URL fragment (browser mode) or CLI token, ensuring the storage service remains unaware of the decryption credentials.

What prevents the cloud storage provider from accessing my files?

All encryption occurs before data reaches the storage backend. Files are split into 4 MiB chunks and encrypted with AES-256-GCM using keys derived from the master secret via HKDF. The storage provider receives only ciphertext chunks and an encrypted manifest, lacking both the encryption keys and the transfer ID required for decryption. This zero-knowledge architecture ensures the provider cannot determine file contents, names, sizes, or hashes.

How does croc verify the integrity of transferred chunks?

croc uses authenticated encryption with associated data (AEAD) via AES-GCM. Each chunk includes a cryptographic authentication tag generated during SealChunk() that is verified during OpenChunk(). Additionally, the ChunkRef structure records the expected PlainSize, and the decryption process validates that the output matches this length. The manifest contains SHA-256 hashes of the original files, allowing receivers to verify integrity after reconstruction.

What is the difference between croc's live transfer and stored transfer modes?

Live transfers establish direct peer-to-peer connections using the webrtc or tcp relays defined in src/message/message.go, streaming data in real-time without intermediate storage. Stored transfers upload encrypted chunks to persistent cloud storage (S3, GCS, etc.) using src/storeclient/client.go and src/store/service.go, allowing asynchronous retrieval. While live transfers require both parties to be simultaneously online, stored transfers enable "send now, download later" workflows while maintaining the same client-side encryption guarantees.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}