How CCR SQLite Store Enables Byte-Exact Payload Recovery in Caveman

The CCR SQLite store guarantees byte-exact payload recovery by storing raw BLOBs in an immutable column, generating deterministic handles from payload hashes, and retrieving data through exact-byte SELECT queries without transformation.

The Caveman engine's CCR (Content-Compressed Retrieval) system relies on a SQLite-backed persistence layer to ensure lossless data recovery. Unlike storage formats that transform or compress the source material, the CCR SQLite store implements a write-once, read-exactly contract. This article breaks down the three architectural decisions in engine/ccr/store_sqlite.go that enable byte-perfect fidelity for any tool payload.

Immutable BLOB Column Storage

The foundation of byte-exact recovery rests on the original column in the recoveries table, defined as a raw BLOB type. When the Put method processes a recovery record, it inserts the payload bytes without compression, encoding, or modification.

In engine/ccr/store_sqlite.go (lines 66-80), the SQL insertion stores rec.Original directly:

// Simplified representation of the Insert logic
INSERT INTO recoveries (handle, original, content_type, compressor, tokens_before, tokens_after, created_at)
VALUES (?, ?, ?, ?, ?, ?, ?)

Because this column stores the exact byte sequence written by the tool, no data loss occurs during persistence. The store configures SQLite with WAL mode (journal_mode(WAL)) and a busy-timeout (busy_timeout(5000)) to ensure concurrent processes can safely read and write without returning partial data or corrupting the immutable BLOBs.

Deterministic Handle Generation

CCR generates handles cryptographically from the payload bytes themselves, creating an idempotent mapping between content and identifier. In the Put method (lines 55-57), the store computes:

handle := Handle(rec.Original)

This deterministic derivation ensures that identical byte sequences always produce the same handle string (prefixed with ccr_). The handle acts as a content-addressable key, allowing the system to detect duplicate payloads and return existing references without creating redundant database entries or altering the stored bytes.

Exact-Byte Retrieval Process

The Get method in engine/ccr/store_sqlite.go (lines 59-66) retrieves data through a targeted query:

SELECT original FROM recoveries WHERE handle = ?

Upon retrieval, the store explicitly copies the bytes into a new slice to prevent shared memory mutations:

append([]byte(nil), obj.Data...)

This defensive copy guarantees that the returned []byte contains exactly the same sequence of bytes that were originally stored, regardless of subsequent database operations or concurrent access patterns by multiple Caveman processes (engine, MCP server, or proxy).

Practical Implementation: Storing and Recovering Payloads

Storing a Payload with Byte-Exact Writes

rec := ccr.Recovery{
    Original:    []byte(`{"result":"hello world"}`),
    ContentType: "application/json",
    Compressor:  "none",
    TokensBefore: 15,
    TokensAfter:  15,
}

// Open the persistent store (defaults to ~/.caveman/ccr.db)
store, err := ccr.Open("")
if err != nil { log.Fatal(err) }
defer store.Close()

// Store returns a deterministic handle derived from the payload hash
handle, err := store.Put(rec)
if err != nil { log.Fatal(err) }

// Handle format: <<ccr:ccr_000001abcd>>
fmt.Println("CCR handle:", handle)

Recovering the Original Bytes

// Retrieve the exact payload from any Caveman process
orig, err := store.Get(handle)
if err != nil {
    if errors.Is(err, ccr.ErrNotFound) {
        log.Fatalf("handle %s not found", handle)
    }
    log.Fatal(err)
}

// orig now holds the exact byte slice that was stored earlier
fmt.Printf("Recovered payload: %s\n", string(orig))

Embedding CCR Markers in Transformed Output

The proxy gateway in proxy/internal/gateway/gateway.go (lines 1095-1100) injects handles into compressed requests using the appendCCRMarker helper:

compressedBody := appendCCRMarker(originalBody, handle)
// Results in payload containing: <<ccr:ccr_000001abcd>>

Summary

• Immutable BLOBs: The original column in engine/ccr/store_sqlite.go stores raw bytes without compression or encoding, ensuring the SQLite database preserves the exact sequence written by the tool.
• Content-Addressable Handles: The Handle() function generates deterministic identifiers from payload bytes, enabling idempotent storage and reliable lookup via Put and Get.
• Defensive Copying: The Get method returns a fresh byte slice via append([]byte(nil), obj.Data...) to prevent memory aliasing and guarantee byte-exact fidelity.
• Concurrent Safety: WAL mode and busy timeouts allow multiple Caveman processes to access the store simultaneously without data corruption or lost writes.

Frequently Asked Questions

Does the CCR SQLite store compress the original payload?

No. While the Recovery struct tracks compression metadata in the Compressor field, the original column always stores the raw, uncompressed bytes. Compression applies only to the transformed output sent to downstream services, not to the recovery data persisted in SQLite.

How does the store handle concurrent writes from multiple processes?

The store initializes SQLite with journal_mode(WAL) and busy_timeout(5000), enabling Write-Ahead Logging. This allows one writer and multiple readers to operate simultaneously without locking conflicts, ensuring that every successful Put persists before any Get can observe the data.

What happens if I attempt to store identical payloads twice?

The deterministic handle generation detects duplicates through content-addressing. When Put receives a payload whose hash matches an existing record, it updates the metadata row but leaves the immutable original BLOB unchanged, returning the existing handle without creating redundant storage.

Where does the actual byte copying occur during retrieval?

In engine/ccr/store_sqlite.go, the Get method executes append([]byte(nil), obj.Data...) after the SELECT original query (lines 59-66). This explicitly allocates a new slice and copies the database bytes, preventing the caller from holding references to internal SQLite memory buffers that might be reused or mutated.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}