# How CCR SQLite Store Enables Byte-Exact Payload Recovery in Caveman

> Learn how CCR SQLite store enables byte-exact payload recovery. It stores raw BLOBs immutably, generates deterministic handles, and retrieves data via exact-byte SELECT queries for guaranteed recovery.

- Repository: [Julius Brussee/caveman](https://github.com/JuliusBrussee/caveman)
- Tags: internals
- Published: 2026-09-04

---

**The CCR SQLite store guarantees byte-exact payload recovery by storing raw BLOBs in an immutable column, generating deterministic handles from payload hashes, and retrieving data through exact-byte SELECT queries without transformation.**

The [Caveman](https://github.com/JuliusBrussee/caveman) engine's CCR (Content-Compressed Retrieval) system relies on a SQLite-backed persistence layer to ensure lossless data recovery. Unlike storage formats that transform or compress the source material, the CCR SQLite store implements a **write-once, read-exactly** contract. This article breaks down the three architectural decisions in [`engine/ccr/store_sqlite.go`](https://github.com/JuliusBrussee/caveman/blob/main/engine/ccr/store_sqlite.go) that enable byte-perfect fidelity for any tool payload.

## Immutable BLOB Column Storage

The foundation of byte-exact recovery rests on the `original` column in the `recoveries` table, defined as a raw `BLOB` type. When the `Put` method processes a recovery record, it inserts the payload bytes without compression, encoding, or modification.

In [`engine/ccr/store_sqlite.go`](https://github.com/JuliusBrussee/caveman/blob/main/engine/ccr/store_sqlite.go) (lines 66-80), the SQL insertion stores `rec.Original` directly:

```go
// Simplified representation of the Insert logic
INSERT INTO recoveries (handle, original, content_type, compressor, tokens_before, tokens_after, created_at)
VALUES (?, ?, ?, ?, ?, ?, ?)

```

Because this column stores the exact byte sequence written by the tool, no data loss occurs during persistence. The store configures SQLite with **WAL mode** (`journal_mode(WAL)`) and a **busy-timeout** (`busy_timeout(5000)`) to ensure concurrent processes can safely read and write without returning partial data or corrupting the immutable BLOBs.

## Deterministic Handle Generation

CCR generates handles cryptographically from the payload bytes themselves, creating an idempotent mapping between content and identifier. In the `Put` method (lines 55-57), the store computes:

```go
handle := Handle(rec.Original)

```

This deterministic derivation ensures that identical byte sequences always produce the same handle string (prefixed with `ccr_`). The handle acts as a **content-addressable key**, allowing the system to detect duplicate payloads and return existing references without creating redundant database entries or altering the stored bytes.

## Exact-Byte Retrieval Process

The `Get` method in [`engine/ccr/store_sqlite.go`](https://github.com/JuliusBrussee/caveman/blob/main/engine/ccr/store_sqlite.go) (lines 59-66) retrieves data through a targeted query:

```sql
SELECT original FROM recoveries WHERE handle = ?

```

Upon retrieval, the store explicitly copies the bytes into a new slice to prevent shared memory mutations:

```go
append([]byte(nil), obj.Data...)

```

This defensive copy guarantees that the returned `[]byte` contains exactly the same sequence of bytes that were originally stored, regardless of subsequent database operations or concurrent access patterns by multiple Caveman processes (engine, MCP server, or proxy).

## Practical Implementation: Storing and Recovering Payloads

### Storing a Payload with Byte-Exact Writes

```go
rec := ccr.Recovery{
    Original:    []byte(`{"result":"hello world"}`),
    ContentType: "application/json",
    Compressor:  "none",
    TokensBefore: 15,
    TokensAfter:  15,
}

// Open the persistent store (defaults to ~/.caveman/ccr.db)
store, err := ccr.Open("")
if err != nil { log.Fatal(err) }
defer store.Close()

// Store returns a deterministic handle derived from the payload hash
handle, err := store.Put(rec)
if err != nil { log.Fatal(err) }

// Handle format: <<ccr:ccr_000001abcd>>
fmt.Println("CCR handle:", handle)

```

### Recovering the Original Bytes

```go
// Retrieve the exact payload from any Caveman process
orig, err := store.Get(handle)
if err != nil {
    if errors.Is(err, ccr.ErrNotFound) {
        log.Fatalf("handle %s not found", handle)
    }
    log.Fatal(err)
}

// orig now holds the exact byte slice that was stored earlier
fmt.Printf("Recovered payload: %s\n", string(orig))

```

### Embedding CCR Markers in Transformed Output

The proxy gateway in [`proxy/internal/gateway/gateway.go`](https://github.com/JuliusBrussee/caveman/blob/main/proxy/internal/gateway/gateway.go) (lines 1095-1100) injects handles into compressed requests using the `appendCCRMarker` helper:

```go
compressedBody := appendCCRMarker(originalBody, handle)
// Results in payload containing: <<ccr:ccr_000001abcd>>

```

## Summary

- **Immutable BLOBs**: The `original` column in [`engine/ccr/store_sqlite.go`](https://github.com/JuliusBrussee/caveman/blob/main/engine/ccr/store_sqlite.go) stores raw bytes without compression or encoding, ensuring the SQLite database preserves the exact sequence written by the tool.
- **Content-Addressable Handles**: The `Handle()` function generates deterministic identifiers from payload bytes, enabling idempotent storage and reliable lookup via `Put` and `Get`.
- **Defensive Copying**: The `Get` method returns a fresh byte slice via `append([]byte(nil), obj.Data...)` to prevent memory aliasing and guarantee byte-exact fidelity.
- **Concurrent Safety**: WAL mode and busy timeouts allow multiple Caveman processes to access the store simultaneously without data corruption or lost writes.

## Frequently Asked Questions

### Does the CCR SQLite store compress the original payload?

No. While the `Recovery` struct tracks compression metadata in the `Compressor` field, the `original` column always stores the raw, uncompressed bytes. Compression applies only to the transformed output sent to downstream services, not to the recovery data persisted in SQLite.

### How does the store handle concurrent writes from multiple processes?

The store initializes SQLite with `journal_mode(WAL)` and `busy_timeout(5000)`, enabling Write-Ahead Logging. This allows one writer and multiple readers to operate simultaneously without locking conflicts, ensuring that every successful `Put` persists before any `Get` can observe the data.

### What happens if I attempt to store identical payloads twice?

The deterministic handle generation detects duplicates through content-addressing. When `Put` receives a payload whose hash matches an existing record, it updates the metadata row but leaves the immutable `original` BLOB unchanged, returning the existing handle without creating redundant storage.

### Where does the actual byte copying occur during retrieval?

In [`engine/ccr/store_sqlite.go`](https://github.com/JuliusBrussee/caveman/blob/main/engine/ccr/store_sqlite.go), the `Get` method executes `append([]byte(nil), obj.Data...)` after the `SELECT original` query (lines 59-66). This explicitly allocates a new slice and copies the database bytes, preventing the caller from holding references to internal SQLite memory buffers that might be reused or mutated.