# Caveman CCR Storage Backends: SQLite vs In-Memory Implementation

> Explore Caveman CCR storage backends: SQLite for persistence and in-memory for WebAssembly. Seamless content compression recovery across platforms with a unified Store interface.

- Repository: [Julius Brussee/caveman](https://github.com/JuliusBrussee/caveman)
- Tags: deep-dive
- Published: 2026-09-06

---

**Caveman CCR provides two production-ready storage backends—a SQLite-backed persistent store for native builds and an in-memory hash map for WebAssembly targets—unified behind a common `Store` interface that enables seamless content compression recovery across platforms.**

The JuliusBrussee/caveman repository implements a Content-Compression-Recovery (CCR) layer that abstracts storage details behind a type-safe API. Understanding the available Caveman CCR storage backends is essential for configuring deployment environments, from persistent server applications to ephemeral browser contexts.

## Overview of Caveman CCR Storage Backends

The CCR layer abstracts storage through a shared `Store` type defined in [`engine/ccr/store.go`](https://github.com/JuliusBrussee/caveman/blob/main/engine/ccr/store.go). This abstraction allows the rest of the codebase—from the inference engine to the MCP server—to remain agnostic to whether data persists to disk or lives only in memory.

The implementation provides two concrete backends selected automatically at compile time via Go build tags:

- **SQLite-backed persistent store**: The default for native (non-Web) builds, providing durable storage across process restarts with configurable budget limits.
- **In-memory map**: Used exclusively for WebAssembly/JavaScript builds where filesystem access is unavailable and data persistence is impossible.

## SQLite-Backed Persistent Storage

In [`engine/ccr/store_sqlite.go`](https://github.com/JuliusBrussee/caveman/blob/main/engine/ccr/store_sqlite.go), the SQLite backend provides durable, budget-enforced storage for native applications.

### Architecture and Concurrency

The SQLite implementation uses a single-writer connection optimized for concurrent access:

- **Build constraint**: Defined with `//go:build !js` to exclude WebAssembly targets.
- **Connection settings**: Configured with `busy_timeout=5000` and `journal_mode=WAL` to minimize lock contention across concurrent Caveman processes.
- **Storage schema**: Maintains two tables—`recoveries` for raw compressed payloads and `typed_objects` for structured metadata about tool results and session data.

### Budget Enforcement and Configuration

The SQLite backend enforces a configurable storage budget to prevent unbounded disk usage:

- **Default location**: `~/.caveman/ccr.db`, overridable via the `CAVEMAN_CCR_DB` environment variable.
- **Budget control**: The `CAVEMAN_CCR_MAX_BYTES` environment variable sets the maximum storage allocation (default 512 MiB).
- **Quota enforcement**: When writes exceed the configured budget, the store returns `ErrBudgetExceeded` rather than consuming additional disk space.

### Opening the SQLite Store

```go
import (
    "log"
    "github.com/JuliusBrussee/caveman/engine/ccr"
)

func main() {
    // Empty path selects the default location or respects CAVEMAN_CCR_DB
    store, err := ccr.Open("")
    if err != nil {
        log.Fatalf("failed to open CCR store: %v", err)
    }
    defer store.Close()

    handle, err := store.Put(ccr.Recovery{
        Original:    []byte("original payload"),
        ContentType: "text/plain",
        TokensBefore: 100,
        TokensAfter:  20,
    })
    if err != nil {
        log.Fatalf("Put failed: %v", err)
    }
    log.Printf("CCR handle: %s", handle)
}

```

## In-Memory WebAssembly Backend

For browser and JavaScript environments, [`engine/ccr/store_wasm.go`](https://github.com/JuliusBrussee/caveman/blob/main/engine/ccr/store_wasm.go) implements a volatile, in-memory storage layer.

### Implementation Details

The WebAssembly backend adapts to sandboxed browser environments:

- **Build constraint**: Active only when compiled with `js && wasm` build tags.
- **Data structures**: Uses `map[string]record` for raw recovery data and `map[string]Object` for typed storage.
- **Concurrency**: All operations are guarded by a `sync.Mutex` to ensure thread-safety within the single-threaded WebAssembly runtime.
- **Lifetime**: Data persists only for the duration of the page session; all CCR content is lost when the browser tab unloads.

### Usage in WebAssembly Contexts

The API remains identical to the SQLite backend, allowing portable code across platforms:

```go
import (
    "log"
    "github.com/JuliusBrussee/caveman/engine/ccr"
)

func main() {
    // In Wasm builds, the path argument is ignored; returns in-memory store
    store, err := ccr.Open("")
    if err != nil {
        log.Fatalf("failed to open CCR store: %v", err)
    }
    defer store.Close()

    handle, _ := store.Put(ccr.Recovery{
        Original:    []byte("wasm payload"),
        ContentType: "application/json",
        TokensBefore: 50,
        TokensAfter:  10,
    })
    recovered, _ := store.Get(handle)
    log.Printf("Recovered in Wasm: %s", string(recovered))
}

```

## Common Store Interface

Both backends implement the identical `Store` interface defined in [`engine/ccr/store.go`](https://github.com/JuliusBrussee/caveman/blob/main/engine/ccr/store.go), ensuring behavioral parity across platforms:

- **`Open(path string)`**: Factory function that instantiates the appropriate backend based on compilation target.
- **`OpenWithBudget(path string, maxBytes int64)`**: Opens a store with explicit budget constraints.
- **`Put(rec Recovery) (string, error)`**: Stores compressed content and returns a handle.
- **`Get(handle string) ([]byte, error)`**: Retrieves original bytes by handle.
- **`PutObject(key string, obj Object) error`**: Stores typed metadata objects.
- **`GetObject(key string) (Object, error)`**: Retrieves structured metadata.

## Summary

- Caveman CCR provides **two storage backends**: SQLite for native persistence and in-memory maps for WebAssembly.
- The **SQLite backend** in [`engine/ccr/store_sqlite.go`](https://github.com/JuliusBrussee/caveman/blob/main/engine/ccr/store_sqlite.go) uses WAL mode and busy timeouts for concurrent access, defaulting to `~/.caveman/ccr.db` with a 512 MiB budget controlled via `CAVEMAN_CCR_MAX_BYTES`.
- The **WebAssembly backend** in [`engine/ccr/store_wasm.go`](https://github.com/JuliusBrussee/caveman/blob/main/engine/ccr/store_wasm.go) uses mutex-protected Go maps and loses all data on page unload, as browsers lack filesystem access.
- Both backends share the **same API surface** through the `Store` interface in [`engine/ccr/store.go`](https://github.com/JuliusBrussee/caveman/blob/main/engine/ccr/store.go), enabling compile-time backend selection without code changes.
- Storage limits are **enforced only in SQLite**; WebAssembly storage relies on the browser's memory management.

## Frequently Asked Questions

### What is the default storage backend for Caveman CCR?

For native builds (Linux, macOS, Windows), the default backend is the **SQLite-backed persistent store** implemented in [`engine/ccr/store_sqlite.go`](https://github.com/JuliusBrussee/caveman/blob/main/engine/ccr/store_sqlite.go). When compiling for WebAssembly with the `js && wasm` build tags, Caveman automatically switches to the **in-memory backend** from [`engine/ccr/store_wasm.go`](https://github.com/JuliusBrussee/caveman/blob/main/engine/ccr/store_wasm.go).

### How does Caveman CCR handle storage limits in the SQLite backend?

The SQLite backend enforces a configurable budget via the `CAVEMAN_CCR_MAX_BYTES` environment variable, defaulting to 512 MiB. When storage usage exceeds this limit, the `Put` method returns `ErrBudgetExceeded`, preventing new compressed payloads from being written to the database at `~/.caveman/ccr.db`.

### Can I use the SQLite backend in WebAssembly builds?

No. WebAssembly builds in browsers do not have access to the host filesystem, making persistent SQLite storage impossible. The `//go:build !js` constraint in [`engine/ccr/store_sqlite.go`](https://github.com/JuliusBrussee/caveman/blob/main/engine/ccr/store_sqlite.go) explicitly excludes WebAssembly targets, forcing the use of the in-memory implementation in [`engine/ccr/store_wasm.go`](https://github.com/JuliusBrussee/caveman/blob/main/engine/ccr/store_wasm.go).

### How does the Store interface ensure backend compatibility?

The `Store` interface declared in [`engine/ccr/store.go`](https://github.com/JuliusBrussee/caveman/blob/main/engine/ccr/store.go) defines common methods like `Open`, `Put`, `Get`, `PutObject`, and `GetObject`. Both [`store_sqlite.go`](https://github.com/JuliusBrussee/caveman/blob/main/store_sqlite.go) and [`store_wasm.go`](https://github.com/JuliusBrussee/caveman/blob/main/store_wasm.go) provide concrete implementations of this interface, allowing the `ccr.Open()` factory function to return the appropriate type at compile time while presenting a unified API to consuming code in the engine and MCP server components.