# Caveman CCR Limitations: Security Risks, Archival Constraints, and Storage Boundaries

> Discover Caveman CCR limitations: unencrypted data storage, 512 MiB limit, and security risks. Learn how its open-fail design impacts archival and exposes sensitive payloads.

- Repository: [Julius Brussee/caveman](https://github.com/JuliusBrussee/caveman)
- Tags: deep-dive
- Published: 2026-09-06

---

**Caveman’s Content-Correction-Recovery (CCR) mechanism stores raw request/response payloads in an unencrypted local SQLite database with a 512 MiB default limit, exposing sensitive data to filesystem-level access while failing open to pass-through mode when storage is exhausted or corrupted.**

Caveman is an open-source compression engine that uses Content-Correction-Recovery (CCR) to preserve exact original bytes during lossy transformations. While CCR enables reversible compression through a local SQLite store, several architectural constraints around **Caveman CCR limitations** affect how teams handle sensitive data and long-term archival. Understanding these boundaries is critical for secure deployment in production environments.

## Data Sensitivity and Unencrypted Storage

The CCR database retains the exact original bytes of every compressed payload, including prompts, embedded credentials, and tool results. According to [`SECURITY.md`](https://github.com/JuliusBrussee/caveman/blob/main/SECURITY.md) lines 13-15, this data resides at `~/.caveman/ccr.db` and must be treated as highly sensitive material. Unlike encrypted vaults, the store relies solely on Unix file permissions for protection, leaving raw payloads vulnerable to host compromise.

### Filesystem Permission Constraints

The SQLite database is created with mode `0600` (read/write for owner only) and explicitly rejects symlinks or non-regular file paths. However, as documented in [`SECURITY.md`](https://github.com/JuliusBrussee/caveman/blob/main/SECURITY.md) lines 27-30, this represents filesystem access control rather than encryption. A compromised host or privileged user can still read the complete contents of the CCR file, creating a significant attack surface for credential theft or data exfiltration.

## Storage Budget Cap and Exhaustion Behavior

CCR implements a hard default budget of **512 MiB** to prevent unbounded disk growth. When the `CAVEMAN_CCR_MAX_BYTES` limit is reached, new compression operations fail to persist original payloads, forcing the engine into pass-through mode where lossy transforms proceed without recovery handles.

### Configuring Budget Limits

Users can override the default budget by setting the `CAVEMAN_CCR_MAX_BYTES` environment variable. However, once exhausted, the system provides no mechanism for selective eviction—new writes simply fail, and `shrink.Shrink()` returns results with empty `RecoveryHandle` values. The following example demonstrates budget-aware configuration:

```go
import (
    "os"
    "github.com/JuliusBrussee/caveman/shrink"
)

func demoCustomStore(input []byte) {
    // Use a custom CCR file and a reduced budget.
    os.Setenv("CAVEMAN_CCR_DB", "/tmp/myccr.db")
    os.Setenv("CAVEMAN_CCR_MAX_BYTES", "10485760") // 10 MiB

    // The store will be opened at the path above; when the 10 MiB budget is hit,
    // further Shrink calls will return the original payload with an empty handle.
    res, err := shrink.Shrink(input)
    if err != nil && res.RecoveryHandle == "" {
        fmt.Println("budget exhausted – fell back to pass-through")
    }
}

```

## Durability Limitations and Corruption Handling

CCR distinguishes between ephemeral in-memory stores and durable disk-backed storage. As implemented in [`shrink/shrink.go`](https://github.com/JuliusBrussee/caveman/blob/main/shrink/shrink.go) lines 25-31, only durable stores yield recovery handles that survive process restarts. In-memory configurations mint handles that become unresolvable once the process terminates, breaking the reversibility guarantee.

### Fail-Open Behavior on Corruption

When the CCR database is corrupted or unavailable, Caveman disables CCR usage entirely and forces **record-pass-through** mode. According to [`proxy/cmd/caveman-proxy/main.go`](https://github.com/JuliusBrussee/caveman/blob/main/proxy/cmd/caveman-proxy/main.go) lines 339-342, the proxy logs a warning and continues operation without native runtime compression. This fail-open design prioritizes availability over archival integrity, meaning security guarantees about payload storage vanish when the database fails.

```go
import (
    "log"
    "github.com/JuliusBrussee/caveman/proxy"
)

func initProxy() {
    // The proxy will automatically detect a corrupted CCR file and
    // disable CCR usage, emitting a warning.
    err := proxy.Start()
    if err != nil {
        log.Fatalf("proxy failed to start: %v", err)
    }
    // No additional code needed – proxy internally forces record pass-through.
}

```

## Practical Implementation Patterns

Despite these limitations, CCR remains useful for recovery workflows when properly configured. The durable store enables cross-process recovery, but teams must implement external encryption at rest and monitor storage budgets to prevent silent fallback to lossy compression.

```go
import (
    "github.com/JuliusBrussee/caveman/shrink"
)

func demoShrinkRecover(input []byte) {
    // Shrink will write the exact original to the durable CCR store
    // (CAVEMAN_CCR_DB or ~/.caveman/ccr.db) and return a handle.
    res, err := shrink.Shrink(input)
    if err != nil {
        // Compression succeeded but CCR persistence failed → pass-through.
        // `res.RecoveryHandle` will be empty.
        fmt.Println("compression succeeded, but CCR unavailable:", err)
        return
    }

    fmt.Printf("compressed %d → %d bytes, handle=%s\n",
        res.TokensBefore, res.TokensAfter, res.RecoveryHandle)

    // Recover the original bytes later (even in another process)
    original, err := shrink.Recover(res.RecoveryHandle)
    if err != nil {
        fmt.Println("recovery failed:", err)
        return
    }
    fmt.Println("recovered original size:", len(original))
}

```

## Summary

- **Unencrypted storage**: CCR stores raw payloads in `~/.caveman/ccr.db` with only `0600` permissions, exposing sensitive data to host compromise without cryptographic protection.
- **512 MiB default limit**: The `CAVEMAN_CCR_MAX_BYTES` budget caps archival capacity; exhaustion forces pass-through mode with empty recovery handles.
- **Ephemeral in-memory stores**: Only durable SQLite stores provide resolvable recovery handles across process boundaries.
- **Corruption handling**: Database corruption triggers record-pass-through mode, disabling CCR and original payload preservation while maintaining system availability.

## Frequently Asked Questions

### Is Caveman CCR data encrypted at rest?

No. According to the Caveman [`SECURITY.md`](https://github.com/JuliusBrussee/caveman/blob/main/SECURITY.md) documentation, CCR SQLite files rely exclusively on filesystem permissions (mode `0600`) and reject symlinks, but they do not implement native encryption. Users requiring encrypted archival must implement filesystem-level encryption or external vault integration.

### What happens when the CCR storage budget is exhausted?

When the `CAVEMAN_CCR_MAX_BYTES` limit (default 512 MiB) is reached, new `shrink.Shrink()` calls fail to persist original payloads. The engine falls back to pass-through mode, returning compressed data with empty `RecoveryHandle` values and preventing future recovery of the original bytes.

### Can CCR recovery handles survive a process restart?

Only when using the durable store (the default `~/.caveman/ccr.db` or a custom path via `CAVEMAN_CCR_DB`). As implemented in [`shrink/shrink.go`](https://github.com/JuliusBrussee/caveman/blob/main/shrink/shrink.go), in-memory stores produce ephemeral handles that become unresolvable after the originating process terminates.

### How does Caveman handle a corrupted CCR database?

The system detects corruption during proxy initialization and automatically disables CCR, forcing record-pass-through mode. According to [`proxy/cmd/caveman-proxy/main.go`](https://github.com/JuliusBrussee/caveman/blob/main/proxy/cmd/caveman-proxy/main.go), this emits a warning log but allows continued operation, sacrificing archival recovery for system availability.