Caveman CCR Limitations: Security Risks, Archival Constraints, and Storage Boundaries

Caveman’s Content-Correction-Recovery (CCR) mechanism stores raw request/response payloads in an unencrypted local SQLite database with a 512 MiB default limit, exposing sensitive data to filesystem-level access while failing open to pass-through mode when storage is exhausted or corrupted.

Caveman is an open-source compression engine that uses Content-Correction-Recovery (CCR) to preserve exact original bytes during lossy transformations. While CCR enables reversible compression through a local SQLite store, several architectural constraints around Caveman CCR limitations affect how teams handle sensitive data and long-term archival. Understanding these boundaries is critical for secure deployment in production environments.

Data Sensitivity and Unencrypted Storage

The CCR database retains the exact original bytes of every compressed payload, including prompts, embedded credentials, and tool results. According to SECURITY.md lines 13-15, this data resides at ~/.caveman/ccr.db and must be treated as highly sensitive material. Unlike encrypted vaults, the store relies solely on Unix file permissions for protection, leaving raw payloads vulnerable to host compromise.

Filesystem Permission Constraints

The SQLite database is created with mode 0600 (read/write for owner only) and explicitly rejects symlinks or non-regular file paths. However, as documented in SECURITY.md lines 27-30, this represents filesystem access control rather than encryption. A compromised host or privileged user can still read the complete contents of the CCR file, creating a significant attack surface for credential theft or data exfiltration.

Storage Budget Cap and Exhaustion Behavior

CCR implements a hard default budget of 512 MiB to prevent unbounded disk growth. When the CAVEMAN_CCR_MAX_BYTES limit is reached, new compression operations fail to persist original payloads, forcing the engine into pass-through mode where lossy transforms proceed without recovery handles.

Configuring Budget Limits

Users can override the default budget by setting the CAVEMAN_CCR_MAX_BYTES environment variable. However, once exhausted, the system provides no mechanism for selective eviction—new writes simply fail, and shrink.Shrink() returns results with empty RecoveryHandle values. The following example demonstrates budget-aware configuration:

import (
    "os"
    "github.com/JuliusBrussee/caveman/shrink"
)

func demoCustomStore(input []byte) {
    // Use a custom CCR file and a reduced budget.
    os.Setenv("CAVEMAN_CCR_DB", "/tmp/myccr.db")
    os.Setenv("CAVEMAN_CCR_MAX_BYTES", "10485760") // 10 MiB

    // The store will be opened at the path above; when the 10 MiB budget is hit,
    // further Shrink calls will return the original payload with an empty handle.
    res, err := shrink.Shrink(input)
    if err != nil && res.RecoveryHandle == "" {
        fmt.Println("budget exhausted – fell back to pass-through")
    }
}

Durability Limitations and Corruption Handling

CCR distinguishes between ephemeral in-memory stores and durable disk-backed storage. As implemented in shrink/shrink.go lines 25-31, only durable stores yield recovery handles that survive process restarts. In-memory configurations mint handles that become unresolvable once the process terminates, breaking the reversibility guarantee.

Fail-Open Behavior on Corruption

When the CCR database is corrupted or unavailable, Caveman disables CCR usage entirely and forces record-pass-through mode. According to proxy/cmd/caveman-proxy/main.go lines 339-342, the proxy logs a warning and continues operation without native runtime compression. This fail-open design prioritizes availability over archival integrity, meaning security guarantees about payload storage vanish when the database fails.

import (
    "log"
    "github.com/JuliusBrussee/caveman/proxy"
)

func initProxy() {
    // The proxy will automatically detect a corrupted CCR file and
    // disable CCR usage, emitting a warning.
    err := proxy.Start()
    if err != nil {
        log.Fatalf("proxy failed to start: %v", err)
    }
    // No additional code needed – proxy internally forces record pass-through.
}

Practical Implementation Patterns

Despite these limitations, CCR remains useful for recovery workflows when properly configured. The durable store enables cross-process recovery, but teams must implement external encryption at rest and monitor storage budgets to prevent silent fallback to lossy compression.

import (
    "github.com/JuliusBrussee/caveman/shrink"
)

func demoShrinkRecover(input []byte) {
    // Shrink will write the exact original to the durable CCR store
    // (CAVEMAN_CCR_DB or ~/.caveman/ccr.db) and return a handle.
    res, err := shrink.Shrink(input)
    if err != nil {
        // Compression succeeded but CCR persistence failed → pass-through.
        // `res.RecoveryHandle` will be empty.
        fmt.Println("compression succeeded, but CCR unavailable:", err)
        return
    }

    fmt.Printf("compressed %d → %d bytes, handle=%s\n",
        res.TokensBefore, res.TokensAfter, res.RecoveryHandle)

    // Recover the original bytes later (even in another process)
    original, err := shrink.Recover(res.RecoveryHandle)
    if err != nil {
        fmt.Println("recovery failed:", err)
        return
    }
    fmt.Println("recovered original size:", len(original))
}

Summary

• Unencrypted storage: CCR stores raw payloads in ~/.caveman/ccr.db with only 0600 permissions, exposing sensitive data to host compromise without cryptographic protection.
• 512 MiB default limit: The CAVEMAN_CCR_MAX_BYTES budget caps archival capacity; exhaustion forces pass-through mode with empty recovery handles.
• Ephemeral in-memory stores: Only durable SQLite stores provide resolvable recovery handles across process boundaries.
• Corruption handling: Database corruption triggers record-pass-through mode, disabling CCR and original payload preservation while maintaining system availability.

Frequently Asked Questions

Is Caveman CCR data encrypted at rest?

No. According to the Caveman SECURITY.md documentation, CCR SQLite files rely exclusively on filesystem permissions (mode 0600) and reject symlinks, but they do not implement native encryption. Users requiring encrypted archival must implement filesystem-level encryption or external vault integration.

What happens when the CCR storage budget is exhausted?

When the CAVEMAN_CCR_MAX_BYTES limit (default 512 MiB) is reached, new shrink.Shrink() calls fail to persist original payloads. The engine falls back to pass-through mode, returning compressed data with empty RecoveryHandle values and preventing future recovery of the original bytes.

Can CCR recovery handles survive a process restart?

Only when using the durable store (the default ~/.caveman/ccr.db or a custom path via CAVEMAN_CCR_DB). As implemented in shrink/shrink.go, in-memory stores produce ephemeral handles that become unresolvable after the originating process terminates.

How does Caveman handle a corrupted CCR database?

The system detects corruption during proxy initialization and automatically disables CCR, forcing record-pass-through mode. According to proxy/cmd/caveman-proxy/main.go, this emits a warning log but allows continued operation, sacrificing archival recovery for system availability.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}