What Happens When CCR Storage Is Unavailable or Full During Compression in Caveman

When CCR storage is unavailable or full, Caveman’s Shrink function falls back to pass-through mode, returning the original uncompressed bytes alongside an operationalError while keeping the application operational.

The Caveman compression library (available at JuliusBrussee/caveman) implements a resilient "fail-open" strategy for its Content-Control-Recovery (CCR) subsystem. When the SQLite-based CCR storage becomes unavailable, corrupted, or exhausted during compression operations, the library prioritizes data availability over compression efficiency by transparently bypassing the persistence layer. This ensures that transient storage issues never cause data loss or application crashes.

How CCR Storage Failures Trigger Pass-Through Mode

In shrink/shrink.go, the Shrink function acts as the primary entry point for compression. Before returning compressed data, the function attempts to write a recovery handle to the CCR store via store.Put. When this operation encounters storage constraints, the library executes a deterministic fallback path.

The Shrink Function's Error Handling Logic

According to the source implementation in [shrink/shrink.go](https://github.com/JuliusBrussee/caveman/blob/main/shrink/shrink.go#L117-L158), the compression flow follows this sequence:

1. The payload is compressed using the core algorithm.
2. The function attempts to persist a CCR handle via store.Put.
3. If store.Put returns an error (indicating the store is closed, unreachable, or full), Shrink immediately returns the original input bytes unchanged.
4. The returned Result struct has its Compressed field set to false and RecoveryHandle left empty.
5. An operationalError wrapping the underlying storage failure is returned to the caller.

This behavior guarantees that even when the default CCR database (typically located at ~/.caveman/ccr.db or specified via CAVEMAN_CCR_DB) cannot be written to, the data still flows through the system uncompressed rather than being dropped or causing a panic.

Detecting Full or Corrupted Storage

The same fallback path activates for multiple storage failure modes:

• Unavailable store: When the SQLite database cannot be opened (missing permissions, corrupted file, or non-existent path).
• Full disk: When the write operation fails due to quota exhaustion or "database or disk is full" errors.
• Read-only environment: When the CCR path points to a read-only filesystem.

In all cases, the caller receives the original payload intact, allowing the application to continue operating while logging the storage issue.

Testing CCR Storage Unavailability

The fallback behavior is explicitly validated in the test suite. The test TestShrinkReturnsPassThroughResultWhenCCRIsUnavailable in [shrink/shrink_test.go](https://github.com/JuliusBrussee/caveman/blob/main/shrink/shrink_test.go#L188-L206) asserts that when the CCR store cannot be opened, the result contains the original payload bytes and the error is classified as an operational error rather than a fatal exception.

This test verifies the contract: storage failures must never result in data loss, only in degraded performance through uncompressed transfer.

Proxy-Layer Integration

Downstream consumers in the proxy layer also handle missing CCR storage gracefully. In [proxy/internal/gateway/compress.go](https://github.com/JuliusBrussee/caveman/blob/main/proxy/internal/gateway/compress.go#L306-L324), the gateway checks for empty RecoveryHandle values before attempting to attach CCR metadata to requests. When the handle is absent (indicating a prior storage failure), the proxy transmits the uncompressed data without modification.

Practical Implementation Examples

Handling CCR Store Failures in Application Code

When calling Shrink, always inspect the error and Compressed flag to detect when pass-through mode has been activated:

package main

import (
	"fmt"
	"strings"
	
	"github.com/juliusbrussee/caveman/shrink"
)

func processPayload(data []byte) ([]byte, error) {
	res, err := shrink.Shrink(data)
	if err != nil {
		// Check if this is an operational error related to CCR storage
		if strings.Contains(err.Error(), "closed") || 
		   strings.Contains(err.Error(), "unavailable") {
			fmt.Printf("CCR store issue: %v - proceeding with uncompressed data\n", err)
			// Data is still available in res.Data (original bytes)
			return res.Data, nil
		}
		return nil, err
	}
	
	if !res.Compressed {
		fmt.Println("Pass-through: data unchanged due to CCR constraints")
	} else {
		fmt.Printf("Compressed %d → %d bytes with handle %s\n", 
			len(data), len(res.Data), res.RecoveryHandle)
	}
	
	return res.Data, nil
}

Explicitly Disabling CCR Storage

For environments where CCR persistence is known to be impossible (such as read-only containers), you can proactively disable the store:

// Disable CCR storage to avoid operational errors
opt := shrink.WithStore(nil)
res, err := shrink.Shrink(payload, opt)
if err != nil {
	// Handle only compression algorithm errors, not storage errors
	log.Fatal(err)
}
fmt.Printf("Compressed without CCR: %v\n", res.Compressed)

Detecting Disk Full Conditions

The underlying SQLite errors propagate through the operationalError type, allowing specific handling for quota issues:

res, err := shrink.Shrink(payload)
if err != nil {
	errStr := err.Error()
	if strings.Contains(errStr, "database or disk is full") {
		log.Println("CCR quota exceeded - consider cleaning old entries or rotating the store")
		// res.Data contains original uncompressed payload
		return res.Data, nil
	}
	return nil, err
}

Summary

• Fail-open design: When CCR storage is unavailable or full, Shrink returns original bytes rather than failing catastrophically.
• Operational continuity: The operationalError type indicates transient storage issues that don't require application restart.
• Pass-through indicators: Check Result.Compressed == false and empty RecoveryHandle to detect when compression was bypassed.
• Source locations: Core logic resides in shrink/shrink.go (lines 117-158), with tests in shrink/shrink_test.go (lines 188-206).
• Proxy resilience: The gateway layer in proxy/internal/gateway/compress.go handles missing CCR handles by transmitting data unchanged.

Frequently Asked Questions

Does data get lost if the CCR store becomes full during compression?

No. When the CCR store encounters a "disk full" error during the store.Put operation, the Shrink function immediately returns the original uncompressed bytes. The data remains intact and accessible through the Result.Data field, ensuring zero data loss even when storage is exhausted.

How can I detect if compression was skipped due to CCR storage issues?

Inspect the Result struct returned by Shrink. If res.Compressed is false and res.RecoveryHandle is empty while an error is present, the library has entered pass-through mode. The error will be of type operationalError indicating the CCR store was closed or unavailable.

Can I run Caveman compression without any CCR storage?

Yes. You can explicitly disable CCR persistence by passing shrink.WithStore(nil) as an option to Shrink. This configuration prevents operational errors related to storage availability and forces the library to operate in a pure compression-only mode without recovery handle generation.

What file path does Caveman use for CCR storage by default?

By default, Caveman stores CCR handles in a SQLite database located at ~/.caveman/ccr.db. You can override this location by setting the CAVEMAN_CCR_DB environment variable to a custom file path before initializing the compression library.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}