# How the caveman-shrink Tool Compresses Tool Catalogs with Byte-Exact Recovery

> Learn how caveman-shrink compresses tool catalogs with byte-exact recovery. This tool uses lossy schema transformation and durable storage for perfect original data restoration. Optimize your tool definitions today.

- Repository: [Julius Brussee/caveman](https://github.com/JuliusBrussee/caveman)
- Tags: how-to-guide
- Published: 2026-09-04

---

**The `caveman-shrink` tool compresses MCP/OpenAI tool-definition catalogs by applying a lossy schema transformation that reduces token count, while simultaneously persisting the original bytes to a durable SQLite-backed CCR store to enable perfect recovery via unique handles.**

The `caveman-shrink` utility from the JuliusBrussee/caveman repository addresses context window limitations in LLM applications by minimizing tool catalog size without sacrificing recoverability. Written in Go, this tool implements a two-phase pipeline: an aggressive compression phase that strips non-essential metadata from tool schemas, and a persistence phase that stores the pristine original in a content-compression-recovery (CCR) database for later byte-exact retrieval.

## Understanding the Compression Pipeline

The compression workflow centers on preserving the *selection surface*—the specific fields required for tool invocation—while discarding human-readable verbosity that consumes tokens but does not affect execution semantics.

### Entry Point and Store Initialization

Compression begins in the `Shrink` function defined in [[`shrink/shrink.go`](https://github.com/JuliusBrussee/caveman/blob/main/shrink/shrink.go)](https://github.com/JuliusBrussee/caveman/blob/main/shrink/shrink.go#L17-L27). This function first initializes a recovery store via `resolveStore`, which defaults to `~/.caveman/ccr.db` or respects the `CAVEMAN_CCR_DB` environment variable for custom paths. This store is a durable, disk-backed SQLite database that persists across process lifecycles, ensuring recovery handles remain valid after application restarts.

### Tool-Schema Compression Strategy

With the store initialized, `Shrink` constructs an engine instance and invokes `engine.Compress` with `schemaType = "toolschema"` (line 24). According to the implementation in [[`engine/compressors/tool_schema.go`](https://github.com/JuliusBrussee/caveman/blob/main/engine/compressors/tool_schema.go)](https://github.com/JuliusBrussee/caveman/blob/main/engine/compressors/tool_schema.go), the `NewToolSchema` compressor performs the following transformations:

- **Drops annotation metadata** such as verbose descriptions and non-essential documentation
- **Truncates description strings** to reduce token overhead
- **Preserves the selection surface**: tool names, parameter names, types, enum values, and required-field lists remain byte-identical

This selective lossy compression ensures the LLM retains all functional information needed to select and invoke tools, while significantly reducing the JSON payload size.

### Durable Recovery Storage

Before emitting the compressed output, the engine writes the **full original bytes** to the CCR store as implemented in [[`engine/engine.go`](https://github.com/JuliusBrussee/caveman/blob/main/engine/engine.go)](https://github.com/JuliusBrussee/caveman/blob/main/engine/engine.go). The engine generates a unique **recovery handle** (`res.RecoveryHandle`) that acts as a cryptographic pointer to the stored original. This handle, along with compression metrics, is returned to the caller in a `Result` struct (lines 97-106 in [`shrink.go`](https://github.com/JuliusBrussee/caveman/blob/main/shrink.go)). If the compression algorithm determines that transformation would not reduce payload size, the function implements fail-open behavior by returning the original input with a ratio of 0 and no recovery handle, preventing storage bloat.

### Exact Recovery Mechanism

To recover the original catalog, the `Recover` function in [[`shrink/shrink.go`](https://github.com/JuliusBrussee/caveman/blob/main/shrink/shrink.go)](https://github.com/JuliusBrussee/caveman/blob/main/shrink/shrink.go#L43-L49) opens the same CCR store using identical resolution logic and retrieves the pristine bytes via `store.Get(handle)`. If the handle does not exist in the store, it returns `ccr.ErrNotFound` without attempting reconstruction or interpolation, guaranteeing that only byte-exact originals are ever returned.

## Implementing Compression and Recovery in Go

The following example demonstrates programmatic usage of the compression API:

```go
package main

import (
    "fmt"
    "os"
    "github.com/JuliusBrussee/caveman/shrink"
)

func main() {
    // Load a JSON catalog (MCP or OpenAI format)
    catalog, err := os.ReadFile("tools.json")
    if err != nil {
        panic(err)
    }

    // Compress the catalog
    res, err := shrink.Shrink(catalog)
    if err != nil {
        panic(err)
    }

    fmt.Printf("Compressed %d → %d tokens (%.2f%% reduction)\n",
        res.TokensBefore, res.TokensAfter, res.Ratio*100)

    // Store the recovery handle for later retrieval
    handle := res.RecoveryHandle
    compressed := res.Output

    // Later, or in a different process, recover the exact original
    original, err := shrink.Recover(handle)
    if err != nil {
        panic(err)
    }

    // Verify byte-exact equality
    if string(original) == string(catalog) {
        fmt.Println("Recovery successful: byte-exact match")
    }
}

```

The `shrink.Result` struct provides comprehensive metadata including `TokensBefore`, `TokensAfter`, `Ratio`, and the crucial `RecoveryHandle` required for restoration.

## Command-Line Usage

The CLI wrapper in [[`shrink/cmd/caveman-shrink/main.go`](https://github.com/JuliusBrussee/caveman/blob/main/shrink/cmd/caveman-shrink/main.go)](https://github.com/JuliusBrussee/caveman/blob/main/shrink/cmd/caveman-shrink/main.go) exposes the same durable recovery semantics:

```bash

# Compress a catalog and capture the recovery handle

caveman-shrink compress tools.json > tools.shrunk.json

# Output includes: handle: <uuid>

# Recover using the printed handle

caveman-shrink recover <handle> > tools.recovered.json

# Verify integrity

diff tools.json tools.recovered.json && echo "Byte-exact recovery confirmed"

```

You can override the default database location by setting the `CAVEMAN_CCR_DB` environment variable to a custom file path before invoking compression or recovery commands.

## Summary

- **Lossy compression** of tool schemas removes descriptions and annotations while preserving functional tool signatures in [[`engine/compressors/tool_schema.go`](https://github.com/JuliusBrussee/caveman/blob/main/engine/compressors/tool_schema.go)](https://github.com/JuliusBrussee/caveman/blob/main/engine/compressors/tool_schema.go).
- **Durable storage** in a SQLite-backed CCR store at `~/.caveman/ccr.db` (or `CAVEMAN_CCR_DB`) ensures recovery handles persist across process boundaries.
- **Byte-exact recovery** is guaranteed by storing the complete original payload before compression, retrievable only via the unique handle returned by `shrink.Shrink()`.
- **Fail-open behavior** prevents storage waste when compression provides no benefit, returning the original payload with a zero ratio.

## Frequently Asked Questions

### What is the CCR store and where is it located?

The CCR (Content-Compression-Recovery) store is a persistent SQLite database defined in [[`engine/ccr/store.go`](https://github.com/JuliusBrussee/caveman/blob/main/engine/ccr/store.go)](https://github.com/JuliusBrussee/caveman/blob/main/engine/ccr/store.go) that maps recovery handles to original byte payloads. By default, it resides at `~/.caveman/ccr.db`, but you can specify an alternative path using the `CAVEMAN_CCR_DB` environment variable before running compression operations.

### What happens if I lose the recovery handle?

If the recovery handle is lost, the original catalog bytes cannot be retrieved. The `shrink.Recover()` function will return `ccr.ErrNotFound` when queried with a non-existent handle. Because the store does not maintain an index of uncompressed-to-compressed mappings, you must preserve the handle alongside the compressed output to enable future recovery.

### Does compression affect tool functionality or LLM selection accuracy?

No. According to the implementation in [[`engine/compressors/tool_schema.go`](https://github.com/JuliusBrussee/caveman/blob/main/engine/compressors/tool_schema.go)](https://github.com/JuliusBrussee/caveman/blob/main/engine/compressors/tool_schema.go), the compressor specifically preserves the *selection surface*—including tool names, parameter names, types, enums, and required fields—ensuring the LLM retains all necessary information to select and invoke tools correctly. Only human-readable descriptions and non-functional metadata are removed.

### How does caveman-shrink handle incompressible catalogs?

The tool implements fail-open logic in the `Shrink` function: if the compression algorithm determines that the transformed output would not be smaller than the input, it returns the original bytes with a ratio of 0 and no recovery handle. This prevents the CCR store from accumulating redundant entries for already-optimal payloads.