How the caveman-shrink Tool Compresses Tool Catalogs with Byte-Exact Recovery

The caveman-shrink tool compresses MCP/OpenAI tool-definition catalogs by applying a lossy schema transformation that reduces token count, while simultaneously persisting the original bytes to a durable SQLite-backed CCR store to enable perfect recovery via unique handles.

The caveman-shrink utility from the JuliusBrussee/caveman repository addresses context window limitations in LLM applications by minimizing tool catalog size without sacrificing recoverability. Written in Go, this tool implements a two-phase pipeline: an aggressive compression phase that strips non-essential metadata from tool schemas, and a persistence phase that stores the pristine original in a content-compression-recovery (CCR) database for later byte-exact retrieval.

Understanding the Compression Pipeline

The compression workflow centers on preserving the selection surface—the specific fields required for tool invocation—while discarding human-readable verbosity that consumes tokens but does not affect execution semantics.

Entry Point and Store Initialization

Compression begins in the Shrink function defined in [shrink/shrink.go](https://github.com/JuliusBrussee/caveman/blob/main/shrink/shrink.go#L17-L27). This function first initializes a recovery store via resolveStore, which defaults to ~/.caveman/ccr.db or respects the CAVEMAN_CCR_DB environment variable for custom paths. This store is a durable, disk-backed SQLite database that persists across process lifecycles, ensuring recovery handles remain valid after application restarts.

Tool-Schema Compression Strategy

With the store initialized, Shrink constructs an engine instance and invokes engine.Compress with schemaType = "toolschema" (line 24). According to the implementation in [engine/compressors/tool_schema.go](https://github.com/JuliusBrussee/caveman/blob/main/engine/compressors/tool_schema.go), the NewToolSchema compressor performs the following transformations:

• Drops annotation metadata such as verbose descriptions and non-essential documentation
• Truncates description strings to reduce token overhead
• Preserves the selection surface: tool names, parameter names, types, enum values, and required-field lists remain byte-identical

This selective lossy compression ensures the LLM retains all functional information needed to select and invoke tools, while significantly reducing the JSON payload size.

Durable Recovery Storage

Before emitting the compressed output, the engine writes the full original bytes to the CCR store as implemented in [engine/engine.go](https://github.com/JuliusBrussee/caveman/blob/main/engine/engine.go). The engine generates a unique recovery handle (res.RecoveryHandle) that acts as a cryptographic pointer to the stored original. This handle, along with compression metrics, is returned to the caller in a Result struct (lines 97-106 in shrink.go). If the compression algorithm determines that transformation would not reduce payload size, the function implements fail-open behavior by returning the original input with a ratio of 0 and no recovery handle, preventing storage bloat.

Exact Recovery Mechanism

To recover the original catalog, the Recover function in [shrink/shrink.go](https://github.com/JuliusBrussee/caveman/blob/main/shrink/shrink.go#L43-L49) opens the same CCR store using identical resolution logic and retrieves the pristine bytes via store.Get(handle). If the handle does not exist in the store, it returns ccr.ErrNotFound without attempting reconstruction or interpolation, guaranteeing that only byte-exact originals are ever returned.

Implementing Compression and Recovery in Go

The following example demonstrates programmatic usage of the compression API:

package main

import (
    "fmt"
    "os"
    "github.com/JuliusBrussee/caveman/shrink"
)

func main() {
    // Load a JSON catalog (MCP or OpenAI format)
    catalog, err := os.ReadFile("tools.json")
    if err != nil {
        panic(err)
    }

    // Compress the catalog
    res, err := shrink.Shrink(catalog)
    if err != nil {
        panic(err)
    }

    fmt.Printf("Compressed %d → %d tokens (%.2f%% reduction)\n",
        res.TokensBefore, res.TokensAfter, res.Ratio*100)

    // Store the recovery handle for later retrieval
    handle := res.RecoveryHandle
    compressed := res.Output

    // Later, or in a different process, recover the exact original
    original, err := shrink.Recover(handle)
    if err != nil {
        panic(err)
    }

    // Verify byte-exact equality
    if string(original) == string(catalog) {
        fmt.Println("Recovery successful: byte-exact match")
    }
}

The shrink.Result struct provides comprehensive metadata including TokensBefore, TokensAfter, Ratio, and the crucial RecoveryHandle required for restoration.

Command-Line Usage

The CLI wrapper in [shrink/cmd/caveman-shrink/main.go](https://github.com/JuliusBrussee/caveman/blob/main/shrink/cmd/caveman-shrink/main.go) exposes the same durable recovery semantics:

# Compress a catalog and capture the recovery handle

caveman-shrink compress tools.json > tools.shrunk.json

# Output includes: handle: <uuid>

# Recover using the printed handle

caveman-shrink recover <handle> > tools.recovered.json

# Verify integrity

diff tools.json tools.recovered.json && echo "Byte-exact recovery confirmed"

You can override the default database location by setting the CAVEMAN_CCR_DB environment variable to a custom file path before invoking compression or recovery commands.

Summary

• Lossy compression of tool schemas removes descriptions and annotations while preserving functional tool signatures in [engine/compressors/tool_schema.go](https://github.com/JuliusBrussee/caveman/blob/main/engine/compressors/tool_schema.go).
• Durable storage in a SQLite-backed CCR store at ~/.caveman/ccr.db (or CAVEMAN_CCR_DB) ensures recovery handles persist across process boundaries.
• Byte-exact recovery is guaranteed by storing the complete original payload before compression, retrievable only via the unique handle returned by shrink.Shrink().
• Fail-open behavior prevents storage waste when compression provides no benefit, returning the original payload with a zero ratio.

Frequently Asked Questions

What is the CCR store and where is it located?

The CCR (Content-Compression-Recovery) store is a persistent SQLite database defined in [engine/ccr/store.go](https://github.com/JuliusBrussee/caveman/blob/main/engine/ccr/store.go) that maps recovery handles to original byte payloads. By default, it resides at ~/.caveman/ccr.db, but you can specify an alternative path using the CAVEMAN_CCR_DB environment variable before running compression operations.

What happens if I lose the recovery handle?

If the recovery handle is lost, the original catalog bytes cannot be retrieved. The shrink.Recover() function will return ccr.ErrNotFound when queried with a non-existent handle. Because the store does not maintain an index of uncompressed-to-compressed mappings, you must preserve the handle alongside the compressed output to enable future recovery.

Does compression affect tool functionality or LLM selection accuracy?

No. According to the implementation in [engine/compressors/tool_schema.go](https://github.com/JuliusBrussee/caveman/blob/main/engine/compressors/tool_schema.go), the compressor specifically preserves the selection surface—including tool names, parameter names, types, enums, and required fields—ensuring the LLM retains all necessary information to select and invoke tools correctly. Only human-readable descriptions and non-functional metadata are removed.

How does caveman-shrink handle incompressible catalogs?

The tool implements fail-open logic in the Shrink function: if the compression algorithm determines that the transformed output would not be smaller than the input, it returns the original bytes with a ratio of 0 and no recovery handle. This prevents the CCR store from accumulating redundant entries for already-optimal payloads.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}