How to Build the Caveman Compression Engine CLI: Complete Setup Guide

Build the Caveman compression engine CLI by cloning the repository and running go build ./engine/cmd/caveman-engine to produce a native binary that implements the Compress, Detect, Retrieve, and Stats operations.

The Caveman compression engine is a self-contained Go binary designed for deterministic text compression in AI workflows. Learning how to build the Caveman compression engine CLI from source gives you full control over the compression pipeline, from the core engine logic to the SQLite-backed recovery store. This guide walks through the build process using the actual source structure found in the JuliusBrussee/caveman repository.

Prerequisites

Before building, ensure your environment meets the following requirements:

• Go 1.22 or later – The project uses Go modules and requires a recent toolchain.
• Git – Needed to clone the repository.

Verify your Go installation:

go version

Build Instructions

Follow these steps to compile the native binary from the source files located in engine/cmd/caveman-engine/main.go.

1. Clone the repository and navigate to the root directory:

git clone https://github.com/JuliusBrussee/caveman.git
cd caveman

2. Download and verify dependencies:

go mod tidy

3. Build the CLI binary:

go build ./engine/cmd/caveman-engine

This produces an executable at ./engine/cmd/caveman-engine/caveman-engine (platform-specific naming may vary).

4. Verify the build:

./engine/cmd/caveman-engine/caveman-engine --help

Architectural Overview

Understanding the source layout helps when customizing or debugging the build.

Entry Point and Command Dispatch

The CLI entry point resides in [engine/cmd/caveman-engine/main.go](https://github.com/JuliusBrussee/caveman/blob/main/engine/cmd/caveman-engine/main.go). The main function parses sub-commands (compress, detect, retrieve, stats) and dispatches to dedicated handlers like runCompress and runDetect.

Engine Core

The core logic lives in engine/engine.go, which defines the Engine type. This struct provides the stable API methods:

• Compress – Applies registered compressors to input bytes.
• RetrieveQuery – Fetches original data from CCR handles.
• Detect – Identifies content types.
• Stats – Returns aggregate compression metrics.

Compressor Registry

The default set of 15 compressors (JSON, log, code, diff, and others) is registered by compressors.Default() in engine/compressors/registry.go. Each compressor implements a pure byte transform, while the engine adds safety checks, token accounting, and optional recovery storage.

Recovery Store (CCR)

When lossy compression produces smaller payloads, the original bytes are stored in a local SQLite database at ~/.caveman/ccr.db. The openEngine helper in main.go initializes this store via ccr.Open(ccrPath()). If the store is unavailable or exceeds budget constraints, the engine returns the original input unchanged.

CLI Usage Examples

Once built, the binary reads from stdin (capped at 64 MiB as defined by maxStdinBytes in main.go) and writes compressed output to stdout, with JSON statistics emitted to stderr.

Compress a JSON payload:

cat payload.json | ./engine/cmd/caveman-engine/caveman-engine compress > compressed.bin

Detect content type:

echo "<html><body>Hello</body></html>" | ./engine/cmd/caveman-engine/caveman-engine detect

Retrieve original bytes using a CCR handle:

./engine/cmd/caveman-engine/caveman-engine retrieve abcdef12345 > original.json

Display aggregate statistics:

./engine/cmd/caveman-engine/caveman-engine stats

List compressor capabilities:

./engine/cmd/caveman-engine/caveman-engine registry

Runtime Configuration

Configure the CLI behavior using environment variables:

• CAVEMAN_CCR_DB – Override the default recovery store path (~/.caveman/ccr.db).
• CAVEMAN_ENGINE_BIN – Path to the binary, respected by the JavaScript launcher.

The CLI enforces a 64 MiB input limit on stdin. Streams exceeding this limit are rejected with an error.

Summary

• Build the Caveman compression engine CLI using go build ./engine/cmd/caveman-engine from the repository root.
• The binary entry point is engine/cmd/caveman-engine/main.go, which dispatches to sub-command handlers.
• Core compression logic resides in engine/engine.go with support from engine/compressors/registry.go.
• Recovery data is stored in a SQLite database managed by engine/ccr/store.go.
• The CLI supports four stable operations: Compress, Detect, Retrieve, and Stats.

Frequently Asked Questions

What Go version is required to build the Caveman compression engine CLI?

The project requires Go 1.22 or later to compile successfully. Verify your version with go version before building, as earlier versions may fail to resolve modern module dependencies.

Where is the compression logic implemented in the source code?

The core compression logic is implemented in engine/engine.go, specifically within the Engine type methods. The Compress method orchestrates the 15 registered compressors defined in engine/compressors/registry.go.

How does the CLI handle data recovery for lossy compression?

When a lossy compressor produces a smaller payload, the engine stores the original bytes in a SQLite database at ~/.caveman/ccr.db (configurable via CAVEMAN_CCR_DB). The retrieve sub-command uses ccr.Open() and RetrieveQuery to fetch the original data using a unique handle.

Can I install the binary globally on my system?

Yes. Move the built binary to a directory in your PATH, or set the CAVEMAN_ENGINE_BIN environment variable to point to the binary location for use with the JavaScript launcher.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}