Where Are the Caveman Compressor Implementations Located?

Caveman's compressor implementations reside in the engine/compressors package, where each supported content type (JSON, text, HTML, code, diff, log, and more) has its own dedicated Go source file.

The Caveman engine uses a pluggable, deterministic compression system to shrink large inputs before they reach downstream consumers. All compressor logic lives in a single package under the repository root, making it easy to locate, extend, and maintain. Whether you're debugging compression behavior or adding a new content-type handler, the engine/compressors directory is your starting point.

The Compressor Interface and Registry

The foundation of the system is defined in engine/compressors/compressor.go. This file declares the Compressor interface that every implementation must satisfy:

type Compressor interface {
    ContentType() string
    SafetyClass() SafetyClass
    Compress(data []byte) []byte
    // Plus optional methods for query-aware and metadata-driven compression
}

The same file houses the registry—a map from content-type strings to concrete compressors. The Default() function populates this registry with all built-in implementations at startup.

JSON Compressor: Lossy S4 Compression

For JSON payloads—especially repetitive tool outputs—the engine/compressors/json.go file provides an S4-style compressor. It elides redundant array elements while preserving error and message sub-trees that carry semantic weight.

This is particularly effective on large API responses or structured logs where objects repeat with minor variations.

Text and HTML Compressors

Plain-text and HTML fragments each have dedicated handlers:

• engine/compressors/text.go – A simple line-based compressor for unstructured text
• engine/compressors/html.go – Deterministic compression for HTML fragments, with auto-detection support

Both implement the same Compressor interface, ensuring consistent behavior across content types.

Code Compressor: Tree-Sitter and Go AST

Source code receives special treatment in engine/compressors/code_listing.go, with CGO-dependent logic split across:

• engine/compressors/code_cgo.go – Tree-sitter-backed parsing when CGO is available
• engine/compressors/code_nocgo.go – Fallback Go AST implementation

The compressor is registered via newCode() in the main registry, selecting the appropriate backend at compile time.

Specialized Compressors

The engine/compressors package includes handlers for domain-specific formats:

Compressor	File	Purpose
Tabular	tabular.go	CSV/TSV compression; keeps headers, samples data rows
Diff	diff.go	Reduces large diffs by limiting context lines
Log	log.go	Retains timestamps and error-level lines, compresses repetitive entries
Terminal	terminal.go	Handles ANSI-escaped terminal output
Repetition	repetition.go	Removes redundant segments in tool-schema outputs
TOON	toon.go	Compresses Tree-Object-Object-Node format used by the engine
Tool-Schema	toolschema.go, toolschema_annotations.go	Lossless and annotated compressors for custom tool-schema format

Auxiliary Utilities

Supporting logic lives in additional files within the same package:

• engine/compressors/redundancy.go – Redundancy elimination algorithms
• engine/compressors/invariants.go – Invariant extraction for structured data
• engine/compressors/relevance.go – Relevance scoring for content prioritization
• engine/compressors/axtree.go – Adaptive tree structures for hierarchical compression

How to Use the Compressor Registry

Retrieving and invoking a compressor follows a consistent pattern:

package main

import (
	"fmt"
	"github.com/JuliusBrussee/caveman/engine/compressors"
)

func main() {
	// Build the default registry with every built-in compressor.
	reg := compressors.Default()

	// Look up the JSON compressor.
	c, ok := reg.For("json")
	if !ok {
		panic("JSON compressor not registered")
	}

	// Sample JSON payload (a long array of similar objects).
	payload := []byte(`{"items":[{"id":1,"msg":"ok"},{"id":2,"msg":"ok"}, … ]}`)

	// Compress the payload.
	if out, ok := c.Compress(payload); ok {
		fmt.Printf("Compressed size: %d → %d bytes\n", len(payload), len(out))
	} else {
		fmt.Println("Compressor rejected payload; using original.")
	}
}

Change the string passed to reg.For() to target other compressors: "text", "html", "code", "diff", "log", "tabular", "terminal", "toolschema", "toon", or "repetition".

Summary

• All Caveman compressor implementations are located in engine/compressors/
• The Compressor interface and registry are defined in compressor.go
• Each content type has a dedicated source file: json.go, text.go, html.go, code_listing.go, tabular.go, diff.go, log.go, terminal.go, repetition.go, toon.go, and toolschema.go
• Auxiliary utilities for redundancy, invariants, and relevance live in the same package
• All compressors are deterministic, idempotent, and fail-closed—failed compression returns ok=false and preserves original bytes

Frequently Asked Questions

What happens if a compressor fails to parse input?

The compressor returns ok=false and the engine forwards the original bytes unchanged. This fail-closed design ensures no data loss occurs due to compression errors.

How do I add a custom compressor to Caveman?

Implement the Compressor interface defined in engine/compressors/compressor.go, then register your implementation in the Default() function with a unique content-type string.

Is the code compressor available without CGO?

Yes. When CGO is disabled, the code compressor falls back to a Go AST-based implementation in engine/compressors/code_nocgo.go. The tree-sitter backend in code_cgo.go provides richer parsing when available.

Where are the compressor tests located?

Each compressor has a corresponding *_test.go file in the engine/compressors directory: json_test.go, code_test.go, text_test.go, and others.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}