# Where Are the Caveman Compressor Implementations Located?

> Discover where Caveman compressor implementations are located in the JuliusBrussee/caveman repository. Find dedicated Go files for JSON text HTML code diff log and more within the engine compressors package.

- Repository: [Julius Brussee/caveman](https://github.com/JuliusBrussee/caveman)
- Tags: how-to-guide
- Published: 2026-09-06

---

**Caveman's compressor implementations reside in the `engine/compressors` package, where each supported content type (JSON, text, HTML, code, diff, log, and more) has its own dedicated Go source file.**

The Caveman engine uses a pluggable, deterministic compression system to shrink large inputs before they reach downstream consumers. All compressor logic lives in a single package under the repository root, making it easy to locate, extend, and maintain. Whether you're debugging compression behavior or adding a new content-type handler, the `engine/compressors` directory is your starting point.

## The Compressor Interface and Registry

The foundation of the system is defined in [`engine/compressors/compressor.go`](https://github.com/JuliusBrussee/caveman/blob/main/engine/compressors/compressor.go). This file declares the `Compressor` interface that every implementation must satisfy:

```go
type Compressor interface {
    ContentType() string
    SafetyClass() SafetyClass
    Compress(data []byte) []byte
    // Plus optional methods for query-aware and metadata-driven compression
}

```

The same file houses the **registry**—a map from content-type strings to concrete compressors. The `Default()` function populates this registry with all built-in implementations at startup.

## JSON Compressor: Lossy S4 Compression

For JSON payloads—especially repetitive tool outputs—the [`engine/compressors/json.go`](https://github.com/JuliusBrussee/caveman/blob/main/engine/compressors/json.go) file provides an S4-style compressor. It elides redundant array elements while preserving error and message sub-trees that carry semantic weight.

This is particularly effective on large API responses or structured logs where objects repeat with minor variations.

## Text and HTML Compressors

Plain-text and HTML fragments each have dedicated handlers:

- **[`engine/compressors/text.go`](https://github.com/JuliusBrussee/caveman/blob/main/engine/compressors/text.go)** – A simple line-based compressor for unstructured text
- **[`engine/compressors/html.go`](https://github.com/JuliusBrussee/caveman/blob/main/engine/compressors/html.go)** – Deterministic compression for HTML fragments, with auto-detection support

Both implement the same `Compressor` interface, ensuring consistent behavior across content types.

## Code Compressor: Tree-Sitter and Go AST

Source code receives special treatment in [`engine/compressors/code_listing.go`](https://github.com/JuliusBrussee/caveman/blob/main/engine/compressors/code_listing.go), with CGO-dependent logic split across:

- [`engine/compressors/code_cgo.go`](https://github.com/JuliusBrussee/caveman/blob/main/engine/compressors/code_cgo.go) – Tree-sitter-backed parsing when CGO is available
- [`engine/compressors/code_nocgo.go`](https://github.com/JuliusBrussee/caveman/blob/main/engine/compressors/code_nocgo.go) – Fallback Go AST implementation

The compressor is registered via `newCode()` in the main registry, selecting the appropriate backend at compile time.

## Specialized Compressors

The `engine/compressors` package includes handlers for domain-specific formats:

| Compressor | File | Purpose |
|---|---|---|
| **Tabular** | [`tabular.go`](https://github.com/JuliusBrussee/caveman/blob/main/tabular.go) | CSV/TSV compression; keeps headers, samples data rows |
| **Diff** | [`diff.go`](https://github.com/JuliusBrussee/caveman/blob/main/diff.go) | Reduces large diffs by limiting context lines |
| **Log** | [`log.go`](https://github.com/JuliusBrussee/caveman/blob/main/log.go) | Retains timestamps and error-level lines, compresses repetitive entries |
| **Terminal** | [`terminal.go`](https://github.com/JuliusBrussee/caveman/blob/main/terminal.go) | Handles ANSI-escaped terminal output |
| **Repetition** | [`repetition.go`](https://github.com/JuliusBrussee/caveman/blob/main/repetition.go) | Removes redundant segments in tool-schema outputs |
| **TOON** | [`toon.go`](https://github.com/JuliusBrussee/caveman/blob/main/toon.go) | Compresses Tree-Object-Object-Node format used by the engine |
| **Tool-Schema** | [`toolschema.go`](https://github.com/JuliusBrussee/caveman/blob/main/toolschema.go), [`toolschema_annotations.go`](https://github.com/JuliusBrussee/caveman/blob/main/toolschema_annotations.go) | Lossless and annotated compressors for custom tool-schema format |

## Auxiliary Utilities

Supporting logic lives in additional files within the same package:

- [`engine/compressors/redundancy.go`](https://github.com/JuliusBrussee/caveman/blob/main/engine/compressors/redundancy.go) – Redundancy elimination algorithms
- [`engine/compressors/invariants.go`](https://github.com/JuliusBrussee/caveman/blob/main/engine/compressors/invariants.go) – Invariant extraction for structured data
- [`engine/compressors/relevance.go`](https://github.com/JuliusBrussee/caveman/blob/main/engine/compressors/relevance.go) – Relevance scoring for content prioritization
- [`engine/compressors/axtree.go`](https://github.com/JuliusBrussee/caveman/blob/main/engine/compressors/axtree.go) – Adaptive tree structures for hierarchical compression

## How to Use the Compressor Registry

Retrieving and invoking a compressor follows a consistent pattern:

```go
package main

import (
	"fmt"
	"github.com/JuliusBrussee/caveman/engine/compressors"
)

func main() {
	// Build the default registry with every built-in compressor.
	reg := compressors.Default()

	// Look up the JSON compressor.
	c, ok := reg.For("json")
	if !ok {
		panic("JSON compressor not registered")
	}

	// Sample JSON payload (a long array of similar objects).
	payload := []byte(`{"items":[{"id":1,"msg":"ok"},{"id":2,"msg":"ok"}, … ]}`)

	// Compress the payload.
	if out, ok := c.Compress(payload); ok {
		fmt.Printf("Compressed size: %d → %d bytes\n", len(payload), len(out))
	} else {
		fmt.Println("Compressor rejected payload; using original.")
	}
}

```

Change the string passed to `reg.For()` to target other compressors: `"text"`, `"html"`, `"code"`, `"diff"`, `"log"`, `"tabular"`, `"terminal"`, `"toolschema"`, `"toon"`, or `"repetition"`.

## Summary

- All Caveman compressor implementations are located in **`engine/compressors/`**
- The **`Compressor` interface** and **registry** are defined in [`compressor.go`](https://github.com/JuliusBrussee/caveman/blob/main/compressor.go)
- Each content type has a **dedicated source file**: [`json.go`](https://github.com/JuliusBrussee/caveman/blob/main/json.go), [`text.go`](https://github.com/JuliusBrussee/caveman/blob/main/text.go), [`html.go`](https://github.com/JuliusBrussee/caveman/blob/main/html.go), [`code_listing.go`](https://github.com/JuliusBrussee/caveman/blob/main/code_listing.go), [`tabular.go`](https://github.com/JuliusBrussee/caveman/blob/main/tabular.go), [`diff.go`](https://github.com/JuliusBrussee/caveman/blob/main/diff.go), [`log.go`](https://github.com/JuliusBrussee/caveman/blob/main/log.go), [`terminal.go`](https://github.com/JuliusBrussee/caveman/blob/main/terminal.go), [`repetition.go`](https://github.com/JuliusBrussee/caveman/blob/main/repetition.go), [`toon.go`](https://github.com/JuliusBrussee/caveman/blob/main/toon.go), and [`toolschema.go`](https://github.com/JuliusBrussee/caveman/blob/main/toolschema.go)
- **Auxiliary utilities** for redundancy, invariants, and relevance live in the same package
- All compressors are **deterministic, idempotent, and fail-closed**—failed compression returns `ok=false` and preserves original bytes

## Frequently Asked Questions

### What happens if a compressor fails to parse input?

The compressor returns `ok=false` and the engine forwards the original bytes unchanged. This **fail-closed** design ensures no data loss occurs due to compression errors.

### How do I add a custom compressor to Caveman?

Implement the `Compressor` interface defined in [`engine/compressors/compressor.go`](https://github.com/JuliusBrussee/caveman/blob/main/engine/compressors/compressor.go), then register your implementation in the `Default()` function with a unique content-type string.

### Is the code compressor available without CGO?

Yes. When CGO is disabled, the code compressor falls back to a Go AST-based implementation in [`engine/compressors/code_nocgo.go`](https://github.com/JuliusBrussee/caveman/blob/main/engine/compressors/code_nocgo.go). The tree-sitter backend in [`code_cgo.go`](https://github.com/JuliusBrussee/caveman/blob/main/code_cgo.go) provides richer parsing when available.

### Where are the compressor tests located?

Each compressor has a corresponding `*_test.go` file in the `engine/compressors` directory: [`json_test.go`](https://github.com/JuliusBrussee/caveman/blob/main/json_test.go), [`code_test.go`](https://github.com/JuliusBrussee/caveman/blob/main/code_test.go), [`text_test.go`](https://github.com/JuliusBrussee/caveman/blob/main/text_test.go), and others.