# How TOON Re-encoding Compresses Text Visually for Smaller Model Input

> Discover TOON re-encoding to compress JSON text visually, slashing token counts by 30-50% for smaller model input without losing data fidelity. Learn how it works.

- Repository: [Julius Brussee/caveman](https://github.com/JuliusBrussee/caveman)
- Tags: deep-dive
- Published: 2026-09-04

---

**TOON (Tabular Object-Oriented Notation) re-encodes uniform JSON arrays as delimiter-separated tables, eliminating braces, quotation marks, and repeated keys to reduce token counts by 30–50% while maintaining lossless data fidelity.**

The Caveman engine leverages TOON re-encoding to minimize the token footprint of structured data consumed by large language models. By transforming repetitive JSON objects into compact tabular representations, the system reduces visual clutter in the context window without sacrificing information integrity.

## The Three-Stage TOON Compression Pipeline

The compression process operates through a strict validation and transformation workflow implemented in the core engine.

### Detecting Uniform Tabular JSON

The compressor first validates that the incoming data matches a *uniform tabular* shape. In [`engine/compressors/toon.go`](https://github.com/JuliusBrussee/caveman/blob/main/engine/compressors/toon.go), the function `parseJSONTOON` checks whether the top-level value is an array of objects where every element shares the identical set of keys. If the JSON is malformed or the structure is non-uniform, the compressor aborts immediately and returns the original payload unchanged【/cache/repos/github.com/JuliusBrussee/caveman/main/engine/compressors/toon.go†L9-L12】.

### Encoding to Delimiter-Separated Format

Once validation passes, the `encodeTOON` function (invoked from `Compress`) constructs a dense tabular representation:

- **Header row**: Contains the shared object keys emitted once
- **Data rows**: Scalar values ordered to match the header sequence
- **Compact syntax**: Eliminates JSON punctuation (`{`, `}`, `:`, `"`) in favor of a single delimiter defined in `EncodeOptions{Delimiter: ','}`

The function returns a byte slice containing only the essential data separated by commas, and the compressor signals success with `ok = true` only when conversion completes【/cache/repos/github.com/JuliusBrussee/caveman/main/engine/compressors/toon.go†L12-L24】.

### Proxy Integration and Forced Re-encoding

The proxy layer can override normal compression thresholds to force TOON encoding for tool results. When the environment variable `CAVE_ENGINE_TOON=best-of` is set, the function `forcedTOONCandidate` in [`proxy/providers/openai/content_compress.go`](https://github.com/JuliusBrussee/caveman/blob/main/proxy/providers/openai/content_compress.go) checks if a candidate block contains tool JSON payload. If conditions match, the block routes through `compressors.NewTOON().Compress` regardless of size constraints【/__modal/volumes/vo-cSqLfqnnIwYXEonuEJnnZa/repos/github.com/JuliusBrussee/caveman/main/proxy/providers/openai/content_compress.go†L429-L436】.

## Why TOON Visually Compresses Text

The token efficiency stems from three structural optimizations:

- **Single Header Emission**: Column names appear once at the top of the table rather than repeating in every object, drastically reducing redundancy for large arrays.
- **Delimiter-Only Syntax**: TOON strips all JSON structural characters—braces, quotation marks, and colons—replacing them with a single lightweight delimiter between fields.
- **Lossless Round-Trip**: The `DecodeTOON` implementation in [`engine/toon.go`](https://github.com/JuliusBrussee/caveman/blob/main/engine/toon.go) validates the tabular format and reconstructs the original JSON structure, ensuring the model receives full-fidelity data in a visually compact representation【/cache/repos/github.com/JuliusBrussee/caveman/main/engine/toon.go†L20-L30】.

## Practical CLI Usage

**Encode JSON to TOON:**

```bash
caveman toon encode <<'EOF'
[
  {"id":1,"name":"Alice","age":30},
  {"id":2,"name":"Bob","age":25}
]
EOF

```

*Output:*

```

id,name,age
1,Alice,30
2,Bob,25

```

**Force TOON in Proxy Sessions:**

```bash
CAVE_ENGINE_TOON=best-of caveman wrap --toon my-agent

```

**Decode TOON back to JSON:**

```bash
caveman toon decode <<'EOF'
id,name,age
1,Alice,30
2,Bob,25
EOF

```

## Summary

- `parseJSONTOON` in [`engine/compressors/toon.go`](https://github.com/JuliusBrussee/caveman/blob/main/engine/compressors/toon.go) validates uniform tabular structure before compression
- Re-encoding via `encodeTOON` eliminates JSON syntactic noise using delimiter-only syntax
- The proxy’s `forcedTOONCandidate` enables mandatory TOON conversion via `CAVE_ENGINE_TOON=best-of`
- TOON achieves 30–50% token reduction for large uniform datasets compared to standard JSON
- Lossless decoding through `DecodeTOON` in [`engine/toon.go`](https://github.com/JuliusBrussee/caveman/blob/main/engine/toon.go) preserves data integrity for downstream agents

## Frequently Asked Questions

### How does TOON handle non-uniform JSON arrays?

If `parseJSONTOON` detects that objects within the array contain different keys or inconsistent shapes, the compressor aborts and returns the original JSON unchanged. This validation occurs in [`engine/compressors/toon.go`](https://github.com/JuliusBrussee/caveman/blob/main/engine/compressors/toon.go) to ensure only compatible data enters the tabular format【/cache/repos/github.com/JuliusBrussee/caveman/main/engine/compressors/toon.go†L9-L12】.

### Can I configure the delimiter used in TOON encoding?

Yes, the `EncodeOptions` struct accepts a `Delimiter` parameter (defaulting to `,`) that controls the character separating fields in the output. This option is passed through the `Compress` method in [`engine/compressors/toon.go`](https://github.com/JuliusBrussee/caveman/blob/main/engine/compressors/toon.go)【/cache/repos/github.com/JuliusBrussee/caveman/main/engine/compressors/toon.go†L12-L15】.

### Is TOON encoding reversible without data loss?

Yes, the `DecodeTOON` function in [`engine/toon.go`](https://github.com/JuliusBrussee/caveman/blob/main/engine/toon.go) validates the TOON format and reconstructs the exact original JSON structure. The round-trip process ensures that scalar types, array ordering, and key names remain identical to the pre-compressed state【/cache/repos/github.com/JuliusBrussee/caveman/main/engine/toon.go†L20-L30】.

### How do I force TOON compression for tool results in the proxy?

Set the environment variable `CAVE_ENGINE_TOON=best-of` before starting the proxy. This activates `forcedTOONCandidate` in [`proxy/providers/openai/content_compress.go`](https://github.com/JuliusBrussee/caveman/blob/main/proxy/providers/openai/content_compress.go), which routes eligible tool-result JSON through the TOON compressor even when the payload size falls below normal compression thresholds【/__modal/volumes/vo-cSqLfqnnIwYXEonuEJnnZa/repos/github.com/JuliusBrussee/caveman/main/proxy/providers/openai/content_compress.go†L429-L436】.