How TOON Re-encoding Compresses Text Visually for Smaller Model Input

TOON (Tabular Object-Oriented Notation) re-encodes uniform JSON arrays as delimiter-separated tables, eliminating braces, quotation marks, and repeated keys to reduce token counts by 30–50% while maintaining lossless data fidelity.

The Caveman engine leverages TOON re-encoding to minimize the token footprint of structured data consumed by large language models. By transforming repetitive JSON objects into compact tabular representations, the system reduces visual clutter in the context window without sacrificing information integrity.

The Three-Stage TOON Compression Pipeline

The compression process operates through a strict validation and transformation workflow implemented in the core engine.

Detecting Uniform Tabular JSON

The compressor first validates that the incoming data matches a uniform tabular shape. In engine/compressors/toon.go, the function parseJSONTOON checks whether the top-level value is an array of objects where every element shares the identical set of keys. If the JSON is malformed or the structure is non-uniform, the compressor aborts immediately and returns the original payload unchanged【/cache/repos/github.com/JuliusBrussee/caveman/main/engine/compressors/toon.go†L9-L12】.

Encoding to Delimiter-Separated Format

Once validation passes, the encodeTOON function (invoked from Compress) constructs a dense tabular representation:

• Header row: Contains the shared object keys emitted once
• Data rows: Scalar values ordered to match the header sequence
• Compact syntax: Eliminates JSON punctuation ({, }, :, ") in favor of a single delimiter defined in EncodeOptions{Delimiter: ','}

The function returns a byte slice containing only the essential data separated by commas, and the compressor signals success with ok = true only when conversion completes【/cache/repos/github.com/JuliusBrussee/caveman/main/engine/compressors/toon.go†L12-L24】.

Proxy Integration and Forced Re-encoding

The proxy layer can override normal compression thresholds to force TOON encoding for tool results. When the environment variable CAVE_ENGINE_TOON=best-of is set, the function forcedTOONCandidate in proxy/providers/openai/content_compress.go checks if a candidate block contains tool JSON payload. If conditions match, the block routes through compressors.NewTOON().Compress regardless of size constraints【/__modal/volumes/vo-cSqLfqnnIwYXEonuEJnnZa/repos/github.com/JuliusBrussee/caveman/main/proxy/providers/openai/content_compress.go†L429-L436】.

Why TOON Visually Compresses Text

The token efficiency stems from three structural optimizations:

• Single Header Emission: Column names appear once at the top of the table rather than repeating in every object, drastically reducing redundancy for large arrays.
• Delimiter-Only Syntax: TOON strips all JSON structural characters—braces, quotation marks, and colons—replacing them with a single lightweight delimiter between fields.
• Lossless Round-Trip: The DecodeTOON implementation in engine/toon.go validates the tabular format and reconstructs the original JSON structure, ensuring the model receives full-fidelity data in a visually compact representation【/cache/repos/github.com/JuliusBrussee/caveman/main/engine/toon.go†L20-L30】.

Practical CLI Usage

Encode JSON to TOON:

caveman toon encode <<'EOF'
[
  {"id":1,"name":"Alice","age":30},
  {"id":2,"name":"Bob","age":25}
]
EOF

Output:

id,name,age
1,Alice,30
2,Bob,25

Force TOON in Proxy Sessions:

CAVE_ENGINE_TOON=best-of caveman wrap --toon my-agent

Decode TOON back to JSON:

caveman toon decode <<'EOF'
id,name,age
1,Alice,30
2,Bob,25
EOF

Summary

• parseJSONTOON in engine/compressors/toon.go validates uniform tabular structure before compression
• Re-encoding via encodeTOON eliminates JSON syntactic noise using delimiter-only syntax
• The proxy’s forcedTOONCandidate enables mandatory TOON conversion via CAVE_ENGINE_TOON=best-of
• TOON achieves 30–50% token reduction for large uniform datasets compared to standard JSON
• Lossless decoding through DecodeTOON in engine/toon.go preserves data integrity for downstream agents

Frequently Asked Questions

How does TOON handle non-uniform JSON arrays?

If parseJSONTOON detects that objects within the array contain different keys or inconsistent shapes, the compressor aborts and returns the original JSON unchanged. This validation occurs in engine/compressors/toon.go to ensure only compatible data enters the tabular format【/cache/repos/github.com/JuliusBrussee/caveman/main/engine/compressors/toon.go†L9-L12】.

Can I configure the delimiter used in TOON encoding?

Yes, the EncodeOptions struct accepts a Delimiter parameter (defaulting to ,) that controls the character separating fields in the output. This option is passed through the Compress method in engine/compressors/toon.go【/cache/repos/github.com/JuliusBrussee/caveman/main/engine/compressors/toon.go†L12-L15】.

Is TOON encoding reversible without data loss?

Yes, the DecodeTOON function in engine/toon.go validates the TOON format and reconstructs the exact original JSON structure. The round-trip process ensures that scalar types, array ordering, and key names remain identical to the pre-compressed state【/cache/repos/github.com/JuliusBrussee/caveman/main/engine/toon.go†L20-L30】.

How do I force TOON compression for tool results in the proxy?

Set the environment variable CAVE_ENGINE_TOON=best-of before starting the proxy. This activates forcedTOONCandidate in proxy/providers/openai/content_compress.go, which routes eligible tool-result JSON through the TOON compressor even when the payload size falls below normal compression thresholds【/__modal/volumes/vo-cSqLfqnnIwYXEonuEJnnZa/repos/github.com/JuliusBrussee/caveman/main/proxy/providers/openai/content_compress.go†L429-L436】.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}