How Caveman's Compression Algorithm Achieves 65% Token Reduction While Preserving Technical Accuracy

Caveman's compression algorithm isolates technical tokens from natural language prose, compresses only the explanatory text through an LLM, and validates output through sentinel markers to achieve a 65% token reduction while maintaining 100% technical accuracy.

The JuliusBrussee/caveman repository solves a critical bottleneck in AI-assisted documentation workflows: reducing token costs without corrupting code integrity. Caveman's compression algorithm specifically targets verbose natural language surrounding technical content, ensuring that every command, URL, and code snippet remains untouched during optimization.

The Architecture Behind Caveman's Compression Algorithm

Detection and Eligibility Screening

The process begins in scripts/detect.py, where the algorithm determines whether a file is safe to compress. It evaluates file extensions and content patterns, marking files as compressible only when they contain pure prose (such as .md or .txt files). The detection module explicitly excludes source code files and scans for technical patterns including code fences (```), inline backticks, URLs, file system paths, and environment variable assignments matching /[A-Z0-9_]{2,}=.+/. When these patterns appear, the surrounding text is flagged as protected and excluded from compression.

Structured Parsing and Token Isolation

The core compression logic in src/mcp-servers/caveman-shrink/compress.js splits the input into three distinct streams:

• Preserve-only tokens: Code blocks, markdown fences, URLs, package manager commands (npm/pip), file paths, and environment variable assignments
• Natural-language sentences: Plain English prose eligible for shortening
• Sentinel markers: Unique placeholders (e.g., «N») injected before processing to verify token integrity

This separation ensures that technical elements never enter the LLM processing pipeline, eliminating the risk of corruption or modification.

LLM Processing with Safety Constraints

Only the natural-language stream is sent to the LLM (Claude or compatible models) with a strict prompt instructing the model to "rewrite the following description in the most concise way possible, preserving all code, URLs, and file-system references exactly as-is." The system targets a minimum 30% reduction for any file to qualify for updating.

Validation and Safety Guardrails

After LLM processing, scripts/validate.py and tests/test_compress_safety.py enforce four critical checks:

• Non-empty verification: Empty or whitespace-only outputs abort the operation, preserving the original file
• Idempotence checking: If compressed output matches input, no write occurs
• Token-count threshold: The algorithm requires a measurable reduction (≥30%) before overwriting
• Sentinel restoration: All sentinel markers must reappear unchanged in the output; missing markers trigger an abort, guaranteeing no technical snippets were stripped

Reassembly and Output Generation

Finally, the system stitches the preserved technical tokens back together with the shortened prose, maintaining exact whitespace, markdown formatting, and line numbers. The result contains identical code, paths, URLs, and error messages, with significantly reduced explanatory text.

Implementing Caveman's Compression Algorithm

Command Line Usage

Compress a single Markdown file from the command line:

caveman-compress docs/README.md

This command loads the detection pipeline from scripts/detect.py, processes the content through compress.js, and overwrites the file only if all safety validations pass.

Node.js Integration

Programmatically compress strings in Node.js applications:

const { compress } = require('./src/mcp-servers/caveman-shrink/compress.js');

const input = `

# My API

The endpoint https://api.example.com/v1/users returns a JSON payload.
Use the \`curl\` command to query it:
\`\`\`bash
curl -H "Authorization: Bearer $API_TOKEN" https://api.example.com/v1/users
\`\`\`
`;

const { compressed, before, after } = compress(input);
console.log(`Reduced ${before - after} tokens`);
console.log(compressed);

Python Helper

The test suite utilizes a Python wrapper for batch processing:

from scripts.compress import compress_file
from pathlib import Path

# Returns True only if the file was safely compressed

ok = compress_file(Path('docs/guide.original.md'))

Summary

• Caveman's compression algorithm achieves 65% token reduction by targeting only natural language while preserving all technical tokens
• The detection phase in scripts/detect.py identifies compressible content and isolates code blocks, URLs, and environment variables
• Structured parsing in compress.js creates three separate streams: preserve-only tokens, natural language, and sentinel markers
• Safety guardrails in scripts/validate.py and tests/test_compress_safety.py enforce non-empty output, idempotence, minimum 30% token reduction, and sentinel marker restoration
• Technical accuracy remains 100% because code, paths, and configuration values never enter the LLM processing pipeline

Frequently Asked Questions

How does Caveman's compression algorithm ensure code blocks remain unchanged?

The algorithm scans for markdown code fences (```) and inline backticks during the detection phase, routing these segments into a preserve-only stream that bypasses the LLM entirely. Sentinel markers verify that these segments remain intact during reassembly, ensuring zero modification of technical content.

What happens if the LLM accidentally removes a technical token?

The sentinel marker system acts as a fail-safe mechanism. If any marker disappears during compression, the validation layer in tests/test_compress_safety.py aborts the write operation and preserves the original file, preventing data loss and maintaining technical accuracy.

Can Caveman's compression algorithm handle mixed content with minimal code snippets?

Yes. The detect.py module evaluates content at the token level rather than relying solely on file extensions. Even in prose-heavy files, inline code, URLs, and file paths are identified and protected, allowing safe compression of technical documentation containing scattered code examples.

What token reduction threshold triggers the compression update?

The algorithm requires a minimum 30% token reduction before overwriting the original file. This threshold, enforced in the validation layer, ensures that only meaningful optimizations are committed, avoiding unnecessary file modifications for marginal gains.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}