How Caveman Compresses Output Tokens While Maintaining Technical Accuracy

Caveman reduces token counts by sending only natural-language prose to Claude while preserving all technical elements—code blocks, URLs, headings, and file paths—through a deterministic extraction, validation, and retry pipeline.

The caveman-compress skill in the JuliusBrussee/caveman repository solves the challenge of compressing markdown documents for LLM context windows without corrupting technical structure. By isolating compressible narrative text from immutable structural elements, the system ensures that compressed output tokens remain technically accurate and functionally identical to the original.

The 9-Step Compression Pipeline

The compression workflow operates as a guarded pipeline where each stage enforces strict preservation rules. Only natural-language sections are eligible for modification, while every technical artifact is protected by deterministic validation logic.

1. File Detection and Filtering

The process begins in skills/caveman-compress/scripts/detect.py where the should_compress() function evaluates file extensions, known code-file names, and content heuristics. This step skips binary code or configuration files and targets only natural-language documents that will benefit from prose compression.

2. Secret Protection

Before any content leaves the local machine, is_sensitive_path() inside skills/caveman-compress/scripts/compress.py enforces a hard-coded deny-list. Files matching patterns like .env, credentials*, or *.pem are immediately rejected, preventing accidental transmission of secrets to the Anthropic API.

3. Front-Matter Preservation

split_frontmatter() extracts YAML front-matter blocks delimited by --- before sending the document body to Claude. This metadata is stored verbatim and re-appended after compression, ensuring that tooling dependencies on front-matter values remain intact.

4. Deterministic Prompt Engineering

The build_compress_prompt() function constructs a tightly-controlled instruction set that explicitly defines the compression boundary. The prompt instructs Claude to shorten only natural-language prose while maintaining exact token identity for code fences, URLs, headings, file paths, and inline code snippets.

5. Claude API Integration

call_claude() in compress.py dynamically selects the invocation method: it prefers the Anthropic SDK when ANTHROPIC_API_KEY is present, otherwise falling back to the claude CLI subprocess. The implementation uses UTF-8 encoding with error-replacement to prevent Unicode crashes on Windows systems.

6. Atomic Backup Strategy

Before overwriting source files, backup_dir_for() calculates a platform-aware directory outside the repository tree and writes a *.original.md backup. The system reads this backup back to verify integrity, ensuring no corruption occurs during the write phase.

7. Structural Validation

The validate() function in skills/caveman-compress/scripts/validate.py executes a comprehensive fidelity check. It verifies heading count equality, exact code-block content match, URL set equality, path set equality, bullet-list stability, and inline-code preservation. Any deviation triggers an abort or repair sequence.

8. Automated Repair Loop

When validation reports errors, build_fix_prompt() generates a targeted correction instruction that identifies only the offending sections. The system permits up to two repair attempts before automatically restoring the original file from backup, guaranteeing no invalid compression persists.

9. Safe File Replacement

After successful validation, the compressed body (with preserved front-matter) overwrites the original document. The backup remains available for manual rollback, completing the atomic transaction.

Practical Usage Examples

Compress a Single File via CLI

# With the repository's virtual environment activated

python -m skills.caveman-compress.scripts.compress path/to/document.md

This command executes the full pipeline: detection, front-matter extraction, Claude API call, validation, and atomic replacement, leaving document.original.md as a backup.

Programmatic Library Integration

from pathlib import Path
from skills.caveman_compress.scripts.compress import compress_file

success = compress_file(Path("README.md"))
print("✅ Compression succeeded" if success else "❌ Compression aborted")

The compress_file() function returns True only after passing structural validation; otherwise it leaves the source file untouched.

Manual Validation for CI Pipelines

python -m skills.caveman_compress.scripts.validate original.md compressed.md

This invocation mirrors the internal validation step, printing error summaries for heading mismatches, code-block alterations, or URL modifications.

Core Implementation Files

• skills/caveman-compress/scripts/compress.py — Core orchestration logic including secret filtering, prompt building, API calling, backup management, and the fix-loop retry mechanism.
• skills/caveman-compress/scripts/detect.py — Content analysis and file-type detection to identify compressible natural-language documents.
• skills/caveman-compress/scripts/validate.py — Structural fidelity engine that enforces technical accuracy across headings, code blocks, URLs, and paths.
• skills/caveman-compress/scripts/cli.py — Lightweight argument parser that forwards CLI inputs to the compression engine.
• tests/test_compress_safety.py — Automated safety suite confirming that compression never drops or alters technical elements.

Summary

• Deterministic extraction separates natural-language prose from technical artifacts before API transmission.
• Multi-layer validation verifies heading counts, code-block equality, URL sets, and inline-code preservation before accepting any compression.
• Automatic retry logic uses targeted fix prompts to repair validation failures without manual intervention.
• Atomic backups ensure the original file can be restored if compression violates technical accuracy constraints.
• Secret protection prevents transmission of sensitive files via hard-coded deny-list patterns.

Frequently Asked Questions

How does Caveman prevent Claude from modifying code blocks during compression?

The build_compress_prompt() function in compress.py explicitly forbids modifications to fenced code blocks, inline code, and technical syntax. The subsequent validate() check in validate.py performs exact string comparisons on all code blocks; if any character differs, the compression is rejected or sent through the fix-loop for correction.

What happens if the compression process accidentally changes a file path or URL?

The validation layer treats URLs and file paths as immutable sets. validate() extracts all URLs and paths from both original and compressed versions using regex patterns, then demands set equality. If a path is altered, added, or removed, the validation fails and the original file is restored from the backup_dir_for() location.

Can Caveman compression run in CI/CD environments without manual oversight?

Yes. The CLI supports non-interactive execution via python -m skills.caveman_compress.scripts.compress, and the programmatic compress_file() function returns boolean status codes suitable for pipeline gates. Additionally, the validate.py module can be invoked independently to verify compressed artifacts against originals before deployment.

Does Caveman support offline compression without calling external APIs?

No. The current implementation in call_claude() requires either the Anthropic SDK with a valid ANTHROPIC_API_KEY or a local claude CLI installation. There is no offline compression engine; the system is specifically designed to leverage Claude's natural-language understanding while enforcing technical constraints through the validation layer.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}