# How Caveman Compresses Output Tokens While Maintaining Technical Accuracy

> Discover how Caveman compresses output tokens for Claude. Learn its unique pipeline for preserving technical accuracy in code, URLs, and more.

- Repository: [Julius Brussee/caveman](https://github.com/JuliusBrussee/caveman)
- Tags: how-to-guide
- Published: 2026-07-07

---

**Caveman reduces token counts by sending only natural-language prose to Claude while preserving all technical elements—code blocks, URLs, headings, and file paths—through a deterministic extraction, validation, and retry pipeline.**

The `caveman-compress` skill in the JuliusBrussee/caveman repository solves the challenge of compressing markdown documents for LLM context windows without corrupting technical structure. By isolating compressible narrative text from immutable structural elements, the system ensures that **compressed output tokens** remain **technically accurate** and functionally identical to the original.

## The 9-Step Compression Pipeline

The compression workflow operates as a guarded pipeline where each stage enforces strict preservation rules. Only natural-language sections are eligible for modification, while every technical artifact is protected by deterministic validation logic.

### 1. File Detection and Filtering

The process begins in [`skills/caveman-compress/scripts/detect.py`](https://github.com/JuliusBrussee/caveman/blob/main/skills/caveman-compress/scripts/detect.py) where the `should_compress()` function evaluates file extensions, known code-file names, and content heuristics. This step skips binary code or configuration files and targets only natural-language documents that will benefit from prose compression.

### 2. Secret Protection

Before any content leaves the local machine, `is_sensitive_path()` inside [`skills/caveman-compress/scripts/compress.py`](https://github.com/JuliusBrussee/caveman/blob/main/skills/caveman-compress/scripts/compress.py) enforces a hard-coded deny-list. Files matching patterns like `.env`, `credentials*`, or `*.pem` are immediately rejected, preventing accidental transmission of secrets to the Anthropic API.

### 3. Front-Matter Preservation

`split_frontmatter()` extracts YAML front-matter blocks delimited by `---` before sending the document body to Claude. This metadata is stored verbatim and re-appended after compression, ensuring that tooling dependencies on front-matter values remain intact.

### 4. Deterministic Prompt Engineering

The `build_compress_prompt()` function constructs a tightly-controlled instruction set that explicitly defines the compression boundary. The prompt instructs Claude to shorten only natural-language prose while maintaining exact token identity for code fences, URLs, headings, file paths, and inline code snippets.

### 5. Claude API Integration

`call_claude()` in [`compress.py`](https://github.com/JuliusBrussee/caveman/blob/main/compress.py) dynamically selects the invocation method: it prefers the Anthropic SDK when `ANTHROPIC_API_KEY` is present, otherwise falling back to the `claude` CLI subprocess. The implementation uses UTF-8 encoding with error-replacement to prevent Unicode crashes on Windows systems.

### 6. Atomic Backup Strategy

Before overwriting source files, `backup_dir_for()` calculates a platform-aware directory outside the repository tree and writes a `*.original.md` backup. The system reads this backup back to verify integrity, ensuring no corruption occurs during the write phase.

### 7. Structural Validation

The `validate()` function in [`skills/caveman-compress/scripts/validate.py`](https://github.com/JuliusBrussee/caveman/blob/main/skills/caveman-compress/scripts/validate.py) executes a comprehensive fidelity check. It verifies heading count equality, exact code-block content match, URL set equality, path set equality, bullet-list stability, and inline-code preservation. Any deviation triggers an abort or repair sequence.

### 8. Automated Repair Loop

When validation reports errors, `build_fix_prompt()` generates a targeted correction instruction that identifies only the offending sections. The system permits up to two repair attempts before automatically restoring the original file from backup, guaranteeing no invalid compression persists.

### 9. Safe File Replacement

After successful validation, the compressed body (with preserved front-matter) overwrites the original document. The backup remains available for manual rollback, completing the atomic transaction.

## Practical Usage Examples

### Compress a Single File via CLI

```bash

# With the repository's virtual environment activated

python -m skills.caveman-compress.scripts.compress path/to/document.md

```

This command executes the full pipeline: detection, front-matter extraction, Claude API call, validation, and atomic replacement, leaving [`document.original.md`](https://github.com/JuliusBrussee/caveman/blob/main/document.original.md) as a backup.

### Programmatic Library Integration

```python
from pathlib import Path
from skills.caveman_compress.scripts.compress import compress_file

success = compress_file(Path("README.md"))
print("✅ Compression succeeded" if success else "❌ Compression aborted")

```

The `compress_file()` function returns `True` only after passing structural validation; otherwise it leaves the source file untouched.

### Manual Validation for CI Pipelines

```bash
python -m skills.caveman_compress.scripts.validate original.md compressed.md

```

This invocation mirrors the internal validation step, printing error summaries for heading mismatches, code-block alterations, or URL modifications.

## Core Implementation Files

- **[`skills/caveman-compress/scripts/compress.py`](https://github.com/JuliusBrussee/caveman/blob/main/skills/caveman-compress/scripts/compress.py)** — Core orchestration logic including secret filtering, prompt building, API calling, backup management, and the fix-loop retry mechanism.
- **[`skills/caveman-compress/scripts/detect.py`](https://github.com/JuliusBrussee/caveman/blob/main/skills/caveman-compress/scripts/detect.py)** — Content analysis and file-type detection to identify compressible natural-language documents.
- **[`skills/caveman-compress/scripts/validate.py`](https://github.com/JuliusBrussee/caveman/blob/main/skills/caveman-compress/scripts/validate.py)** — Structural fidelity engine that enforces technical accuracy across headings, code blocks, URLs, and paths.
- **[`skills/caveman-compress/scripts/cli.py`](https://github.com/JuliusBrussee/caveman/blob/main/skills/caveman-compress/scripts/cli.py)** — Lightweight argument parser that forwards CLI inputs to the compression engine.
- **[`tests/test_compress_safety.py`](https://github.com/JuliusBrussee/caveman/blob/main/tests/test_compress_safety.py)** — Automated safety suite confirming that compression never drops or alters technical elements.

## Summary

- **Deterministic extraction** separates natural-language prose from technical artifacts before API transmission.
- **Multi-layer validation** verifies heading counts, code-block equality, URL sets, and inline-code preservation before accepting any compression.
- **Automatic retry logic** uses targeted fix prompts to repair validation failures without manual intervention.
- **Atomic backups** ensure the original file can be restored if compression violates technical accuracy constraints.
- **Secret protection** prevents transmission of sensitive files via hard-coded deny-list patterns.

## Frequently Asked Questions

### How does Caveman prevent Claude from modifying code blocks during compression?

The `build_compress_prompt()` function in [`compress.py`](https://github.com/JuliusBrussee/caveman/blob/main/compress.py) explicitly forbids modifications to fenced code blocks, inline code, and technical syntax. The subsequent `validate()` check in [`validate.py`](https://github.com/JuliusBrussee/caveman/blob/main/validate.py) performs exact string comparisons on all code blocks; if any character differs, the compression is rejected or sent through the fix-loop for correction.

### What happens if the compression process accidentally changes a file path or URL?

The validation layer treats URLs and file paths as immutable sets. `validate()` extracts all URLs and paths from both original and compressed versions using regex patterns, then demands set equality. If a path is altered, added, or removed, the validation fails and the original file is restored from the `backup_dir_for()` location.

### Can Caveman compression run in CI/CD environments without manual oversight?

Yes. The CLI supports non-interactive execution via `python -m skills.caveman_compress.scripts.compress`, and the programmatic `compress_file()` function returns boolean status codes suitable for pipeline gates. Additionally, the [`validate.py`](https://github.com/JuliusBrussee/caveman/blob/main/validate.py) module can be invoked independently to verify compressed artifacts against originals before deployment.

### Does Caveman support offline compression without calling external APIs?

No. The current implementation in `call_claude()` requires either the Anthropic SDK with a valid `ANTHROPIC_API_KEY` or a local `claude` CLI installation. There is no offline compression engine; the system is specifically designed to leverage Claude's natural-language understanding while enforcing technical constraints through the validation layer.