How Does `/caveman-compress` Rewrite Memory Files: A Complete Technical Breakdown

The /caveman-compress skill rewrites memory files by creating secure out-of-tree backups, preserving YAML frontmatter, prompting Claude to compress natural language while keeping technical artefacts intact, validating the output against strict rules, and atomically writing the result back only if the content changed.

The caveman repository by JuliusBrussee provides a Claude-Code skill that transforms verbose memory files—such as CLAUDE.md, todos.md, or preferences.md—into compact "caveman" representations. Understanding how this rewrite pipeline works reveals a token-aware architecture that guarantees no data loss while significantly reducing context window consumption.

The Seven-Step Compression Pipeline

The rewrite process in skills/caveman-compress/scripts/compress.py follows a strict orchestration that separates safety checks from LLM operations.

1. Detect File Compressibility

Before any rewrite occurs, the should_compress(filepath) function (imported from detect.py) validates that the target contains natural language rather than code or configuration. According to the source at lines 45–47 of compress.py, if the file type is excluded—such as code files, configs, or backups—the skill exits immediately without touching the file system.

2. Guard Against Secrets Exposure

The is_sensitive_path(filepath) function applies a heuristic deny-list using regex patterns on basenames, known path components, and name tokens. As implemented in lines 93–104 of compress.py, this prevents the skill from processing files that might contain credentials, API keys, or other private data, refusing to proceed if the path matches sensitive patterns.

3. Create Out-of-Tree Backups

The backup_dir_for(filepath) function computes a platform-aware backup directory located outside the source repository. At lines 69–81 of compress.py, it writes the original file to <backup_dir>/<stem>.original.md, ensuring the skill’s auto-loader cannot re-ingest the backup as a live file. This guarantees users retain a human-readable original they can edit and recompress later.

4. Isolate YAML Frontmatter

The split_frontmatter(text) function (lines 28–40) extracts any leading --- YAML block before compression begins. Since LLMs tend to rewrite frontmatter, this preservation step ensures metadata remains untouched. The block is stored separately and reattached after the compression step completes.

5. Execute LLM Compression

The core rewrite happens in call_claude(prompt), which uses the Anthropic SDK when ANTHROPIC_API_KEY is set, or falls back to the claude --print CLI. The prompt is constructed by build_compress_prompt(body) (lines 71–87), which enforces strict constraints: the model must not alter code fences, URLs, headings, file paths, or other technical artefacts. Only the natural language content is compressed into the terse "caveman" style.

6. Validate and Fix Iteratively

The compressed output passes to validate(original, compressed) from validate.py. If validation fails—detecting missing URLs, heading mismatches, or other structural corruption—the build_fix_prompt function creates a targeted repair prompt (lines 90–101). This focused approach asks Claude to fix only the reported errors without re-compressing the entire file, making up to two retry attempts before aborting.

7. Atomic Write-Back

If validation succeeds and the compressed body differs from the original, the skill reassembles frontmatter + compressed_body and writes it back to the original filepath. As shown in lines 96–100 and 124–128, this write operation only occurs when content actually changes, leaving the *.original.md backup untouched for recovery purposes.

Safety Mechanisms and Token Efficiency

The architecture is explicitly token-aware: only the initial compression and possible targeted fixes consume LLM tokens, while detection, backup, and validation run as pure Python operations. By preserving structural elements—headings, code blocks, URLs, and file paths—downstream tooling such as caveman-stats (located in src/hooks/caveman-stats.js) can reliably parse these files for token savings reports without encountering unexpected format changes.

Usage Examples

Chat Invocation

/caveman-compress CLAUDE.md

This produces two files: CLAUDE.md containing the compressed caveman version (approximately 46% fewer tokens), and CLAUDE.original.md containing the untouched backup.

Command-Line Execution

python -m skills.caveman-compress.scripts.compress CLAUDE.md

The script prints progress messages ("Processing", "Compressing with Claude…", "Backup file already exists…") and aborts safely if the file is empty, already compressed, or flagged as sensitive.

Python API Integration

from pathlib import Path
from skills.caveman_compress.scripts.compress import compress_file

success = compress_file(Path("docs/preferences.md"))

The function returns True on successful rewrite, False if the file was skipped due to compressibility rules, and raises exceptions for fatal conditions such as missing files, oversized files, or sensitive-path refusals.

Summary

• /caveman-compress operates through a seven-step pipeline that prioritizes data safety over compression speed.
• Out-of-tree backups ensure originals persist outside the repository at paths computed by backup_dir_for().
• Frontmatter preservation via split_frontmatter() protects metadata from LLM drift.
• Validation with retry logic guarantees structural integrity using validate() and targeted build_fix_prompt() calls.
• Token efficiency is achieved by limiting LLM calls to compression and error-fixing only, with all safety checks running locally in Python.

Frequently Asked Questions

What happens if the compression produces corrupted output?

The validate() function compares the compressed output against the original, checking for preserved URLs, intact headings, and unmodified code blocks. If validation fails, the system generates a targeted fix prompt via build_fix_prompt() and attempts up to two correction cycles before aborting, ensuring the original file remains untouched.

How does the skill prevent accidental processing of sensitive files?

Before any rewrite, is_sensitive_path() applies regex-based heuristics to the basename and path components, checking against deny-lists of credential-bearing filenames. If the file appears to contain secrets, the skill exits immediately with a refusal, preventing potential exposure of API keys or passwords to the LLM.

Can I recover the original file after compression?

Yes. The backup_dir_for() function creates a *.original.md copy in a platform-specific directory outside your repository before any modification occurs. This backup persists after the rewrite, allowing you to manually restore the verbose version or recompress it later if needed.

What types of files will the skill refuse to compress?

The should_compress() function explicitly excludes code files, configuration files, backups, and any non-natural-language content. The skill only processes files identified as containing prose suitable for semantic compression, skipping binary files, JSON configs, and source code automatically.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}