# How Does `/caveman-compress` Rewrite Memory Files: A Complete Technical Breakdown

> Discover how caveman-compress rewrites memory files. Learn about secure backups, YAML preservation, Claude compression, validation, and atomic writes for technical content.

- Repository: [Julius Brussee/caveman](https://github.com/JuliusBrussee/caveman)
- Tags: deep-dive
- Published: 2026-07-11

---

**The `/caveman-compress` skill rewrites memory files by creating secure out-of-tree backups, preserving YAML frontmatter, prompting Claude to compress natural language while keeping technical artefacts intact, validating the output against strict rules, and atomically writing the result back only if the content changed.**

The `caveman` repository by JuliusBrussee provides a Claude-Code skill that transforms verbose memory files—such as [`CLAUDE.md`](https://github.com/JuliusBrussee/caveman/blob/main/CLAUDE.md), [`todos.md`](https://github.com/JuliusBrussee/caveman/blob/main/todos.md), or [`preferences.md`](https://github.com/JuliusBrussee/caveman/blob/main/preferences.md)—into compact "caveman" representations. Understanding how this rewrite pipeline works reveals a token-aware architecture that guarantees no data loss while significantly reducing context window consumption.

## The Seven-Step Compression Pipeline

The rewrite process in [`skills/caveman-compress/scripts/compress.py`](https://github.com/JuliusBrussee/caveman/blob/main/skills/caveman-compress/scripts/compress.py) follows a strict orchestration that separates safety checks from LLM operations.

### 1. Detect File Compressibility

Before any rewrite occurs, the `should_compress(filepath)` function (imported from [`detect.py`](https://github.com/JuliusBrussee/caveman/blob/main/detect.py)) validates that the target contains natural language rather than code or configuration. According to the source at lines 45–47 of [`compress.py`](https://github.com/JuliusBrussee/caveman/blob/main/compress.py), if the file type is excluded—such as code files, configs, or backups—the skill exits immediately without touching the file system.

### 2. Guard Against Secrets Exposure

The `is_sensitive_path(filepath)` function applies a heuristic deny-list using regex patterns on basenames, known path components, and name tokens. As implemented in lines 93–104 of [`compress.py`](https://github.com/JuliusBrussee/caveman/blob/main/compress.py), this prevents the skill from processing files that might contain credentials, API keys, or other private data, refusing to proceed if the path matches sensitive patterns.

### 3. Create Out-of-Tree Backups

The `backup_dir_for(filepath)` function computes a platform-aware backup directory located outside the source repository. At lines 69–81 of [`compress.py`](https://github.com/JuliusBrussee/caveman/blob/main/compress.py), it writes the original file to `<backup_dir>/<stem>.original.md`, ensuring the skill’s auto-loader cannot re-ingest the backup as a live file. This guarantees users retain a human-readable original they can edit and recompress later.

### 4. Isolate YAML Frontmatter

The `split_frontmatter(text)` function (lines 28–40) extracts any leading `---` YAML block before compression begins. Since LLMs tend to rewrite frontmatter, this preservation step ensures metadata remains untouched. The block is stored separately and reattached after the compression step completes.

### 5. Execute LLM Compression

The core rewrite happens in `call_claude(prompt)`, which uses the Anthropic SDK when `ANTHROPIC_API_KEY` is set, or falls back to the `claude --print` CLI. The prompt is constructed by `build_compress_prompt(body)` (lines 71–87), which enforces strict constraints: the model must not alter code fences, URLs, headings, file paths, or other technical artefacts. Only the natural language content is compressed into the terse "caveman" style.

### 6. Validate and Fix Iteratively

The compressed output passes to `validate(original, compressed)` from [`validate.py`](https://github.com/JuliusBrussee/caveman/blob/main/validate.py). If validation fails—detecting missing URLs, heading mismatches, or other structural corruption—the `build_fix_prompt` function creates a targeted repair prompt (lines 90–101). This focused approach asks Claude to fix only the reported errors without re-compressing the entire file, making up to two retry attempts before aborting.

### 7. Atomic Write-Back

If validation succeeds and the compressed body differs from the original, the skill reassembles `frontmatter + compressed_body` and writes it back to the original `filepath`. As shown in lines 96–100 and 124–128, this write operation only occurs when content actually changes, leaving the `*.original.md` backup untouched for recovery purposes.

## Safety Mechanisms and Token Efficiency

The architecture is explicitly **token-aware**: only the initial compression and possible targeted fixes consume LLM tokens, while detection, backup, and validation run as pure Python operations. By preserving structural elements—headings, code blocks, URLs, and file paths—downstream tooling such as `caveman-stats` (located in [`src/hooks/caveman-stats.js`](https://github.com/JuliusBrussee/caveman/blob/main/src/hooks/caveman-stats.js)) can reliably parse these files for token savings reports without encountering unexpected format changes.

## Usage Examples

### Chat Invocation

```markdown
/caveman-compress CLAUDE.md

```

This produces two files: [`CLAUDE.md`](https://github.com/JuliusBrussee/caveman/blob/main/CLAUDE.md) containing the compressed caveman version (approximately 46% fewer tokens), and [`CLAUDE.original.md`](https://github.com/JuliusBrussee/caveman/blob/main/CLAUDE.original.md) containing the untouched backup.

### Command-Line Execution

```bash
python -m skills.caveman-compress.scripts.compress CLAUDE.md

```

The script prints progress messages ("Processing", "Compressing with Claude…", "Backup file already exists…") and aborts safely if the file is empty, already compressed, or flagged as sensitive.

### Python API Integration

```python
from pathlib import Path
from skills.caveman_compress.scripts.compress import compress_file

success = compress_file(Path("docs/preferences.md"))

```

The function returns `True` on successful rewrite, `False` if the file was skipped due to compressibility rules, and raises exceptions for fatal conditions such as missing files, oversized files, or sensitive-path refusals.

## Summary

- **`/caveman-compress`** operates through a seven-step pipeline that prioritizes data safety over compression speed.
- **Out-of-tree backups** ensure originals persist outside the repository at paths computed by `backup_dir_for()`.
- **Frontmatter preservation** via `split_frontmatter()` protects metadata from LLM drift.
- **Validation with retry logic** guarantees structural integrity using `validate()` and targeted `build_fix_prompt()` calls.
- **Token efficiency** is achieved by limiting LLM calls to compression and error-fixing only, with all safety checks running locally in Python.

## Frequently Asked Questions

### What happens if the compression produces corrupted output?

The `validate()` function compares the compressed output against the original, checking for preserved URLs, intact headings, and unmodified code blocks. If validation fails, the system generates a targeted fix prompt via `build_fix_prompt()` and attempts up to two correction cycles before aborting, ensuring the original file remains untouched.

### How does the skill prevent accidental processing of sensitive files?

Before any rewrite, `is_sensitive_path()` applies regex-based heuristics to the basename and path components, checking against deny-lists of credential-bearing filenames. If the file appears to contain secrets, the skill exits immediately with a refusal, preventing potential exposure of API keys or passwords to the LLM.

### Can I recover the original file after compression?

Yes. The `backup_dir_for()` function creates a `*.original.md` copy in a platform-specific directory outside your repository before any modification occurs. This backup persists after the rewrite, allowing you to manually restore the verbose version or recompress it later if needed.

### What types of files will the skill refuse to compress?

The `should_compress()` function explicitly excludes code files, configuration files, backups, and any non-natural-language content. The skill only processes files identified as containing prose suitable for semantic compression, skipping binary files, JSON configs, and source code automatically.