How caveman-compress Reduces Memory File Size for Future Sessions

caveman-compress permanently rewrites large markdown memory files into compact representations, reducing token consumption by approximately 46% for every subsequent Claude session while preserving code blocks, URLs, and structural elements.

The caveman-compress skill in the JuliusBrussee/caveman repository transforms verbose project memory files into dense, token-efficient formats. By compressing files once before they are loaded into context, subsequent sessions process significantly fewer tokens without losing critical structural information.

The Compression Pipeline

Detecting Compressible Files

The process begins in skills/caveman-compress/scripts/detect.py, where detect.should_compress() (lines 8-28) evaluates whether a file contains natural language suitable for compression or code/configuration that should be skipped. The function examines file extensions, known filenames, and simple heuristics to make this determination.

Backup Creation and Front-Matter Handling

Before any LLM interaction, compress_file() in skills/caveman-compress/scripts/compress.py creates a platform-aware backup directory via backup_dir_for() (lines 53-56) and writes the original content to <name>.original.md outside the source tree (lines 69-90). If the file contains YAML front-matter, split_frontmatter() (lines 28-36) strips it before compression and reattaches it verbatim afterward, ensuring Claude never rewrites metadata sections.

LLM Compression and Validation

The call_claude() function (lines 22-34) sends a single prompt built by build_compress_prompt() to Claude, instructing it to compress natural language while preserving code fences, URLs, headings, and lists. After receiving the compressed text, validate() in skills/caveman-compress/scripts/validate.py (lines 36-85) performs structural checks on headings, code-block integrity, URL preservation, and inline-code counts. Errors trigger hard failures, while warnings allow the compressed file to remain.

Targeted Retries

If validation fails, build_fix_prompt() requests Claude to repair only the broken sections rather than recompressing the entire file. The system retries up to two times (MAX_RETRIES = 2 at line 16 of compress.py) before aborting. Once validation passes, the compressed text overwrites the original file (lines 13-15).

Token Savings and Permanent Reduction

The compression occurs exactly once per file. After this initial optimization, the memory file itself is permanently smaller, meaning every future Claude session loads approximately 46% fewer tokens (based on benchmarks in the README, lines 34-41). Only the initial compression and optional fix calls consume LLM tokens; all subsequent reads use the reduced local file.

Safety Mechanisms

The skill implements multiple safeguards in skills/caveman-compress/scripts/compress.py. The is_sensitive_path() function (lines 42-54) blocks files containing credentials before any network transfer. Backup verification ensures the original content is readable and matches the in-memory copy before proceeding (lines 99-106), aborting the operation if disk writes fail. Additionally, Claude is explicitly instructed not to wrap output in extra markdown fences, preventing unnecessary token inflation.

Usage Examples

Compress a memory file using the built-in skill:

/caveman-compress CLAUDE.md

Run the Python script directly:

python -m skills.caveman-compress.scripts.compress CLAUDE.md

Inspect the backup file:

cat ~/.local/share/caveman-compress/backups/caveman/CLAUDE.original.md

Summary

• caveman-compress permanently rewrites memory files, reducing token load by ~46% for future sessions
• The pipeline follows Detect → Backup → Compress → Validate → Retry stages
• Original files are preserved in ~/.local/share/caveman-compress/backups/ with the .original.md suffix
• Validation ensures structural integrity of code blocks, URLs, and headings
• Safety checks prevent compression of sensitive credential files

Frequently Asked Questions

How much does caveman-compress reduce file size?

According to the README.md (lines 34-41), caveman-compress achieves approximately 46% token reduction on average. The compression happens once, and the file remains permanently smaller for all future sessions.

Where are the original files stored?

Before compression, compress_file() creates a backup in a platform-specific directory (typically ~/.local/share/caveman-compress/backups/ on Linux) with the naming pattern <filename>.original.md. This backup remains untouched and available for manual editing.

What happens if compression fails?

If validation detects structural errors, the system enters a targeted fix loop using build_fix_prompt(), retrying up to two times (MAX_RETRIES = 2). If validation fails after all retries, the operation aborts without modifying the original file, leaving it intact for the next session.

Does caveman-compress modify code blocks?

No. The compression prompt explicitly instructs Claude to preserve code fences, URLs, headings, and bullet lists. The validate() function in validate.py (lines 36-85) specifically checks code-block integrity to ensure no modification occurs during compression.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}