How caveman-compress Preserves Code Blocks and URLs When Compressing Markdown

caveman-compress preserves code blocks and URLs by isolating YAML front-matter, enforcing strict structural rules via LLM prompting, stripping outer markdown fences, and running post-compression validation with automatic repair.

The caveman-compress tool in the JuliusBrussee/caveman repository compresses natural-language prose in Markdown files while ensuring that every code snippet, inline backtick, URL, heading, and file path remains untouched. This article examines the five-step pipeline implemented in skills/caveman-compress/scripts/compress.py that guarantees structural integrity during compression.

Step 1: Isolate YAML Front-Matter with split_frontmatter()

Before any compression occurs, caveman-compress separates the document's YAML front-matter from its body to prevent accidental modification of metadata.

The split_frontmatter() function (lines 28–40 in compress.py) detects content before the first --- delimiter, removes it from the compression input, and stores it for later reassembly. This ensures that Jekyll-style headers, TOML blocks, or other front-matter configurations remain exactly as they were in the original file.

Step 2: Enforce Structural Rules via build_compress_prompt()

The core protection mechanism lies in the build_compress_prompt() function (lines 73–80 in compress.py). This function constructs a "preserve-structure" prompt that explicitly instructs Claude not to modify:

• Code fences (triple backticks)
• Inline backticks
• URLs
• Headings
• File paths

By sending only the document body (without front-matter) alongside these strict prohibitions, the LLM returns compressed natural-language prose while leaving all technical elements intact.

Step 3: Remove LLM Wrapper Fencing with strip_llm_wrapper()

Large language models occasionally wrap their entire response in markdown code fences for emphasis. The strip_llm_wrapper() function (lines 17–22 and 106–112 in compress.py) uses the OUTER_FENCE_REGEX pattern to detect and remove these outer wrappers.

This step is critical because it prevents nested fencing that could corrupt the document's actual code blocks. After stripping the outer fence, inner code fences remain exactly as they were in the original text.

Step 4: Validate and Repair with validate() and build_fix_prompt()

After receiving the compressed text, caveman-compress runs a validation loop to catch any violations. The validate() function (lines 90–101 in compress.py) checks for specific errors including:

• "Code block mismatch": When opening and closing fence counts differ
• "Missing URL": When URLs detected in the original are absent in the output
• "Heading alteration": When heading structures change unexpectedly

If validate() detects errors, the system calls build_fix_prompt() (lines 190–205 in compress.py) to generate a repair request. This second prompt asks Claude to restore the exact original fragments while maintaining the compression. The process retries until validation passes or reaches the maximum attempt limit.

Step 5: Reassemble the Final Document

Once validation passes, the tool writes the final output. Lines 96–98 and 129–132 in compress.py concatenate the preserved front-matter with the validated compressed body using filepath.write_text(frontmatter + compressed_body).

This guarantee means that a code block like:

def process_data():
    return "compressed"

Or a URL like https://example.com/api will appear identically in the output file, even as surrounding paragraphs shrink significantly.

Using caveman-compress: CLI and Programmatic Examples

You can invoke the compression engine from the command line or integrate it directly into Python applications.

Compress a single file in place:

# From the repository root

$ python skills/caveman-compress/scripts/compress.py docs/guide.md
Processing: /path/to/docs/guide.md
Compressing with Claude...
Validation passed

Use the tool programmatically:

from pathlib import Path
from skills.caveman_compress.scripts.compress import compress_file

# Returns True if compression succeeded, False otherwise

result = compress_file(Path("docs/guide.md"))
print("Compressed?", result)

The compress_file() function orchestrates the entire pipeline, returning a boolean status that indicates whether the file was successfully compressed and validated.

Key Source Files and Their Roles

File	Purpose
skills/caveman-compress/scripts/compress.py	Main orchestration containing split_frontmatter(), build_compress_prompt(), strip_llm_wrapper(), validate(), and build_fix_prompt()
skills/caveman-compress/scripts/detect.py	Implements should_compress() to determine if a file contains enough natural language to warrant compression
skills/caveman-compress/scripts/validate.py	Parses compressed output and emits specific error codes like "code block mismatch" or "missing URL" for the repair loop
skills/caveman-compress/SKILL.md	Documentation covering usage patterns and architectural decisions

Summary

• Front-matter isolation via split_frontmatter() protects metadata before compression begins.
• Strict LLM prompting in build_compress_prompt() explicitly forbids modifying code blocks, URLs, and structural elements.
• Wrapper removal using strip_llm_wrapper() and OUTER_FENCE_REGEX prevents fence nesting artifacts.
• Validation and repair through validate() and build_fix_prompt() create a self-healing system that restores exact original fragments when errors occur.
• Verbatim reassembly ensures the final file contains unchanged technical elements alongside compressed prose.

Frequently Asked Questions

What happens if Claude accidentally modifies a code block?

If validate() detects a code block mismatch or alteration, caveman-compress triggers a repair loop using build_fix_prompt(). This second prompt instructs Claude to restore the exact original code fence while maintaining the compression of surrounding text. The system retries until validation passes or exhausts the configured attempt limit.

How does caveman-compress handle YAML front-matter?

The split_frontmatter() function removes anything before the first --- delimiter before sending content to the LLM. This front-matter is stored separately and concatenated back to the compressed body during the final write operation, ensuring that metadata fields remain completely untouched.

Can I use caveman-compress programmatically in Python?

Yes. Import compress_file() from skills.caveman_compress.scripts.compress and pass a pathlib.Path object to the function. It returns a boolean indicating success and handles the entire pipeline including validation and repair automatically.

What types of validation errors trigger the repair loop?

The validate() function specifically checks for code block mismatches (uneven fence counts), missing URLs (when original URLs disappear from output), and altered headings. Any of these violations feed into build_fix_prompt() to generate targeted repair instructions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}