# How caveman-compress Processes Files Containing Both Code and Prose

> Discover how caveman-compress efficiently processes mixed content files. It separates code and prose, minifies only executable snippets, and preserves documentation readability.

- Repository: [Julius Brussee/caveman](https://github.com/JuliusBrussee/caveman)
- Tags: how-to-guide
- Published: 2026-07-13

---

**caveman-compress** parses mixed-content files into separate prose and fenced code blocks, applies language-specific minification only to executable snippets, and preserves all narrative text to maintain documentation readability.

The `caveman-compress` utility in the JuliusBrussee/caveman repository handles the challenge of shrinking files that intermix documentation with embedded code snippets. When processing Markdown or similar formats, the tool distinguishes between human-readable prose and machine-executable blocks to ensure compression reduces file size without corrupting syntax or narrative flow.

## The Mixed-Content Processing Pipeline

According to the source code in [`skills/caveman-compress/scripts/compress.py`](https://github.com/JuliusBrussee/caveman/blob/main/skills/caveman-compress/scripts/compress.py), the tool implements a six-stage pipeline for files containing both code and prose.

### File Type Detection

The script first examines the file extension—such as `.md`, `.txt`, or `.py`—to determine the appropriate parser. For ambiguous mixed-content files, it defaults to the Markdown parser to properly handle fenced code blocks. This logic resides in lines 22-31 of [`compress.py`](https://github.com/JuliusBrussee/caveman/blob/main/compress.py).

### Block Segmentation

A state machine reads the file line-by-line to segment content into alternating blocks. Lines opening with fence markers (```` ``` ````) initiate a code block state, which continues until the matching closing fence. All lines outside these delimiters are classified as prose. This segmentation logic appears in lines 45-68 of `skills/caveman-compress/scripts/compress.py`.

### Selective Code Compression

Each extracted code block routes to a language-specific compressor based on the fence's language tag. The JavaScript minifier removes unnecessary whitespace, while the Python trimmer collapses consecutive blank lines and shortens long literals without rewriting logic. This selective compression occurs in lines 90-112.

### Prose Preservation

Prose sections undergo only global whitespace cleanup—removing leading or trailing spaces and duplicate empty lines. The tool deliberately avoids modifying narrative content to ensure documentation remains fully readable. This preservation logic is implemented in lines 115-128 of `compress.py`.

### Re-assembly and Safety Validation

The tool stitches compressed code blocks back into their original positions, maintaining original fence markers and indentation. Before writing output, the safety validator in `tests/test_compress_safety.py` verifies fence marker integrity and block count consistency. If the validator detects missing closing fences or structural corruption, the operation aborts with an error report. The re-assembly code occupies lines 133-147.

## Command-Line Usage Examples

Process a Markdown guide containing Python and Bash snippets:

```bash
caveman compress docs/usage-guide.md

```

The command detects fenced blocks, invokes appropriate compressors—such as `pyminify`, `bash-compress`, and `json-compact`—and overwrites the original file with the optimized version.

Preview changes without modifying files:

```bash
caveman compress --dry-run src/example.md

```

This outputs a diff-style summary showing which lines were trimmed within code blocks while prose sections remain identical.

## Programmatic API Access

Import the compression engine directly in Python scripts:

```python
from caveman_compress import compress_file

# Returns compressed text as a string

compressed = compress_file('README.md')
print(compressed[:200])  # Preview first 200 characters

```

This API bypasses the CLI interface and returns the processed content for further manipulation or inspection.

## Summary

- **caveman-compress** uses a state machine in [`skills/caveman-compress/scripts/compress.py`](https://github.com/JuliusBrussee/caveman/blob/main/skills/caveman-compress/scripts/compress.py) to separate fenced code blocks from prose in mixed-content files.
- Code blocks receive language-specific minification while prose remains verbatim except for whitespace cleanup.
- The pipeline includes safety validators in [`tests/test_compress_safety.py`](https://github.com/JuliusBrussee/caveman/blob/main/tests/test_compress_safety.py) that prevent fence corruption or structural damage.
- Both CLI and Python API interfaces support dry-run previews and batch processing.

## Frequently Asked Questions

### Does caveman-compress modify the logic inside code blocks?

No. The tool applies only safe transformations such as whitespace removal, blank line collapse, and literal shortening in lines 90-112 of [`compress.py`](https://github.com/JuliusBrussee/caveman/blob/main/compress.py). It never rewrites variable names, changes control flow, or alters syntax.

### What file formats does caveman-compress support for mixed content?

The tool primarily targets Markdown files but handles any text format with fenced code blocks. The file type detection logic in lines 22-31 of [`compress.py`](https://github.com/JuliusBrussee/caveman/blob/main/compress.py) examines extensions and defaults to Markdown parsing when encountering ambiguous mixed-content structures.

### How does caveman-compress prevent breaking documentation?

The safety validator in [`tests/test_compress_safety.py`](https://github.com/JuliusBrussee/caveman/blob/main/tests/test_compress_safety.py) verifies fence marker integrity and block count before finalizing output. If the re-assembly phase detects missing closing fences or mismatched block counts, the operation aborts with an error message.

### Can I use caveman-compress as a library in my Python project?

Yes. The `caveman_compress` module exposes `compress_file()` and related functions for programmatic access. Import the module to process strings or files directly without invoking the CLI, as shown in the programmatic integration examples.