# Headroom Compression Pipeline Configuration Options: A Complete Guide

> Explore Headroom compression pipeline configuration options in this comprehensive guide. Customize statistical crushing, cache alignment, and more with HeadroomConfig.

- Repository: [Tejas Chopra/headroom](https://github.com/chopratejas/headroom)
- Tags: how-to-guide
- Published: 2026-06-13

---

**Headroom exposes its compression pipeline through a hierarchy of Python dataclasses in [`headroom/config.py`](https://github.com/chopratejas/headroom/blob/main/headroom/config.py), allowing fine-grained control over statistical crushing, cache alignment, reversible storage, and tool-specific compressors via `HeadroomConfig` and its sub-configs.**

Headroom's compression pipeline is highly customizable through a structured configuration system. The SDK uses a hierarchy of Python dataclasses to expose tunable knobs for every stage of the pipeline, from pre-compression dynamic content detection to provider-specific cache optimization. All configuration options are defined in [`headroom/config.py`](https://github.com/chopratejas/headroom/blob/main/headroom/config.py) and respected throughout the Rust-backed pipeline stages.

## Core Configuration Architecture

The `HeadroomConfig` dataclass serves as the top-level entry point that aggregates sub-configurations controlling specific pipeline stages. According to the source code in [`headroom/config.py`](https://github.com/chopratejas/headroom/blob/main/headroom/config.py), this master config holds references to every modular component including `smart_crusher`, `cache_aligner`, `cache_optimizer`, `ccr`, and `prefix_freeze`.

The pipeline execution flow follows this sequence:

1. **Content Routing** – Detects tool result types (search, log, diff, code, HTML) and selects the appropriate compressor
2. **Tool-Specific Compression** – Runs the selected compressor using its dedicated configuration
3. **SmartCrusher** – Applies statistical pruning to remaining JSON arrays using `SmartCrusherConfig`
4. **Cache Alignment** – Normalizes dynamic content and freezes stable prefixes via `CacheAlignerConfig` and `PrefixFreezeConfig`
5. **CCR Application** – Optionally caches original payloads and inserts retrieval markers controlled by `CCRConfig`

## Statistical Compression Settings

The `SmartCrusherConfig` dataclass controls the core statistical compression of tool outputs. This default compressor analyzes item variance and uniqueness to determine optimal compression ratios.

Key configuration fields include:

- **`max_items_after_crush`** – Upper limit on items retained after analysis
- **`variance_threshold`** – Tightness of change-point detection (lower values = more aggressive compression)
- **`uniqueness_threshold`** – Minimum uniqueness score to preserve an item
- **`similarity_threshold`** – Clustering threshold for deduplication
- **`min_items_to_analyze`** and **`min_tokens_to_crush`** – Minimum thresholds before compression activates
- **`preserve_change_points`** – Boolean to retain significant statistical discontinuities
- **`anchor`** and **`relevance`** – Scoring configurations for item importance
- **`dedup_identical_items`** – Toggle for duplicate removal
- **`toin_confidence_threshold`** – Confidence level for statistical boundary detection

```python
from headroom.config import HeadroomConfig, SmartCrusherConfig

cfg = HeadroomConfig(
    smart_crusher=SmartCrusherConfig(
        max_items_after_crush=20,
        variance_threshold=1.5,
        relevance_threshold=0.3,
        dedup_identical_items=False,
    )
)

```

## Pre-Compression and Cache Optimization

Before final message assembly, the pipeline normalizes dynamic content and optimizes for provider-specific caching behaviors.

### CacheAlignerConfig

The `CacheAlignerConfig` in [`headroom/config.py`](https://github.com/chopratejas/headroom/blob/main/headroom/config.py) handles prefix stabilization and dynamic content detection:

- **`enabled`** – Master toggle for alignment processing
- **`use_dynamic_detector`** – Enables tiered detection for dynamic content
- **`detection_tiers`** – List of detection methods (e.g., `["regex", "ner"]`)
- **`entropy_threshold`** – Threshold for randomness detection in content
- **`date_patterns`** – Regex patterns for date normalization
- **`normalize_whitespace`** and **`collapse_blank_lines`** – Text normalization toggles
- **`dynamic_tail_separator`** – Boundary marker for dynamic suffixes

### CacheOptimizerConfig

Provider-specific optimizations for LLM caching (Anthropic, OpenAI) are controlled by `CacheOptimizerConfig`:

- **`auto_detect_provider`** – Automatic provider identification
- **`min_cacheable_tokens`** – Minimum token threshold for cache entry creation
- **`enable_semantic_cache`** – Toggle for semantic similarity caching
- **`semantic_cache_similarity`** – Similarity threshold for cache hits
- **`semantic_cache_max_entries`** and **`semantic_cache_ttl_seconds`** – Cache size and expiration controls

### PrefixFreezeConfig

The `PrefixFreezeConfig` preserves already-cached prefix tokens across turns to maintain provider-side prefix caching:

- **`enabled`** – Toggle for prefix freezing
- **`min_cached_tokens`** – Minimum tokens to preserve
- **`session_ttl_seconds`** – Session expiration for frozen prefixes
- **`force_compress_threshold`** – Compression ratio threshold for forced optimization

```python
from headroom.config import CacheAlignerConfig, CacheOptimizerConfig, PrefixFreezeConfig

cfg = HeadroomConfig(
    cache_aligner=CacheAlignerConfig(
        enabled=True,
        use_dynamic_detector=True,
        detection_tiers=["regex", "ner"],
        entropy_threshold=0.6,
    ),
    cache_optimizer=CacheOptimizerConfig(enable_semantic_cache=True),
    prefix_freeze=PrefixFreezeConfig(force_compress_threshold=0.4),
)

```

## Reversible Compression Options

The `CCRConfig` (Compress-Cache-Retrieve) enables reversible compression by storing original payloads and inserting retrieval markers, defined in [`headroom/config.py`](https://github.com/chopratejas/headroom/blob/main/headroom/config.py).

Configuration fields include:

- **`enabled`** – Master toggle for CCR functionality
- **`store_max_entries`** and **`store_ttl_seconds`** – Storage limits and expiration
- **`inject_retrieval_marker`** – Boolean to insert markers into compressed output
- **`marker_template`** – Format string for markers (supports `{original_count}`, `{compressed_count}`, `{hash}`, `{ttl_minutes}`)
- **`min_items_to_cache`** – Minimum array size before caching activates
- **`inject_tool`** and **`inject_system_instructions`** – Controls for additional context injection
- **`feedback_enabled`** – Toggle for compression feedback loops

```python
from headroom.config import CCRConfig

ccr_cfg = CCRConfig(
    enabled=True,
    min_items_to_cache=10,
    inject_retrieval_marker=True,
    marker_template=(
        "\n[{original_count} items compressed to {compressed_count}. "
        "Retrieve more: hash={hash}. Expires in {ttl_minutes}m.]"
    ),
)

```

## Tool-Specific Compressor Configurations

Headroom provides specialized compressors for different content types, each with dedicated configuration dataclasses.

### SearchCompressorConfig

Located in [`headroom/transforms/search_compressor.py`](https://github.com/chopratejas/headroom/blob/main/headroom/transforms/search_compressor.py), this configures grep/ripgrep result compression:

- **`max_matches_per_file`** and **`max_total_matches`** – Match limits
- **`max_files`** – Maximum files to include
- **`always_keep_first`** and **`always_keep_last`** – Boundary preservation
- **`context_keywords`** – Keywords for relevance boosting
- **`boost_errors`** – Boolean to prioritize error matches
- **`enable_ccr`** and **`min_matches_for_ccr`** – CCR integration settings

```python
from headroom.transforms.search_compressor import SearchCompressorConfig

search_cfg = SearchCompressorConfig(
    max_matches_per_file=3,
    max_total_matches=15,
    boost_errors=False,
    context_keywords=["auth", "token"],
)

```

### LogCompressorConfig

Defined in [`headroom/transforms/log_compressor.py`](https://github.com/chopratejas/headroom/blob/main/headroom/transforms/log_compressor.py) for build output and log compression:

- **`max_errors`** and **`max_warnings`** – Error/warning limits
- **`error_context_lines`** – Context lines around errors
- **`keep_first_error`** and **`keep_last_error`** – Boundary preservation
- **`max_stack_traces`** and **`stack_trace_max_lines`** – Stack trace limits
- **`dedupe_warnings`** – Warning deduplication toggle
- **`keep_summary_lines`** and **`max_total_lines`** – Line count controls
- **`enable_ccr`** and **`min_lines_for_ccr`** – CCR integration

### DiffCompressorConfig

For git diff compression in [`headroom/transforms/diff_compressor.py`](https://github.com/chopratejas/headroom/blob/main/headroom/transforms/diff_compressor.py):

- **`max_context_lines`** – Context lines per hunk
- **`max_hunks_per_file`** – Hunk limits
- **`max_files`** – File count limits
- **`always_keep_additions`** and **`always_keep_deletions`** – Change preservation
- **`enable_ccr`** and **`min_lines_for_ccr`** – CCR settings

### CodeCompressorConfig

AST-based source code compression in [`headroom/transforms/code_compressor.py`](https://github.com/chopratejas/headroom/blob/main/headroom/transforms/code_compressor.py):

- **`min_tokens_for_compression`** – Threshold for AST processing
- **`target_compression_rate`** – Desired compression ratio
- **`docstring_mode`** – Handling of documentation strings
- **`language_aware`** – Language-specific parsing toggle
- **`ignore_patterns`** – File patterns to exclude

### HTMLExtractorConfig

HTML extraction and compression in [`headroom/transforms/html_extractor.py`](https://github.com/chopratejas/headroom/blob/main/headroom/transforms/html_extractor.py):

- **`min_chars_for_block_compression`** – Block size threshold
- **`max_block_size`** – Maximum compressed block size
- **`strip_comments`** – HTML comment removal
- **`preserve_doctype`** – Doctype preservation toggle

## Adaptive Sizing and Profile Presets

The `CompressionProfile` dataclass and `PROFILE_PRESETS` dictionary provide per-tool bias controls that influence how aggressive the adaptive sizer behaves.

Available presets include:

- **`conservative`** – Preserves more content (high `min_k`, low bias)
- **`moderate`** – Balanced approach
- **`aggressive`** – Maximizes compression (low `min_k`, high bias)

The `DEFAULT_TOOL_PROFILES` mapping in [`headroom/config.py`](https://github.com/chopratejas/headroom/blob/main/headroom/config.py) assigns default profiles to specific tool names (e.g., `Grep`, `Bash`, `WebFetch`).

```python
from headroom.config import PROFILE_PRESETS, DEFAULT_TOOL_PROFILES

# Configure aggressive compression for search, conservative for bash

DEFAULT_TOOL_PROFILES.update({
    "Bash": PROFILE_PRESETS["conservative"],
    "Grep": PROFILE_PRESETS["aggressive"],
})

```

## Complete Configuration Example

Below is a comprehensive example combining multiple configuration layers:

```python
from headroom.config import (
    HeadroomConfig, SmartCrusherConfig, CacheAlignerConfig,
    CacheOptimizerConfig, CCRConfig, PrefixFreezeConfig
)
from headroom.transforms.log_compressor import LogCompressorConfig

cfg = HeadroomConfig(
    store_url="https://cache.example.com",
    default_mode="smart",
    smart_crusher=SmartCrusherConfig(
        max_items_after_crush=25,
        variance_threshold=1.2,
        dedup_identical_items=True,
    ),
    cache_aligner=CacheAlignerConfig(
        enabled=True,
        use_dynamic_detector=True,
        detection_tiers=["regex", "ner"],
    ),
    cache_optimizer=CacheOptimizerConfig(
        enable_semantic_cache=True,
        semantic_cache_ttl_seconds=300,
    ),
    ccr=CCRConfig(
        enabled=True,
        store_ttl_seconds=600,
        inject_retrieval_marker=True,
    ),
    prefix_freeze=PrefixFreezeConfig(
        enabled=True,
        min_cached_tokens=100,
    ),
    pipeline_extensions=[],
)

```

## Summary

- **HeadroomConfig** aggregates all sub-configurations and serves as the primary entry point in [`headroom/config.py`](https://github.com/chopratejas/headroom/blob/main/headroom/config.py)
- **SmartCrusherConfig** controls statistical compression with parameters for variance thresholds, item limits, and deduplication
- **CacheAlignerConfig** and **CacheOptimizerConfig** handle pre-compression normalization and provider-specific caching
- **CCRConfig** enables reversible compression with customizable retrieval markers and storage policies
- **PrefixFreezeConfig** preserves cached prefix tokens across conversation turns
- Tool-specific configs (**SearchCompressorConfig**, **LogCompressorConfig**, **DiffCompressorConfig**, **CodeCompressorConfig**, **HTMLExtractorConfig**) fine-tune compression for different content types
- **CompressionProfile** presets and **DEFAULT_TOOL_PROFILES** allow per-tool bias configuration

## Frequently Asked Questions

### How do I disable specific compression stages in Headroom?

Set the `enabled` field to `False` in the relevant configuration dataclass. For example, to disable the SmartCrusher while keeping other stages active, instantiate `SmartCrusherConfig(enabled=False)` and pass it to `HeadroomConfig(smart_crusher=...)`. Each sub-config respects its own enabled toggle without affecting other pipeline stages.

### What is the difference between SmartCrusher and tool-specific compressors?

**SmartCrusher** applies statistical analysis to JSON arrays after initial processing, using variance and uniqueness metrics to prune items. **Tool-specific compressors** (like `SearchCompressorConfig` or `LogCompressorConfig`) run first and apply domain-specific logic (e.g., keeping error lines in logs or context around matches in search results). The pipeline runs tool compressors first, then SmartCrusher processes the remaining structured data.

### How does CCRConfig preserve original data for retrieval?

When `CCRConfig.enabled` is `True`, the system stores the original uncompressed payload in a cache layer (defined in [`headroom/cache/compression_store.py`](https://github.com/chopratejas/headroom/blob/main/headroom/cache/compression_store.py)) and replaces the content with a retrieval marker. The `marker_template` field controls the placeholder text, which includes the cache hash and expiration. You can retrieve the original data using the hash embedded in the marker during decompression.

### Which configuration options affect provider-side caching performance?

**CacheOptimizerConfig** directly controls provider-specific caching with fields like `auto_detect_provider`, `min_cacheable_tokens`, and `enable_semantic_cache`. Additionally, **PrefixFreezeConfig** preserves cached prefix tokens across API calls through `min_cached_tokens` and `session_ttl_seconds`, which helps maintain cache hits on providers like Anthropic and OpenAI that support prefix caching.