Headroom Compression Pipeline Configuration Options: A Complete Guide

Headroom exposes its compression pipeline through a hierarchy of Python dataclasses in headroom/config.py, allowing fine-grained control over statistical crushing, cache alignment, reversible storage, and tool-specific compressors via HeadroomConfig and its sub-configs.

Headroom's compression pipeline is highly customizable through a structured configuration system. The SDK uses a hierarchy of Python dataclasses to expose tunable knobs for every stage of the pipeline, from pre-compression dynamic content detection to provider-specific cache optimization. All configuration options are defined in headroom/config.py and respected throughout the Rust-backed pipeline stages.

Core Configuration Architecture

The HeadroomConfig dataclass serves as the top-level entry point that aggregates sub-configurations controlling specific pipeline stages. According to the source code in headroom/config.py, this master config holds references to every modular component including smart_crusher, cache_aligner, cache_optimizer, ccr, and prefix_freeze.

The pipeline execution flow follows this sequence:

1. Content Routing – Detects tool result types (search, log, diff, code, HTML) and selects the appropriate compressor
2. Tool-Specific Compression – Runs the selected compressor using its dedicated configuration
3. SmartCrusher – Applies statistical pruning to remaining JSON arrays using SmartCrusherConfig
4. Cache Alignment – Normalizes dynamic content and freezes stable prefixes via CacheAlignerConfig and PrefixFreezeConfig
5. CCR Application – Optionally caches original payloads and inserts retrieval markers controlled by CCRConfig

Statistical Compression Settings

The SmartCrusherConfig dataclass controls the core statistical compression of tool outputs. This default compressor analyzes item variance and uniqueness to determine optimal compression ratios.

Key configuration fields include:

• max_items_after_crush – Upper limit on items retained after analysis
• variance_threshold – Tightness of change-point detection (lower values = more aggressive compression)
• uniqueness_threshold – Minimum uniqueness score to preserve an item
• similarity_threshold – Clustering threshold for deduplication
• min_items_to_analyze and min_tokens_to_crush – Minimum thresholds before compression activates
• preserve_change_points – Boolean to retain significant statistical discontinuities
• anchor and relevance – Scoring configurations for item importance
• dedup_identical_items – Toggle for duplicate removal
• toin_confidence_threshold – Confidence level for statistical boundary detection

from headroom.config import HeadroomConfig, SmartCrusherConfig

cfg = HeadroomConfig(
    smart_crusher=SmartCrusherConfig(
        max_items_after_crush=20,
        variance_threshold=1.5,
        relevance_threshold=0.3,
        dedup_identical_items=False,
    )
)

Pre-Compression and Cache Optimization

Before final message assembly, the pipeline normalizes dynamic content and optimizes for provider-specific caching behaviors.

CacheAlignerConfig

The CacheAlignerConfig in headroom/config.py handles prefix stabilization and dynamic content detection:

• enabled – Master toggle for alignment processing
• use_dynamic_detector – Enables tiered detection for dynamic content
• detection_tiers – List of detection methods (e.g., ["regex", "ner"])
• entropy_threshold – Threshold for randomness detection in content
• date_patterns – Regex patterns for date normalization
• normalize_whitespace and collapse_blank_lines – Text normalization toggles
• dynamic_tail_separator – Boundary marker for dynamic suffixes

CacheOptimizerConfig

Provider-specific optimizations for LLM caching (Anthropic, OpenAI) are controlled by CacheOptimizerConfig:

• auto_detect_provider – Automatic provider identification
• min_cacheable_tokens – Minimum token threshold for cache entry creation
• enable_semantic_cache – Toggle for semantic similarity caching
• semantic_cache_similarity – Similarity threshold for cache hits
• semantic_cache_max_entries and semantic_cache_ttl_seconds – Cache size and expiration controls

PrefixFreezeConfig

The PrefixFreezeConfig preserves already-cached prefix tokens across turns to maintain provider-side prefix caching:

• enabled – Toggle for prefix freezing
• min_cached_tokens – Minimum tokens to preserve
• session_ttl_seconds – Session expiration for frozen prefixes
• force_compress_threshold – Compression ratio threshold for forced optimization

from headroom.config import CacheAlignerConfig, CacheOptimizerConfig, PrefixFreezeConfig

cfg = HeadroomConfig(
    cache_aligner=CacheAlignerConfig(
        enabled=True,
        use_dynamic_detector=True,
        detection_tiers=["regex", "ner"],
        entropy_threshold=0.6,
    ),
    cache_optimizer=CacheOptimizerConfig(enable_semantic_cache=True),
    prefix_freeze=PrefixFreezeConfig(force_compress_threshold=0.4),
)

Reversible Compression Options

The CCRConfig (Compress-Cache-Retrieve) enables reversible compression by storing original payloads and inserting retrieval markers, defined in headroom/config.py.

Configuration fields include:

• enabled – Master toggle for CCR functionality
• store_max_entries and store_ttl_seconds – Storage limits and expiration
• inject_retrieval_marker – Boolean to insert markers into compressed output
• marker_template – Format string for markers (supports {original_count}, {compressed_count}, {hash}, {ttl_minutes})
• min_items_to_cache – Minimum array size before caching activates
• inject_tool and inject_system_instructions – Controls for additional context injection
• feedback_enabled – Toggle for compression feedback loops

from headroom.config import CCRConfig

ccr_cfg = CCRConfig(
    enabled=True,
    min_items_to_cache=10,
    inject_retrieval_marker=True,
    marker_template=(
        "\n[{original_count} items compressed to {compressed_count}. "
        "Retrieve more: hash={hash}. Expires in {ttl_minutes}m.]"
    ),
)

Tool-Specific Compressor Configurations

Headroom provides specialized compressors for different content types, each with dedicated configuration dataclasses.

SearchCompressorConfig

Located in headroom/transforms/search_compressor.py, this configures grep/ripgrep result compression:

• max_matches_per_file and max_total_matches – Match limits
• max_files – Maximum files to include
• always_keep_first and always_keep_last – Boundary preservation
• context_keywords – Keywords for relevance boosting
• boost_errors – Boolean to prioritize error matches
• enable_ccr and min_matches_for_ccr – CCR integration settings

from headroom.transforms.search_compressor import SearchCompressorConfig

search_cfg = SearchCompressorConfig(
    max_matches_per_file=3,
    max_total_matches=15,
    boost_errors=False,
    context_keywords=["auth", "token"],
)

LogCompressorConfig

Defined in headroom/transforms/log_compressor.py for build output and log compression:

• max_errors and max_warnings – Error/warning limits
• error_context_lines – Context lines around errors
• keep_first_error and keep_last_error – Boundary preservation
• max_stack_traces and stack_trace_max_lines – Stack trace limits
• dedupe_warnings – Warning deduplication toggle
• keep_summary_lines and max_total_lines – Line count controls
• enable_ccr and min_lines_for_ccr – CCR integration

DiffCompressorConfig

For git diff compression in headroom/transforms/diff_compressor.py:

• max_context_lines – Context lines per hunk
• max_hunks_per_file – Hunk limits
• max_files – File count limits
• always_keep_additions and always_keep_deletions – Change preservation
• enable_ccr and min_lines_for_ccr – CCR settings

CodeCompressorConfig

AST-based source code compression in headroom/transforms/code_compressor.py:

• min_tokens_for_compression – Threshold for AST processing
• target_compression_rate – Desired compression ratio
• docstring_mode – Handling of documentation strings
• language_aware – Language-specific parsing toggle
• ignore_patterns – File patterns to exclude

HTMLExtractorConfig

HTML extraction and compression in headroom/transforms/html_extractor.py:

• min_chars_for_block_compression – Block size threshold
• max_block_size – Maximum compressed block size
• strip_comments – HTML comment removal
• preserve_doctype – Doctype preservation toggle

Adaptive Sizing and Profile Presets

The CompressionProfile dataclass and PROFILE_PRESETS dictionary provide per-tool bias controls that influence how aggressive the adaptive sizer behaves.

Available presets include:

• conservative – Preserves more content (high min_k, low bias)
• moderate – Balanced approach
• aggressive – Maximizes compression (low min_k, high bias)

The DEFAULT_TOOL_PROFILES mapping in headroom/config.py assigns default profiles to specific tool names (e.g., Grep, Bash, WebFetch).

from headroom.config import PROFILE_PRESETS, DEFAULT_TOOL_PROFILES

# Configure aggressive compression for search, conservative for bash

DEFAULT_TOOL_PROFILES.update({
    "Bash": PROFILE_PRESETS["conservative"],
    "Grep": PROFILE_PRESETS["aggressive"],
})

Complete Configuration Example

Below is a comprehensive example combining multiple configuration layers:

from headroom.config import (
    HeadroomConfig, SmartCrusherConfig, CacheAlignerConfig,
    CacheOptimizerConfig, CCRConfig, PrefixFreezeConfig
)
from headroom.transforms.log_compressor import LogCompressorConfig

cfg = HeadroomConfig(
    store_url="https://cache.example.com",
    default_mode="smart",
    smart_crusher=SmartCrusherConfig(
        max_items_after_crush=25,
        variance_threshold=1.2,
        dedup_identical_items=True,
    ),
    cache_aligner=CacheAlignerConfig(
        enabled=True,
        use_dynamic_detector=True,
        detection_tiers=["regex", "ner"],
    ),
    cache_optimizer=CacheOptimizerConfig(
        enable_semantic_cache=True,
        semantic_cache_ttl_seconds=300,
    ),
    ccr=CCRConfig(
        enabled=True,
        store_ttl_seconds=600,
        inject_retrieval_marker=True,
    ),
    prefix_freeze=PrefixFreezeConfig(
        enabled=True,
        min_cached_tokens=100,
    ),
    pipeline_extensions=[],
)

Summary

• HeadroomConfig aggregates all sub-configurations and serves as the primary entry point in headroom/config.py
• SmartCrusherConfig controls statistical compression with parameters for variance thresholds, item limits, and deduplication
• CacheAlignerConfig and CacheOptimizerConfig handle pre-compression normalization and provider-specific caching
• CCRConfig enables reversible compression with customizable retrieval markers and storage policies
• PrefixFreezeConfig preserves cached prefix tokens across conversation turns
• Tool-specific configs (SearchCompressorConfig, LogCompressorConfig, DiffCompressorConfig, CodeCompressorConfig, HTMLExtractorConfig) fine-tune compression for different content types
• CompressionProfile presets and DEFAULT_TOOL_PROFILES allow per-tool bias configuration

Frequently Asked Questions

How do I disable specific compression stages in Headroom?

Set the enabled field to False in the relevant configuration dataclass. For example, to disable the SmartCrusher while keeping other stages active, instantiate SmartCrusherConfig(enabled=False) and pass it to HeadroomConfig(smart_crusher=...). Each sub-config respects its own enabled toggle without affecting other pipeline stages.

What is the difference between SmartCrusher and tool-specific compressors?

SmartCrusher applies statistical analysis to JSON arrays after initial processing, using variance and uniqueness metrics to prune items. Tool-specific compressors (like SearchCompressorConfig or LogCompressorConfig) run first and apply domain-specific logic (e.g., keeping error lines in logs or context around matches in search results). The pipeline runs tool compressors first, then SmartCrusher processes the remaining structured data.

How does CCRConfig preserve original data for retrieval?

When CCRConfig.enabled is True, the system stores the original uncompressed payload in a cache layer (defined in headroom/cache/compression_store.py) and replaces the content with a retrieval marker. The marker_template field controls the placeholder text, which includes the cache hash and expiration. You can retrieve the original data using the hash embedded in the marker during decompression.

Which configuration options affect provider-side caching performance?

CacheOptimizerConfig directly controls provider-specific caching with fields like auto_detect_provider, min_cacheable_tokens, and enable_semantic_cache. Additionally, PrefixFreezeConfig preserves cached prefix tokens across API calls through min_cached_tokens and session_ttl_seconds, which helps maintain cache hits on providers like Anthropic and OpenAI that support prefix caching.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}