How to Debug Compression Issues with the Pipeline in Headroom: A Step-by-Step Guide

Debug Headroom compression issues by systematically tracing through detection, masking, compression, and CCR storage stages using debug logging, manual mask inspection, and metrics endpoints.

Headroom’s compression pipeline is a multi-stage system that processes content through detection, structural masking, entropy preservation, and compression before storing originals in CCR (Compress-Cache-Retrieve). When you encounter unexpected compression ratios, missing tokens, or retrieval failures, you can pinpoint the exact stage causing problems by examining the pipeline’s internal state.

Enable Debug Logging for Pipeline Visibility

Headroom emits OTLP traces and standard logs for every pipeline step. Set the environment variable HEADROOM_LOG_LEVEL=DEBUG before running your process to expose detailed stage information.

The logs categorize each pipeline phase under specific Python logger names:

• headroom.compression.detector – Content type detection and confidence scores
• headroom.compression.handlers – Handler selection and mask preservation ratios
• headroom.compression.masks – Entropy mask token counts
• headroom.compression.universal – Byte span compression details
• headroom.cache.compression_store – CCR storage operations

You can capture these using standard Python logging or tail the log directory:

export HEADROOM_LOG_LEVEL=DEBUG
export PYTHONVERBOSE=1
tail -f $HOME/.headroom/logs/*.log

Inspect Content Detection Results

The pipeline begins in headroom/compression/universal.py where the UniversalCompressor class detects content type using Magika ML or a fallback detector. Detection errors propagate through all subsequent stages, so verify the DetectionResult object first.

Use the return_detection=True flag to inspect detection metadata:

from headroom.compression import UniversalCompressor

compressor = UniversalCompressor()
result = compressor.compress(my_text, return_detection=True)
print(result.detection)

# Output: {'content_type': 'CODE', 'confidence': 0.97, 'language': 'python'}

If detection confidence is low or incorrect, check that the magika package is installed (pip install "headroom-ai[magika]") and that the input is not being truncated before detection. The truncation logic resides in _estimate_tokens within the same file.

Verify Structure Mask Generation

After detection, the pipeline routes content to a specific handler (e.g., JSONStructureHandler in headroom/compression/handlers/json_handler.py or the code handler in headroom/compression/handlers/code_handler.py). Each handler returns a StructureMask that marks structural tokens for preservation.

Examine the mask’s preservation_ratio to determine if too much or too little content is being protected:

from headroom.compression.handlers.json_handler import JSONStructureHandler

handler = JSONStructureHandler()
mask_result = handler.get_mask(json_text, list(json_text))
print(mask_result.mask.preservation_ratio)

# 0.41 indicates 41% of tokens preserved as structure

A very low preservation ratio suggests aggressive masking. Adjust handler thresholds like preserve_short_values or entropy_threshold in your handler configuration. If the mask is empty, verify that Tree-sitter is installed for code handlers (pip show tree_sitter).

Check Entropy Preservation Masking

When use_entropy_preservation is enabled (default in UniversalCompressorConfig), Headroom applies an additional entropy-based mask to preserve high-entropy tokens like UUIDs and hashes. This logic is implemented in headroom/compression/masks.py.

Test entropy preservation independently:

from headroom.compression.masks import compute_entropy_mask

tokens = list(content)
entropy_mask = compute_entropy_mask(tokens, threshold=0.85)
print(f"Preserved {entropy_mask.preservation_ratio * 100}% high-entropy tokens")

If this ratio is unexpectedly high, you are preventing effective compression. Lower the entropy_threshold or disable the feature:

from headroom.compression import UniversalCompressorConfig

config = UniversalCompressorConfig(use_entropy_preservation=False)
compressor = UniversalCompressor(config=config)

Isolate the Compression Function

The pipeline ultimately calls _compress_fn in headroom/compression/universal.py, which either uses KompressCompressor.compress (if installed) or falls back to _simple_compress. Isolate this stage by injecting a debug function:

def debug_compress(span):
    print(f"Compressing span of {len(span)} characters")
    return span  # Return unmodified to test masking logic

compressor = UniversalCompressor(compress_fn=debug_compress)
result = compressor.compress(large_input)

If span lengths appear correct but output remains large, the issue lies in the mask union operation (structure_mask.union(entropy_mask)) or in the handler’s get_mask implementation rather than the compression algorithm itself.

Validate CCR Storage Operations

When ccr_enabled=True, the compressor stores the original content via headroom/cache/compression_store.py. Check the CompressionResult.ccr_key field to verify storage succeeded:

from headroom.compression import compress, UniversalCompressorConfig

result = compress(
    content=large_text,
    config=UniversalCompressorConfig(ccr_enabled=True)
)

if result.ccr_key is None:
    print("CCR storage failed - check headroom.cache.compression_store logs")
else:
    print(f"Original stored under key: {result.ccr_key}")

Ensure the CCR store path (default ~/.headroom/session_stats.jsonl) is writable and the cache module is importable. Storage failures cause the pipeline to fall back to no-op keys, which can be observed in the debug logs under headroom.cache.compression_store.

Monitor Pipeline Metrics

Headroom exposes Prometheus-formatted metrics at /metrics and JSON debug dumps at /debug/compression when running the proxy server.

Query compression performance counters:

curl http://localhost:8787/metrics | grep headroom_compression
curl http://localhost:8787/debug/compression

Key metrics include headroom_compression_ratio_avg and headroom_compression_savings_usd, defined in the repository’s wiki/metrics.md. These reveal whether the pipeline is consistently under-performing across all requests or if the issue is request-specific.

Run Unit Tests for Targeted Debugging

Headroom includes comprehensive tests for each compressor stage. Run specific test suites to isolate failing components:

pytest tests/test_transforms_log_compressor.py -k compress
pytest tests/test_transforms_content_router.py -k detection

Failed tests provide stack traces pointing directly to problematic functions like _detect_format, _dedupe_similar, or _compress_with_mask.

Verify Dependency Versions

Missing or mismatched dependencies cause silent fallbacks that degrade compression:

• Magika: Required for ML-based content detection (pip show magika)
• Kompress: Optional ML compressor for advanced compression (pip show kompress)
• Tree-sitter: Required for code structure parsing (pip show tree_sitter)

Check your installed versions against the requirements in requirements.txt or Cargo.toml. If Kompress is missing, the pipeline silently falls back to simple truncation, which may explain poor compression ratios.

Summary

• Enable debug logging using HEADROOM_LOG_LEVEL=DEBUG to trace every pipeline stage through headroom.compression.universal.py.
• Verify detection using return_detection=True and ensure Magika is installed for accurate content-type identification.
• Inspect structure masks via handler classes in headroom/compression/handlers/ to confirm preservation ratios match expectations.
• Test entropy masking independently using compute_entropy_mask from headroom/compression/masks.py.
• Isolate compression by substituting compress_fn with a debug stub to verify span lengths before compression.
• Check CCR storage by examining the ccr_key field in CompressionResult and ensuring the cache store is reachable.
• Monitor metrics via /metrics and /debug/compression endpoints to identify systemic vs. isolated issues.
• Validate dependencies for Magika, Kompress, and Tree-sitter to prevent silent fallback behaviors.

Frequently Asked Questions

Why is my compression ratio unexpectedly low in Headroom?

Low compression ratios typically indicate that the structure mask or entropy mask is preserving too many tokens. Check the preservation_ratio in your handler’s StructureMask and the entropy mask settings in headroom/compression/masks.py. If the combined mask preserves more than 60-70% of tokens, adjust the entropy_threshold or handler-specific settings like preserve_short_values to allow more content through to the compression stage.

How do I disable specific pipeline stages for debugging?

Use UniversalCompressorConfig to toggle individual stages. Set use_magika=False to force fallback detection, use_entropy_preservation=False to skip entropy masking, or ccr_enabled=False to bypass CCR storage. You can also inject custom handlers via compressor.register_handler() to test specific content types without affecting the global configuration.

Where does Headroom store the original content during compression?

When CCR is enabled (default), Headroom stores originals in the Compress-Cache-Retrieve system implemented in headroom/cache/compression_store.py, typically under ~/.headroom/session_stats.jsonl. The CompressionResult object returns a ccr_key reference that points to this stored content. If ccr_key is None, storage failed and the pipeline operated in fallback mode.

What should I check if Headroom fails to detect my content type correctly?

First, verify that magika is installed (pip install "headroom-ai[magika]"). Then inspect the detection result using return_detection=True in the compress() call. If the input is truncated before detection, check the _estimate_tokens function in headroom/compression/universal.py. For edge cases, you can force a specific handler by calling compressor.register_handler() with your preferred handler class before compression.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}