# How to Debug Compression Issues with the Pipeline in Headroom: A Step-by-Step Guide

> Resolve Headroom compression issues with our step-by-step guide. Learn to debug detection, masking, and compression stages using logs, inspection, and metrics in the chopratejas/headroom repo.

- Repository: [Tejas Chopra/headroom](https://github.com/chopratejas/headroom)
- Tags: how-to-guide
- Published: 2026-06-14

---

**Debug Headroom compression issues by systematically tracing through detection, masking, compression, and CCR storage stages using debug logging, manual mask inspection, and metrics endpoints.**

Headroom’s compression pipeline is a multi-stage system that processes content through detection, structural masking, entropy preservation, and compression before storing originals in CCR (Compress-Cache-Retrieve). When you encounter unexpected compression ratios, missing tokens, or retrieval failures, you can pinpoint the exact stage causing problems by examining the pipeline’s internal state.

## Enable Debug Logging for Pipeline Visibility

Headroom emits OTLP traces and standard logs for every pipeline step. Set the environment variable `HEADROOM_LOG_LEVEL=DEBUG` before running your process to expose detailed stage information.

The logs categorize each pipeline phase under specific Python logger names:

- `headroom.compression.detector` – Content type detection and confidence scores
- `headroom.compression.handlers` – Handler selection and mask preservation ratios
- `headroom.compression.masks` – Entropy mask token counts
- `headroom.compression.universal` – Byte span compression details
- `headroom.cache.compression_store` – CCR storage operations

You can capture these using standard Python logging or tail the log directory:

```bash
export HEADROOM_LOG_LEVEL=DEBUG
export PYTHONVERBOSE=1
tail -f $HOME/.headroom/logs/*.log

```

## Inspect Content Detection Results

The pipeline begins in [`headroom/compression/universal.py`](https://github.com/chopratejas/headroom/blob/main/headroom/compression/universal.py) where the `UniversalCompressor` class detects content type using Magika ML or a fallback detector. Detection errors propagate through all subsequent stages, so verify the `DetectionResult` object first.

Use the `return_detection=True` flag to inspect detection metadata:

```python
from headroom.compression import UniversalCompressor

compressor = UniversalCompressor()
result = compressor.compress(my_text, return_detection=True)
print(result.detection)

# Output: {'content_type': 'CODE', 'confidence': 0.97, 'language': 'python'}

```

If detection confidence is low or incorrect, check that the **magika** package is installed (`pip install "headroom-ai[magika]"`) and that the input is not being truncated before detection. The truncation logic resides in `_estimate_tokens` within the same file.

## Verify Structure Mask Generation

After detection, the pipeline routes content to a specific handler (e.g., `JSONStructureHandler` in [`headroom/compression/handlers/json_handler.py`](https://github.com/chopratejas/headroom/blob/main/headroom/compression/handlers/json_handler.py) or the code handler in [`headroom/compression/handlers/code_handler.py`](https://github.com/chopratejas/headroom/blob/main/headroom/compression/handlers/code_handler.py)). Each handler returns a `StructureMask` that marks structural tokens for preservation.

Examine the mask’s `preservation_ratio` to determine if too much or too little content is being protected:

```python
from headroom.compression.handlers.json_handler import JSONStructureHandler

handler = JSONStructureHandler()
mask_result = handler.get_mask(json_text, list(json_text))
print(mask_result.mask.preservation_ratio)

# 0.41 indicates 41% of tokens preserved as structure

```

A very low preservation ratio suggests aggressive masking. Adjust handler thresholds like `preserve_short_values` or `entropy_threshold` in your handler configuration. If the mask is empty, verify that Tree-sitter is installed for code handlers (`pip show tree_sitter`).

## Check Entropy Preservation Masking

When `use_entropy_preservation` is enabled (default in `UniversalCompressorConfig`), Headroom applies an additional entropy-based mask to preserve high-entropy tokens like UUIDs and hashes. This logic is implemented in [`headroom/compression/masks.py`](https://github.com/chopratejas/headroom/blob/main/headroom/compression/masks.py).

Test entropy preservation independently:

```python
from headroom.compression.masks import compute_entropy_mask

tokens = list(content)
entropy_mask = compute_entropy_mask(tokens, threshold=0.85)
print(f"Preserved {entropy_mask.preservation_ratio * 100}% high-entropy tokens")

```

If this ratio is unexpectedly high, you are preventing effective compression. Lower the `entropy_threshold` or disable the feature:

```python
from headroom.compression import UniversalCompressorConfig

config = UniversalCompressorConfig(use_entropy_preservation=False)
compressor = UniversalCompressor(config=config)

```

## Isolate the Compression Function

The pipeline ultimately calls `_compress_fn` in [`headroom/compression/universal.py`](https://github.com/chopratejas/headroom/blob/main/headroom/compression/universal.py), which either uses `KompressCompressor.compress` (if installed) or falls back to `_simple_compress`. Isolate this stage by injecting a debug function:

```python
def debug_compress(span):
    print(f"Compressing span of {len(span)} characters")
    return span  # Return unmodified to test masking logic

compressor = UniversalCompressor(compress_fn=debug_compress)
result = compressor.compress(large_input)

```

If span lengths appear correct but output remains large, the issue lies in the mask union operation (`structure_mask.union(entropy_mask)`) or in the handler’s `get_mask` implementation rather than the compression algorithm itself.

## Validate CCR Storage Operations

When `ccr_enabled=True`, the compressor stores the original content via [`headroom/cache/compression_store.py`](https://github.com/chopratejas/headroom/blob/main/headroom/cache/compression_store.py). Check the `CompressionResult.ccr_key` field to verify storage succeeded:

```python
from headroom.compression import compress, UniversalCompressorConfig

result = compress(
    content=large_text,
    config=UniversalCompressorConfig(ccr_enabled=True)
)

if result.ccr_key is None:
    print("CCR storage failed - check headroom.cache.compression_store logs")
else:
    print(f"Original stored under key: {result.ccr_key}")

```

Ensure the CCR store path (default `~/.headroom/session_stats.jsonl`) is writable and the cache module is importable. Storage failures cause the pipeline to fall back to no-op keys, which can be observed in the debug logs under `headroom.cache.compression_store`.

## Monitor Pipeline Metrics

Headroom exposes Prometheus-formatted metrics at `/metrics` and JSON debug dumps at `/debug/compression` when running the proxy server.

Query compression performance counters:

```bash
curl http://localhost:8787/metrics | grep headroom_compression
curl http://localhost:8787/debug/compression

```

Key metrics include `headroom_compression_ratio_avg` and `headroom_compression_savings_usd`, defined in the repository’s [`wiki/metrics.md`](https://github.com/chopratejas/headroom/blob/main/wiki/metrics.md). These reveal whether the pipeline is consistently under-performing across all requests or if the issue is request-specific.

## Run Unit Tests for Targeted Debugging

Headroom includes comprehensive tests for each compressor stage. Run specific test suites to isolate failing components:

```bash
pytest tests/test_transforms_log_compressor.py -k compress
pytest tests/test_transforms_content_router.py -k detection

```

Failed tests provide stack traces pointing directly to problematic functions like `_detect_format`, `_dedupe_similar`, or `_compress_with_mask`.

## Verify Dependency Versions

Missing or mismatched dependencies cause silent fallbacks that degrade compression:

- **Magika**: Required for ML-based content detection (`pip show magika`)
- **Kompress**: Optional ML compressor for advanced compression (`pip show kompress`)
- **Tree-sitter**: Required for code structure parsing (`pip show tree_sitter`)

Check your installed versions against the requirements in [`requirements.txt`](https://github.com/chopratejas/headroom/blob/main/requirements.txt) or [`Cargo.toml`](https://github.com/chopratejas/headroom/blob/main/Cargo.toml). If Kompress is missing, the pipeline silently falls back to simple truncation, which may explain poor compression ratios.

## Summary

- **Enable debug logging** using `HEADROOM_LOG_LEVEL=DEBUG` to trace every pipeline stage through [`headroom.compression.universal.py`](https://github.com/chopratejas/headroom/blob/main/headroom.compression.universal.py).
- **Verify detection** using `return_detection=True` and ensure Magika is installed for accurate content-type identification.
- **Inspect structure masks** via handler classes in `headroom/compression/handlers/` to confirm preservation ratios match expectations.
- **Test entropy masking** independently using `compute_entropy_mask` from [`headroom/compression/masks.py`](https://github.com/chopratejas/headroom/blob/main/headroom/compression/masks.py).
- **Isolate compression** by substituting `compress_fn` with a debug stub to verify span lengths before compression.
- **Check CCR storage** by examining the `ccr_key` field in `CompressionResult` and ensuring the cache store is reachable.
- **Monitor metrics** via `/metrics` and `/debug/compression` endpoints to identify systemic vs. isolated issues.
- **Validate dependencies** for Magika, Kompress, and Tree-sitter to prevent silent fallback behaviors.

## Frequently Asked Questions

### Why is my compression ratio unexpectedly low in Headroom?

Low compression ratios typically indicate that the structure mask or entropy mask is preserving too many tokens. Check the `preservation_ratio` in your handler’s `StructureMask` and the entropy mask settings in [`headroom/compression/masks.py`](https://github.com/chopratejas/headroom/blob/main/headroom/compression/masks.py). If the combined mask preserves more than 60-70% of tokens, adjust the `entropy_threshold` or handler-specific settings like `preserve_short_values` to allow more content through to the compression stage.

### How do I disable specific pipeline stages for debugging?

Use `UniversalCompressorConfig` to toggle individual stages. Set `use_magika=False` to force fallback detection, `use_entropy_preservation=False` to skip entropy masking, or `ccr_enabled=False` to bypass CCR storage. You can also inject custom handlers via `compressor.register_handler()` to test specific content types without affecting the global configuration.

### Where does Headroom store the original content during compression?

When CCR is enabled (default), Headroom stores originals in the Compress-Cache-Retrieve system implemented in [`headroom/cache/compression_store.py`](https://github.com/chopratejas/headroom/blob/main/headroom/cache/compression_store.py), typically under `~/.headroom/session_stats.jsonl`. The `CompressionResult` object returns a `ccr_key` reference that points to this stored content. If `ccr_key` is `None`, storage failed and the pipeline operated in fallback mode.

### What should I check if Headroom fails to detect my content type correctly?

First, verify that `magika` is installed (`pip install "headroom-ai[magika]"`). Then inspect the detection result using `return_detection=True` in the `compress()` call. If the input is truncated before detection, check the `_estimate_tokens` function in [`headroom/compression/universal.py`](https://github.com/chopratejas/headroom/blob/main/headroom/compression/universal.py). For edge cases, you can force a specific handler by calling `compressor.register_handler()` with your preferred handler class before compression.