Debugging Headroom: What to Do When Compression Breaks Agent Behavior

When Headroom compression breaks agent behavior, the root cause is typically a misrouted payload in the Content Router, a timeout or exception in the Rust-backed compression layer, or a failure to persist metadata in the Compression Store (CCR) that forces the pipeline to return uncompressed text.

Headroom is an open-source token-compression framework designed to reduce LLM context-window usage through content-specific transforms. When the compression pipeline malfunctions, downstream agents may exhibit garbled output, context loss, or complete stalls. Debugging these issues requires tracing the execution flow through [headroom/transforms/pipeline.py](https://github.com/chopratejas/headroom/blob/main/headroom/transforms/pipeline.py) and verifying the health of each architectural layer.

How Headroom's Compression Pipeline Works

The Headroom architecture routes raw text through a series of specialized compressors before the final payload reaches the agent. Understanding these layers is essential for identifying where the breakage occurs.

Content Router

The Content Router residing in [headroom/transforms/pipeline.py](https://github.com/chopratejas/headroom/blob/main/headroom/transforms/pipeline.py) inspects incoming content and selects the appropriate compressor based on payload type—code, logs, search results, or tabular data. Mis-routing (for example, sending code to the log compressor) leads to unexpected truncation or token explosions.

Transform-Specific Compressors

Each route invokes a dedicated compressor implemented in its own module:

• SmartCrusher ([headroom/transforms/smart_crusher.py](https://github.com/chopratejas/headroom/blob/main/headroom/transforms/smart_crusher.py)): AST-based compression for source code using Tree-sitter.
• LogCompressor ([headroom/transforms/log_compressor.py](https://github.com/chopratejas/headroom/blob/main/headroom/transforms/log_compressor.py)): Pattern-based reduction for log lines.
• SearchCompressor ([headroom/transforms/search_compressor.py](https://github.com/chopratejas/headroom/blob/main/headroom/transforms/search_compressor.py)): Snippet summarization for search results.
• Kompress ([headroom/transforms/kompress_compressor.py](https://github.com/chopratejas/headroom/blob/main/headroom/transforms/kompress_compressor.py)): Rust-backed neural compression requiring native binaries in headroom/_core.

Compression Store (CCR)

The Compression Cache and Repository (CCR) in [headroom/cache/compression_store.py](https://github.com/chopratejas/headroom/blob/main/headroom/cache/compression_store.py) persists compression metadata for learning and caching. When the store is unavailable, the pipeline falls back to passthrough mode, returning the original text and breaking agents that expect compressed input.

Observability Layer

Metrics emission is handled by [headroom/transforms/observability.py](https://github.com/chopratejas/headroom/blob/main/headroom/transforms/observability.py). Missing observability signals can mask silent failures, making it appear as though compression succeeded when it actually aborted.

Common Failure Triggers That Disrupt Agents

Missing Tree-sitter Dependencies

SmartCrusher relies on the tree_sitter Python package to parse source code. If Tree-sitter cannot be imported, the compressor falls back to a no-op, but the router still expects a shortened result. This mismatch causes downstream agents to receive the original, oversized text.

Compression Store Errors

The store may raise exceptions (for example, database connection loss) inside the get_compression_store() function (see lines 795–818 in [headroom/transforms/smart_crusher.py](https://github.com/chopratejas/headroom/blob/main/headroom/transforms/smart_crusher.py)). While the code catches and logs these errors, a complete store outage forces the pipeline to skip the record_compression step, leading to repeated re-compression of the same payload.

Threshold Misconfiguration

Every compressor defines a min_compression_ratio_for_ccr threshold (for example, 0.5 for logs, 0.8 for search). If the actual compression ratio exceeds this value, the result is treated as passthrough. Agents relying on a compressed payload then receive the full original text, blowing token budgets.

Pipeline Timeouts

The environment variable HEADROOM_COMPRESSION_TIMEOUT_SECONDS defaults to 30 seconds. Long-running Rust models (Kompress) that exceed this limit abort and return the original text, causing agents to process unreduced context.

Tag Protector Interference

The Tag Protector in [headroom/transforms/tag_protector.py](https://github.com/chopratejas/headroom/blob/main/headroom/transforms/tag_protector.py) shields custom XML tags from compression via protect_tags and restore_tags. Incorrect boundary definitions can cause the protector to strip essential markers, leaving the router without valid compression hints and breaking agent parsers.

Step-by-Step Debugging Workflow

1. Validate Router Decisions Inspect debug logs for messages like “ContentRouter: routing to log_compressor”. Verify the selected compressor matches the payload type—code should route to SmartCrusher, logs to LogCompressor.

2. Check Compressor Health

   • Search logs for compression_ratio. A value of 1.0 indicates no compression occurred.
   • Confirm the Rust binary is loadable by checking for warnings such as “Kompress model X not cached” in headroom/_core.

3. Confirm Store Availability Look for the log line “CCR mirror: compression_store module unavailable”. If present, verify that the optional headroom-cache package is installed and that HEADROOM_CACHE_URL (if used) is correctly configured.

4. Review Threshold Settings Open the compressor config (for example, LogCompressorConfig in [headroom/transforms/log_compressor.py](https://github.com/chopratejas/headroom/blob/main/headroom/transforms/log_compressor.py)). Lower min_compression_ratio_for_ccr temporarily to force compression events and surface hidden paths.

5. Enable Detailed Observability Instantiate a PrometheusCompressionObserver (see [headroom/transforms/observability.py](https://github.com/chopratejas/headroom/blob/main/headroom/transforms/observability.py)) and call record_compression after each transform. A sudden drop to zero events indicates a silent failure.

6. Reproduce with Unit Tests Run the relevant test suites to verify compressor behavior:

   pytest tests/test_transforms_log_compressor.py::test_log_compression_ratio
   pytest tests/test_transforms_search_compressor.py::test_search_compression_result_properties

   These tests assert that result.compression_ratio stays below the configured threshold and that the store interaction succeeds.

7. Inspect Tag Boundaries Verify that custom tags are correctly escaped as <mytag>…</mytag>. Ensure protect_tags wraps content without stripping inner text, which would confuse the downstream agent parser.

Configuration Fixes and Validation Code

Forcing the Log Compressor to Run

Use this snippet to bypass the compression ratio threshold and verify the compressor logic itself:

from headroom.transforms.log_compressor import LogCompressor, LogCompressorConfig

cfg = LogCompressorConfig(min_compression_ratio_for_ccr=0.0)  # disable threshold

compressor = LogCompressor(cfg)

payload = "2023-01-01 12:00:00 INFO Starting service...\n" * 20
result = compressor.apply(payload, compression_policy=None)

print("Compressed length:", len(result.compressed))
print("Ratio:", result.compression_ratio)

Relevant source: [headroom/transforms/log_compressor.py](https://github.com/chopratejas/headroom/blob/main/headroom/transforms/log_compressor.py)

Recording a Compression Event Manually

Verify that observability hooks are correctly wired by emitting a manual metric:

from headroom.transforms.observability import PrometheusCompressionObserver
from headroom.transforms.smart_crusher import SmartCrusher

observer = PrometheusCompressionObserver()
crusher = SmartCrusher()
payload = "def foo():\n    return 42\n"
result = crusher.apply(payload, compression_policy=None)

observer.record_compression(
    strategy="smart_crusher",
    compression_ratio=result.compression_ratio,
    original_len=len(payload),
    compressed_len=len(result.compressed),
)

Relevant source: [headroom/transforms/observability.py](https://github.com/chopratejas/headroom/blob/main/headroom/transforms/observability.py)

Simulating a Store Failure

Test graceful degradation when the CCR store is unreachable:

from unittest.mock import patch
from headroom.transforms.smart_crusher import SmartCrusher

def broken_store(*_, **__):
    raise RuntimeError("Simulated store outage")

with patch("headroom.transforms.smart_crusher.get_compression_store", broken_store):
    crusher = SmartCrusher()
    # This will still compress but skip storing the metadata

    result = crusher.apply("some long text …")
    print("Compression succeeded; store was bypassed.")

Relevant source: Lines 795–819 in [headroom/transforms/smart_crusher.py](https://github.com/chopratejas/headroom/blob/main/headroom/transforms/smart_crusher.py)

Summary

• Headroom routes content through specialized compressors in [headroom/transforms/pipeline.py](https://github.com/chopratejas/headroom/blob/main/headroom/transforms/pipeline.py); mis-routing is the most common cause of broken agent behavior.
• Tree-sitter and Rust binaries (headroom/_core) are hard dependencies for code and neural compression; missing them triggers silent no-op fallbacks.
• Thresholds defined by min_compression_ratio_for_ccr can cause the pipeline to return uncompressed text if the ratio is too high.
• Timeouts governed by HEADROOM_COMPRESSION_TIMEOUT_SECONDS abort slow compressors, returning the original payload.
• CCR store failures (monitored via get_compression_store()) disable metadata persistence but should not block the compression itself.
• Use unit tests from tests/test_transforms_log_compressor.py and tests/test_transforms_search_compressor.py to validate fixes.

Frequently Asked Questions

Why is my agent receiving uncompressed text even though Headroom is enabled?

The Content Router may have selected a passthrough route due to a high min_compression_ratio_for_ccr threshold, or the Compression Store may be unavailable, forcing a fallback. Check debug logs for compression_ratio: 1.0 and verify that HEADROOM_COMPRESSION_TIMEOUT_SECONDS has not been exceeded.

How do I identify which compressor Headroom selected for my payload?

Enable debug logging by setting export HEADROOM_LOG_LEVEL=debug, then look for entries emitted by [headroom/transforms/pipeline.py](https://github.com/chopratejas/headroom/blob/main/headroom/transforms/pipeline.py) containing the string “ContentRouter: routing to”. The log line will specify the compressor class name (for example, smart_crusher or log_compressor).

What causes the "compression_store module unavailable" warning in logs?

This warning appears when the CCR import in [headroom/cache/compression_store.py](https://github.com/chopratejas/headroom/blob/main/headroom/cache/compression_store.py) fails, usually because the optional headroom-cache dependency is not installed or because HEADROOM_CACHE_URL points to an unreachable endpoint. Install the extra dependency or correct the URL to resolve the warning.

How can I adjust compression thresholds to force compression during testing?

Lower the min_compression_ratio_for_ccr value in the compressor’s configuration class (for example, LogCompressorConfig or the equivalent in SmartCrusher). Setting it to 0.0 disables the threshold entirely, ensuring the compressor always runs and allowing you to verify agent behavior against compressed output regardless of the achieved ratio.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}