# Headroom's Kompress Compressor: ModernBERT Token Classification for LLM Context

> Discover Headroom's Kompress compressor, leveraging ModernBERT token classification for efficient LLM context compression. Achieve 80-95% reduction while preserving meaning.

- Repository: [Tejas Chopra/headroom](https://github.com/chopratejas/headroom)
- Tags: deep-dive
- Published: 2026-06-13

---

**Kompress is Headroom's built-in ML compressor that uses a dual-head ModernBERT model to classify tokens as keep or discard, achieving 80-95% compression while preserving semantic meaning for downstream LLMs.**

The `chopratejas/headroom` repository includes a sophisticated token classification system called Kompress that reduces context window usage by intelligently pruning unnecessary tokens from tool outputs and assistant messages. This compressor leverages a custom dual-head ModernBERT architecture to perform binary token classification and span importance scoring, enabling aggressive compression without sacrificing the contextual details required by large language models.

## How Kompress Works: Dual-Head ModernBERT Architecture

In [`headroom/transforms/kompress_compressor.py`](https://github.com/chopratejas/headroom/blob/main/headroom/transforms/kompress_compressor.py), the `HeadroomCompressorModel` implements a specialized architecture built on ModernBERT, a BERT-style encoder optimized for token-level relevance tasks rather than standard masked language modeling.

The model features two complementary classification heads:

### Token-Level Binary Classification

The `self.token_head` applies a binary linear classification to each input token, predicting whether individual tokens should be retained or discarded. This head operates on the hidden states produced by the ModernBERT encoder base.

### Span Importance Scoring

A lightweight 1-D convolutional neural network (`self.span_conv`) performs span-importance scoring to identify semantically coherent regions like code blocks or file paths. This prevents the compressor from fragmenting important contextual spans during the pruning process.

The final keep mask is computed as the logical OR of the token head's decision and a "borderline-token-boosted-by-span" rule (see lines 31-33 of the model definition), ensuring that tokens within important spans receive preservation priority.

## The Three-Stage Compression Pipeline

Kompress operates through a distinct inference pipeline implemented in the `_compress` method:

### 1. Model Loading and Initialization

On first use, the system auto-downloads model weights from `chopratejas/kompress-v2-base` and tokenizer configuration from `answerdotai/ModernBERT-base`. The loading logic in `_load_kompress` (lines 78-104) supports multiple backends:

- **ONNX**: Lightweight CPU inference
- **PyTorch**: GPU or Apple Silicon (MPS) acceleration
- **Disable**: Passthrough mode via `HEADROOM_KOMPRESS_BACKEND=disable`

### 2. Chunked Inference

Input text is split into 350-word chunks. Each chunk undergoes tokenization and forward pass through the model. The `get_keep_mask` method (for ONNX) or `get_scores` method (for PyTorch) generates binary masks or probability scores. The PyTorch implementation combines token-head probabilities with span convolutions to determine final retention decisions.

### 3. Re-assembly and CCR Caching

Kept token indices merge back into the compressed string. If the compression ratio falls below the CCR threshold (approximately 0.8), the result is stored via `_store_in_ccr` (lines 140-150) with a hash-based cache key appended to the output for later retrieval.

## Installation and Backend Configuration

Kompress operates as an opt-in ML compression feature. Install the optional dependencies:

```bash
pip install "headroom-ai[ml]"

```

Control the inference backend using environment variables:

```python
import os
os.environ["HEADROOM_KOMPRESS_BACKEND"] = "onnx"  # or "pytorch", "disable"

```

When running on GPU hardware, Kompress supports batch inference for multiple texts in a single forward pass, providing approximately 2× speedup for three or more concurrent items.

## Source Code Implementation Details

The core implementation resides in [`headroom/transforms/kompress_compressor.py`](https://github.com/chopratejas/headroom/blob/main/headroom/transforms/kompress_compressor.py), which defines:

- `KompressConfig`: Runtime configuration dataclass
- `KompressResult`: Compression metrics including `tokens_saved` and `savings_percentage`
- `KompressCompressor`: Main interface with `compress()` and `compress_batch()` methods

The compression store implementation in [`headroom/cache/compression_store.py`](https://github.com/chopratejas/headroom/blob/main/headroom/cache/compression_store.py) provides hash-based retrieval for cached compressions, enabling the CCR (Compression Cache and Retrieval) system to serve repeat queries without re-computation.

Unit tests in [`tests/test_transforms/test_kompress_compressor.py`](https://github.com/chopratejas/headroom/blob/main/tests/test_transforms/test_kompress_compressor.py) verify importability without PyTorch dependencies, passthrough behavior when disabled, and core compression logic.

## Practical Usage Examples

Compress a single long tool output:

```python
from headroom.transforms.kompress_compressor import KompressCompressor

compressor = KompressCompressor()
result = compressor.compress(
    """This is a very long JSON payload returned by a tool call ...
    (imagine many lines of diagnostics)"""
)

print("Compressed:", result.compressed)
print(f"Saved {result.tokens_saved} tokens ({result.savings_percentage:.1f}%).")

```

Process multiple responses simultaneously using batch inference:

```python
texts = ["...first long response...", "...second long response..."]
batch_results = compressor.compress_batch(texts)

for r in batch_results:
    print(r.compression_ratio, r.compressed[:100])

```

Retrieve previously compressed content via CCR hash:

```python
from headroom.cache.compression_store import get_compression_store

store = get_compression_store()
original, compressed = store.retrieve("0a1b2c3d4e5f...")
print(compressed)

```

## Summary

- **Headroom's Kompress compressor** uses a dual-head ModernBERT architecture to classify tokens for retention or removal, achieving 80-95% compression with minimal semantic loss.
- The system combines **token-level binary classification** (`self.token_head`) with **span-importance scoring** (`self.span_conv`) to preserve contextual coherence.
- Three-stage pipeline includes model loading (ONNX/PyTorch backends), chunked inference with 350-word segments, and CCR caching via `_store_in_ccr`.
- Install via `pip install "headroom-ai[ml]"` and configure using the `HEADROOM_KOMPRESS_BACKEND` environment variable.
- Core implementation is located in [`headroom/transforms/kompress_compressor.py`](https://github.com/chopratejas/headroom/blob/main/headroom/transforms/kompress_compressor.py) with caching support in [`headroom/cache/compression_store.py`](https://github.com/chopratejas/headroom/blob/main/headroom/cache/compression_store.py).

## Frequently Asked Questions

### How does Kompress differ from standard text compression algorithms?

Unlike general-purpose compression algorithms that operate on byte sequences, Kompress performs **semantic token classification** using ModernBERT to identify which tokens can be removed without affecting LLM comprehension. This approach targets the specific needs of language model context windows rather than file-size reduction, typically achieving 80-95% token reduction while maintaining ≤5% meaning loss for downstream tasks.

### What hardware backends does Kompress support?

Kompress supports three backend modes controlled via the `HEADROOM_KOMPRESS_BACKEND` environment variable: **ONNX** for lightweight CPU inference, **PyTorch** for CUDA or Apple Silicon (MPS) acceleration, and **disable** for passthrough mode. The system auto-detects available hardware and can utilize GPU batch processing for approximately 2× speedup when compressing three or more texts concurrently.

### Where is the compressed data cached and how is it retrieved?

Compressed results meeting the CCR threshold (compression ratio < 0.8) are stored via the `_store_in_ccr` method in [`headroom/cache/compression_store.py`](https://github.com/chopratejas/headroom/blob/main/headroom/cache/compression_store.py). The system generates a hash-based cache key appended to the output. Retrieve cached content using `get_compression_store().retrieve("hash_key")`, which returns both the original and compressed text without re-running the ModernBERT model.

### Can I use Kompress without installing PyTorch?

Yes. The [`tests/test_transforms/test_kompress_compressor.py`](https://github.com/chopratejas/headroom/blob/main/tests/test_transforms/test_kompress_compressor.py) verifies that Kompress remains importable and functional without PyTorch installed by using the ONNX backend or disabled mode. Install the base `headroom-ai` package without the `[ml]` extra for systems where PyTorch dependencies are undesirable, though this limits you to the `disable` backend or requires manual ONNX runtime setup.