# Kompress-v2-base ML Model: A ModernBERT-Based Prose Compressor for Headroom

> Discover Kompress-v2-base, a ModernBERT prose compressor achieving 80-95% text reduction by classifying tokens and scoring spans, preserving semantic meaning.

- Repository: [Tejas Chopra/headroom](https://github.com/chopratejas/headroom)
- Tags: deep-dive
- Published: 2026-06-18

---

**Kompress-v2-base is the default ModernBERT-based token compressor that classifies each token as "keep" or "discard" while scoring token spans via a CNN, enabling 80-95% compression of prose while preserving semantic meaning.**

The **Kompress-v2-base ML model** is the core neural compressor shipped with the Headroom open-source framework. Hosted on Hugging Face under the identifier `chopratejas/kompress-v2-base`, this model automatically downloads on first use and integrates seamlessly into Headroom's text processing pipeline via the `KompressCompressor` class.

## Architecture and Design of Kompress-v2-base

Kompress-v2-base implements a **dual-head architecture** built on the ModernBERT transformer architecture. The model processes tokenized text to determine which elements contribute to semantic meaning and which can be safely removed.

### Dual-Head Classification Approach

The model employs two complementary heads working in tandem:

1. **Binary Classification Head**: Assigns each token a "keep" or "discard" label based on individual importance
2. **CNN Span Scoring Head**: Evaluates the importance of token spans using a small convolutional neural network

During inference, these heads are combined so that borderline tokens within high-importance spans are retained. This approach prevents the loss of critical contextual information that might occur with simple token-level filtering alone.

### Performance Benchmarks

According to the test data in the Headroom source code, Kompress-v2-base achieves **80-95% token compression** while maintaining a high F-score of approximately **0.913** on the labeled test split. This compression ratio represents the reduction from original token count to compressed token count, not character count, making it particularly effective for long-form prose where redundant tokens are common.

## Configuration and Implementation Details

The compressor's behavior is controlled through the `KompressConfig` dataclass defined in [`headroom/transforms/kompress_compressor.py`](https://github.com/chopratejas/headroom/blob/main/headroom/transforms/kompress_compressor.py) (lines 90-96).

### KompressConfig Parameters

Key configuration fields include:

- **`model_id`**: Defaults to `chopratejas/kompress-v2-base` (Hugging Face repository containing model weights and tokenizer)
- **`chunk_words`**: Maximum 350 words processed per inference chunk (prevents memory issues with long documents)
- **`score_threshold`**: 0.5 probability threshold for token retention when no explicit target ratio is provided
- **`device`**: Auto-selection prefers CUDA, then MPS, then CPU; can be forced via configuration or environment variable

### Backend Selection (ONNX vs PyTorch)

Headroom loads Kompress-v2-base via two possible backends determined by the `HEADROOM_KOMPRESS_BACKEND` environment variable:

- **ONNX Runtime**: Lightweight CPU inference optimized for speed and minimal dependencies
- **PyTorch**: Full-stack inference with optional CUDA/MPS support for GPU acceleration

The default selection automatically prefers ONNX-CPU and falls back to PyTorch if ONNX is unavailable, as implemented in [`headroom/transforms/kompress_compressor.py`](https://github.com/chopratejas/headroom/blob/main/headroom/transforms/kompress_compressor.py) (lines 100-115).

## Using Kompress-v2-base in Headroom

The `KompressCompressor` class provides the primary interface for prose compression, handling tokenization, chunking, model inference, and result reconstruction.

### Basic Compression Example

For straightforward compression of long text:

```python
from headroom.transforms.kompress_compressor import KompressCompressor

compressor = KompressCompressor()               # uses KompressConfig defaults

long_text = """Your long prose …"""
result = compressor.compress(long_text)

print("Compressed:", result.compressed)
print("Original tokens:", result.original_tokens)
print("Compressed tokens:", result.compressed_tokens)
print("Ratio:", result.compression_ratio)

```

The method returns a `KompressResult` object containing the compressed string and metadata including original/compressed token counts and the compression ratio.

### Batch Processing and Custom Configuration

Override defaults for specific latency or quality requirements:

```python
from headroom.transforms.kompress_compressor import KompressCompressor, KompressConfig

cfg = KompressConfig(
    model_id="chopratejas/kompress-v2-base",  # default – shown for clarity

    chunk_words=200,                         # smaller chunks for lower latency

    score_threshold=0.45,                    # more aggressive pruning

    device="cuda"                            # force GPU (requires PyTorch)

)
compressor = KompressCompressor(cfg)

texts = ["First long paragraph …", "Second long paragraph …"]
batch_results = compressor.compress_batch(texts)

for r in batch_results:
    print(r.compressed, r.compression_ratio)

```

### Direct Model Access for Diagnostics

Access the underlying model and tokenizer for debugging or custom inference:

```python
from headroom.transforms.kompress_compressor import _load_kompress, HF_MODEL_ID

model, tokenizer, backend = _load_kompress(HF_MODEL_ID)
print("Backend:", backend)          # "onnx" or "pytorch"

print("Model type:", type(model))   # _OnnxModel or torch.nn.Module

```

When processing text longer than ten words, the compressor splits input into chunks, tokenizes using the ModernBERT tokenizer, feeds tokens to the model, and produces either a **keep-mask** (ONNX) or **score list** (PyTorch). Tokens passing the threshold are retained; others are dropped, with the final string returned along with statistics in the `KompressResult` dataclass (lines 112-119).

## Summary

- **Kompress-v2-base** is a ModernBERT-based dual-head model hosted at `chopratejas/kompress-v2-base` that achieves 80-95% token compression with ~0.913 F-score.
- The architecture combines binary token classification with CNN-based span scoring to preserve semantic meaning during aggressive compression.
- Configuration is managed through `KompressConfig` in [`headroom/transforms/kompress_compressor.py`](https://github.com/chopratejas/headroom/blob/main/headroom/transforms/kompress_compressor.py), supporting customizable chunk sizes, thresholds, and device selection.
- Headroom supports both **ONNX Runtime** (CPU-optimized) and **PyTorch** (GPU-capable) backends, selectable via the `HEADROOM_KOMPRESS_BACKEND` environment variable.
- The `KompressCompressor` class handles chunking, inference, and result packaging, returning detailed statistics via the `KompressResult` dataclass.

## Frequently Asked Questions

### What makes Kompress-v2-base different from other text compression models?

Unlike rule-based compressors or simple extractive summarizers, Kompress-v2-base uses a **neural token-level approach** that removes individual tokens rather than entire sentences. This granular classification, combined with span-aware CNN scoring, preserves grammatical structure and meaning even at 80-95% compression rates, whereas traditional methods often degrade readability at high compression levels.

### How does the dual-head architecture work in Kompress-v2-base?

The **binary head** makes keep/discard decisions for each token independently, while the **CNN head** evaluates the importance of surrounding token spans. During inference, these signals are fused so that tokens with marginal individual scores are retained if they fall within high-importance spans. This prevents the loss of contextually critical phrases that might be fragmented by purely token-level filtering.

### Can I use Kompress-v2-base without installing PyTorch?

Yes. Headroom supports **ONNX Runtime** as a lightweight alternative that requires only CPU resources and minimal dependencies. Set `HEADROOM_KOMPRESS_BACKEND=onnx` or allow the auto-selection to default to ONNX when PyTorch is unavailable. The ONNX backend provides faster cold-start times for CPU-only deployments while maintaining the same compression quality.

### What compression ratios can I expect with Kompress-v2-base?

According to the Headroom source code benchmarks, Kompress-v2-base typically achieves **80-95% compression** of token counts while maintaining high semantic fidelity (F-score ≈ 0.913). The actual ratio depends on text redundancy and the `score_threshold` parameter; lowering the threshold below the default 0.5 increases compression at the potential cost of information loss.