# How the Kompress‑Base ML Model Compresses Text: Token‑Level Classification with Span Boosting

> Discover how the Kompress-Base ML model compresses text using token-level classification and span boosting. Learn about its ModernBERT architecture and achieve efficient text compression.

- Repository: [Tejas Chopra/headroom](https://github.com/chopratejas/headroom)
- Tags: deep-dive
- Published: 2026-06-16

---

**The Kompress‑base model uses a dual‑head ModernBERT architecture to classify each token as keep or discard, then enhances decisions with span‑level importance scoring to produce compressed text.**

The Kompress‑base model (hosted at HuggingFace ID `chopratejas/kompress‑v2‑base`) is the core compression engine in the `chopratejas/headroom` repository. Unlike traditional extractive summarizers that rely on heuristic sentence selection, this **neural text compressor** operates at the token level, making binary keep/discard decisions while leveraging contiguous span information to preserve semantic coherence. This article breaks down exactly how the model architecture, decision logic, and runtime pipeline work together to achieve efficient text compression.

## Model Architecture: Dual‑Head ModernBERT Design

The Kompress‑base architecture is defined in `HeadroomCompressorModel` at lines 89‑112 of [`headroom/transforms/kompress_compressor.py`](https://github.com/chopratejas/headroom/blob/main/headroom/transforms/kompress_compressor.py). The implementation uses three key components:

- **ModernBERT encoder** – Initialized via `AutoModel.from_pretrained`, this produces 768‑dimensional hidden representations for each input token
- **Token classification head** (`self.token_head`) – A binary classifier that outputs logits for *keep* vs *discard* decisions per token
- **Span convolution head** (`self.span_conv`) – A 1‑D convolutional network that scores contiguous token spans, enabling the model to identify and preserve important multi‑token phrases

This dual‑head design allows the model to consider both individual token importance and contextual span information when making compression decisions.

## Token‑Level Decision Logic with Borderline Handling

The compression decision pipeline lives in `get_keep_mask` (lines 110‑132). The logic follows a three‑stage process:

1. **Hard token decisions** – The token head produces logits; tokens with higher *keep* logit are marked for retention (`token_keep`)
2. **Borderline identification** – Tokens with keep probability between 0.3 and 0.5 enter a "borderline" region where the decision is uncertain
3. **Span boost rescue** – If the span head marks the same borderline region as important (`span_boost`), those tokens get promoted to the keep set

The final keep mask is the **union** of hard token decisions and borderline‑plus‑span rescues. This approach prevents the model from dropping semantically important tokens that might have marginal individual scores but high contextual relevance within key spans.

## Ratio‑Based Compression via Token Scoring

When a target compression ratio is specified, the model switches to scoring mode via `get_scores` (lines 134‑139). The per‑token scoring formula combines both heads:

```python

# Token probability from the binary classification head

token_prob = token_head_output

# Span boost factor (0.5 base + 0.5 * normalized span score)

span_boost = 0.5 + 0.5 * span_scores

# Final score is the product

final_score = token_prob * span_boost

```

The compressor then ranks tokens by this composite score and keeps the highest‑scoring tokens until the requested ratio is achieved. This scoring mechanism ensures that tokens within important spans receive proportional boost while maintaining the probabilistic foundation from the token head.

## Runtime Pipeline: Chunking and Model Loading

The `compress` method (lines 158‑210) handles the end‑to‑end compression workflow:

### Input Chunking

Long texts are split into **chunks of 350 words by default** to manage memory and computational constraints. Each chunk is processed independently, then reassembled into the final compressed output.

### Model Loading Optimization

The `_load_kompress` function (lines 278‑311) implements automatic runtime selection:

- **ONNX runtime** – Used when available for faster inference with lower memory overhead
- **PyTorch fallback** – Used when ONNX is not available, with automatic CUDA detection for GPU acceleration

This dual‑path loading ensures optimal performance across different deployment environments.

### Batch Processing

For high‑throughput scenarios, the `compress_batch` method processes multiple texts in parallel, automatically leveraging GPU acceleration when PyTorch with CUDA is available.

## Post‑Processing and Compression Cache

When the resulting compression ratio falls below 0.8, the compressor optionally stores results in the **Compression Cache Repository (CCR)** via `_store_in_ccr` (lines 800‑815). This caching mechanism:

- Avoids redundant re‑compression of similar texts
- Appends retrieval hints to cached entries for faster lookup
- Integrates with the broader Headroom caching infrastructure defined in [`headroom/cache/compression_store.py`](https://github.com/chopratejas/headroom/blob/main/headroom/cache/compression_store.py)

## Practical Usage Examples

### Basic Single‑Text Compression

```python
from headroom.transforms.kompress_compressor import KompressCompressor

# Initialize with default config (uses HF_MODEL_ID)

compressor = KompressCompressor()

long_text = "The quick brown fox jumps over the lazy dog. " * 20

result = compressor.compress(long_text)
print(f"Compressed: {result.compressed}")
print(f"Ratio: {result.compression_ratio}")

```

### Custom Model and Aggressive Compression

```python
from headroom.transforms.kompress_compressor import KompressCompressor, KompressConfig

config = KompressConfig(
    model_id="chopratejas/kompress-finance",  # Domain‑specific variant

    chunk_words=100,                          # Smaller chunks for long documents

    score_threshold=0.45                      # More aggressive compression

)
compressor = KompressCompressor(config)

# Force 30% retention rate

result = compressor.compress(long_text, target_ratio=0.3)

```

### Batch Processing with GPU Acceleration

```python
texts = [
    "First long chat log with technical details...",
    "Second long tool output with execution traces...",
    "Third verbose documentation excerpt..."
]

# Automatic GPU utilization when available

batch_results = compressor.compress_batch(texts, target_ratio=0.4)

for r in batch_results:
    print(f"{r.compression_ratio:.2f}: {r.compressed[:80]}...")

```

## Key Implementation Files

The complete Kompress‑base implementation spans these source files:

- **[`headroom/transforms/kompress_compressor.py`](https://github.com/chopratejas/headroom/blob/main/headroom/transforms/kompress_compressor.py)** – Core `HeadroomCompressorModel` class, `KompressCompressor` transform, and compression logic
- **[`headroom/transforms/base.py`](https://github.com/chopratejas/headroom/blob/main/headroom/transforms/base.py)** – Abstract `Transform` base class that defines the interface
- **[`headroom/config.py`](https://github.com/chopratejas/headroom/blob/main/headroom/config.py)** – `TransformResult` dataclass used for structured output
- **[`headroom/cache/compression_store.py`](https://github.com/chopratejas/headroom/blob/main/headroom/cache/compression_store.py)** – CCR cache implementation for storing compressed results
- **[`scripts/export_kompress_v2_onnx.py`](https://github.com/chopratejas/headroom/blob/main/scripts/export_kompress_v2_onnx.py)** – ONNX export script for optimized inference

## Summary

- **Kompress‑base uses ModernBERT with dual heads**: a token classifier for binary keep/discard decisions and a span convnet for contextual importance scoring
- **Decision logic combines hard thresholds with span boosting**: borderline tokens (probability 0.3‑0.5) get rescued if the span head marks them important
- **Ratio‑based compression uses multiplicative scoring**: final token scores combine token probability with span boost factors (0.5 + 0.5 × span_score)
- **Runtime optimization includes ONNX/PyTorch auto‑selection and 350‑word chunking** for efficient memory usage
- **Optional CCR caching** stores results when compression ratio < 0.8 for retrieval‑augmented scenarios

## Frequently Asked Questions

### What makes Kompress‑base different from traditional extractive summarization?

Traditional extractive summarizers select entire sentences or paragraphs based on heuristic features like position or term frequency. Kompress‑base operates at the **token level**, allowing it to preserve critical phrases within otherwise discardable sentences. The dual‑head architecture also captures **span‑level dependencies** that pure sentence‑selectors miss, producing more coherent compression at arbitrary ratios.

### How does the model handle very long documents?

The `compress` method implements **automatic chunking at 350 words** (configurable via `chunk_words`). Each chunk is processed independently through the ModernBERT encoder, then outputs are concatenated. This sliding‑window approach avoids the quadratic memory scaling of full‑document self‑attention while maintaining local coherence. For GPU batch processing, `compress_batch` parallelizes chunks across available CUDA devices.

### Can I use Kompress‑base without HuggingFace dependencies?

Yes. The runtime automatically selects **ONNX inference** when available, eliminating the PyTorch/HuggingFace dependency chain for production deployment. Run [`scripts/export_kompress_v2_onnx.py`](https://github.com/chopratejas/headroom/blob/main/scripts/export_kompress_v2_onnx.py) to convert the HuggingFace checkpoint to ONNX format. The `_load_kompress` function (lines 278‑311) handles ONNX Runtime initialization with graceful fallback to PyTorch when needed.

### What compression ratios are practical with this model?

The model supports **arbitrary target ratios** via the `target_ratio` parameter (0.0 to 1.0). In practice, ratios between 0.3 and 0.6 maintain readable output for most domains. Below 0.3, coherence degrades as the span head cannot compensate for excessive token dropping. The default `score_threshold` of 0.5 provides balanced compression; lowering to 0.3‑0.4 increases aggressiveness at the cost of occasional semantic drift.