Kompress-v2-base ML Model: A ModernBERT-Based Prose Compressor for Headroom

Kompress-v2-base is the default ModernBERT-based token compressor that classifies each token as "keep" or "discard" while scoring token spans via a CNN, enabling 80-95% compression of prose while preserving semantic meaning.

The Kompress-v2-base ML model is the core neural compressor shipped with the Headroom open-source framework. Hosted on Hugging Face under the identifier chopratejas/kompress-v2-base, this model automatically downloads on first use and integrates seamlessly into Headroom's text processing pipeline via the KompressCompressor class.

Architecture and Design of Kompress-v2-base

Kompress-v2-base implements a dual-head architecture built on the ModernBERT transformer architecture. The model processes tokenized text to determine which elements contribute to semantic meaning and which can be safely removed.

Dual-Head Classification Approach

The model employs two complementary heads working in tandem:

1. Binary Classification Head: Assigns each token a "keep" or "discard" label based on individual importance
2. CNN Span Scoring Head: Evaluates the importance of token spans using a small convolutional neural network

During inference, these heads are combined so that borderline tokens within high-importance spans are retained. This approach prevents the loss of critical contextual information that might occur with simple token-level filtering alone.

Performance Benchmarks

According to the test data in the Headroom source code, Kompress-v2-base achieves 80-95% token compression while maintaining a high F-score of approximately 0.913 on the labeled test split. This compression ratio represents the reduction from original token count to compressed token count, not character count, making it particularly effective for long-form prose where redundant tokens are common.

Configuration and Implementation Details

The compressor's behavior is controlled through the KompressConfig dataclass defined in headroom/transforms/kompress_compressor.py (lines 90-96).

KompressConfig Parameters

Key configuration fields include:

• model_id: Defaults to chopratejas/kompress-v2-base (Hugging Face repository containing model weights and tokenizer)
• chunk_words: Maximum 350 words processed per inference chunk (prevents memory issues with long documents)
• score_threshold: 0.5 probability threshold for token retention when no explicit target ratio is provided
• device: Auto-selection prefers CUDA, then MPS, then CPU; can be forced via configuration or environment variable

Backend Selection (ONNX vs PyTorch)

Headroom loads Kompress-v2-base via two possible backends determined by the HEADROOM_KOMPRESS_BACKEND environment variable:

• ONNX Runtime: Lightweight CPU inference optimized for speed and minimal dependencies
• PyTorch: Full-stack inference with optional CUDA/MPS support for GPU acceleration

The default selection automatically prefers ONNX-CPU and falls back to PyTorch if ONNX is unavailable, as implemented in headroom/transforms/kompress_compressor.py (lines 100-115).

Using Kompress-v2-base in Headroom

The KompressCompressor class provides the primary interface for prose compression, handling tokenization, chunking, model inference, and result reconstruction.

Basic Compression Example

For straightforward compression of long text:

from headroom.transforms.kompress_compressor import KompressCompressor

compressor = KompressCompressor()               # uses KompressConfig defaults

long_text = """Your long prose …"""
result = compressor.compress(long_text)

print("Compressed:", result.compressed)
print("Original tokens:", result.original_tokens)
print("Compressed tokens:", result.compressed_tokens)
print("Ratio:", result.compression_ratio)

The method returns a KompressResult object containing the compressed string and metadata including original/compressed token counts and the compression ratio.

Batch Processing and Custom Configuration

Override defaults for specific latency or quality requirements:

from headroom.transforms.kompress_compressor import KompressCompressor, KompressConfig

cfg = KompressConfig(
    model_id="chopratejas/kompress-v2-base",  # default – shown for clarity

    chunk_words=200,                         # smaller chunks for lower latency

    score_threshold=0.45,                    # more aggressive pruning

    device="cuda"                            # force GPU (requires PyTorch)

)
compressor = KompressCompressor(cfg)

texts = ["First long paragraph …", "Second long paragraph …"]
batch_results = compressor.compress_batch(texts)

for r in batch_results:
    print(r.compressed, r.compression_ratio)

Direct Model Access for Diagnostics

Access the underlying model and tokenizer for debugging or custom inference:

from headroom.transforms.kompress_compressor import _load_kompress, HF_MODEL_ID

model, tokenizer, backend = _load_kompress(HF_MODEL_ID)
print("Backend:", backend)          # "onnx" or "pytorch"

print("Model type:", type(model))   # _OnnxModel or torch.nn.Module

When processing text longer than ten words, the compressor splits input into chunks, tokenizes using the ModernBERT tokenizer, feeds tokens to the model, and produces either a keep-mask (ONNX) or score list (PyTorch). Tokens passing the threshold are retained; others are dropped, with the final string returned along with statistics in the KompressResult dataclass (lines 112-119).

Summary

• Kompress-v2-base is a ModernBERT-based dual-head model hosted at chopratejas/kompress-v2-base that achieves 80-95% token compression with ~0.913 F-score.
• The architecture combines binary token classification with CNN-based span scoring to preserve semantic meaning during aggressive compression.
• Configuration is managed through KompressConfig in headroom/transforms/kompress_compressor.py, supporting customizable chunk sizes, thresholds, and device selection.
• Headroom supports both ONNX Runtime (CPU-optimized) and PyTorch (GPU-capable) backends, selectable via the HEADROOM_KOMPRESS_BACKEND environment variable.
• The KompressCompressor class handles chunking, inference, and result packaging, returning detailed statistics via the KompressResult dataclass.

Frequently Asked Questions

What makes Kompress-v2-base different from other text compression models?

Unlike rule-based compressors or simple extractive summarizers, Kompress-v2-base uses a neural token-level approach that removes individual tokens rather than entire sentences. This granular classification, combined with span-aware CNN scoring, preserves grammatical structure and meaning even at 80-95% compression rates, whereas traditional methods often degrade readability at high compression levels.

How does the dual-head architecture work in Kompress-v2-base?

The binary head makes keep/discard decisions for each token independently, while the CNN head evaluates the importance of surrounding token spans. During inference, these signals are fused so that tokens with marginal individual scores are retained if they fall within high-importance spans. This prevents the loss of contextually critical phrases that might be fragmented by purely token-level filtering.

Can I use Kompress-v2-base without installing PyTorch?

Yes. Headroom supports ONNX Runtime as a lightweight alternative that requires only CPU resources and minimal dependencies. Set HEADROOM_KOMPRESS_BACKEND=onnx or allow the auto-selection to default to ONNX when PyTorch is unavailable. The ONNX backend provides faster cold-start times for CPU-only deployments while maintaining the same compression quality.

What compression ratios can I expect with Kompress-v2-base?

According to the Headroom source code benchmarks, Kompress-v2-base typically achieves 80-95% compression of token counts while maintaining high semantic fidelity (F-score ≈ 0.913). The actual ratio depends on text redundancy and the score_threshold parameter; lowering the threshold below the default 0.5 increases compression at the potential cost of information loss.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}