How Kompress-base HuggingFace Model Compression Works in Headroom: A Technical Deep Dive

Headroom's Kompress transform automatically downloads a ModernBERT-based model from HuggingFace (chopratejas/kompress-v2-base) and applies intelligent token pruning to long tool outputs and assistant messages through a multi-stage pipeline involving chunking, dual-head neural inference, and cache-controlled retrieval.

The Kompress-base HuggingFace model compression system serves as the core text reduction engine within the Headroom framework. Located in the chopratejas/headroom repository, this implementation processes lengthy generated content by identifying semantically critical tokens while discarding redundant information. The architecture supports multiple execution backends and includes privacy-preserving caching mechanisms to optimize for both latency and computational efficiency.

Model Architecture and Backend Selection

The compression system centers on a dual-head ModernBERT architecture that provides flexible inference strategies based on the execution environment.

The ModernBERT Base and Dual-Head Design

In headroom/transforms/kompress_compressor.py, the HeadroomCompressorModel class (defined in _get_model_class() at lines 83-115) extends the standard ModernBERT architecture with two specialized heads. The first head generates binary keep-masks through get_keep_mask() (lines 118-132), determining whether individual tokens should be retained. The second head employs a 1-D CNN to produce span-importance scores via get_scores() (lines 134-141), ranking token significance for ratio-based compression.

Backend Abstraction: ONNX vs. PyTorch

The system supports multiple execution backends through the KompressBackend literal type, allowing deployment flexibility. The _selected_backend() function (lines 100-117) determines whether to load the ONNX runtime, PyTorch weights, or CoreML variants based on the HEADROOM_KOMPRESS_BACKEND environment variable.

The _load_kompress() orchestrator (lines 122-165) manages lazy downloading of model artifacts from HuggingFace. For ONNX execution, _load_kompress_onnx() (lines 372-442) initializes an _OnnxModel wrapper (lines 148-166) that provides a unified interface matching the PyTorch implementation. PyTorch execution flows through _load_kompress_pytorch() (lines 506-558), which handles device placement and GPU optimization.

The 9-Step Compression Pipeline

The compress() method implements a sophisticated multi-stage pipeline that processes text while respecting the model's 512-token limit.

1. Model Selection and Lazy Loading

The pipeline begins with backend selection and lazy model initialization. When allow_download=False is specified, the system operates in cache-only mode, avoiding network I/O during startup. The _load_modernbert_tokenizer() function (lines 494-503) concurrently loads the answerdotai/ModernBERT-base tokenizer with the same caching semantics.

2. Text Chunking

Input text is segmented into chunks of 350 words (configurable via self.config.chunk_words) to ensure each inference request remains under the 512-token limit. The chunking logic (lines 88-90) calculates chunk_start and chunk_words indices to partition long documents without breaking semantic continuity.

3. Token Encoding

For each chunk, the tokenizer returns input_ids, attention_mask, and a critical word_ids mapping that links token positions back to original word indices. ONNX backends receive NumPy tensors, while PyTorch uses torch.Tensor objects.

4. Dual-Mode Inference

The model executes either get_keep_mask() for binary classification or get_scores() for importance ranking, depending on whether a target_ratio compression parameter is supplied. This selection determines whether the system uses threshold-based or ratio-based token retention.

5. Token-to-Word Mapping

Using the word_ids mapping, token-level predictions collapse to word-level decisions. For threshold-based compression, words with scores exceeding 0.5 (configurable via score_threshold) are retained. For ratio-based compression, the top-k words by score are selected to meet the target ratio.

6. Text Reassembly

Kept word indices are sorted and joined into the compressed output string. The system calculates the compression ratio, and if the reduction exceeds 20% (ratio < 0.8), the result is stored in the CCR (Cache-Controlled Retrieval) system via _store_in_ccr() (lines 1200-1210).

7. Privacy-Preserving Caching

The CCR system uses _kompress_content_signature() (lines 202-237) to generate privacy-preserving signatures for cache keys, ensuring sensitive content remains secure while enabling retrieval of previously computed compressions.

8. Telemetry and Performance Monitoring

The pipeline emits slow-path warnings when inference exceeds 1000ms and optionally records compression metrics through record_compression telemetry hooks (lines 100-110).

9. Batch Optimization

For GPU-enabled PyTorch backends, compress_batch() (lines 95-110) aggregates multiple chunks into single forward passes (batch size ≈ 32), providing approximately 2× speedup for multi-document processing. The _should_use_sequential_fallback() helper (lines 120-125) determines when to batch versus process sequentially.

Configuration and Customization

The KompressConfig class (lines 286-306) exposes granular control over compression behavior:

• model_id: Specify alternative HuggingFace models (e.g., chopratejas/kompress-finance)
• chunk_words: Adjust chunk size (default 350)
• score_threshold: Modify retention aggressiveness (default 0.5)
• enable_ccr: Toggle Cache-Controlled Retrieval
• device: Target specific GPU/CPU placement

Implementation Examples

Basic Single-String Compression

from headroom.transforms.kompress_compressor import KompressCompressor

# Initialize with default kompress-v2-base model

compressor = KompressCompressor()

# Compress lengthy tool output

long_text = "Detailed assistant response exceeding token limits..."
result = compressor.compress(long_text)

print(f"Compressed: {result.compressed}")
print(f"Ratio: {result.compression_ratio}")

Backend Selection and GPU Acceleration

import os

# Configure execution backend

os.environ["HEADROOM_KOMPRESS_BACKEND"] = "pytorch"  # or "onnx", "onnx_coreml"

os.environ["CUDA_VISIBLE_DEVICES"] = "0"

compressor = KompressCompressor()
result = compressor.compress("Very long text...")

Batch Processing for High Throughput

texts = [
    "First assistant message...",
    "Second tool output...",
    "Third document..."
]

compressor = KompressCompressor()
results = compressor.compress_batch(texts)

for r in results:
    print(f"Achieved ratio: {r.compression_ratio}")

Custom Model Configuration

from headroom.transforms.kompress_compressor import KompressConfig, KompressCompressor

cfg = KompressConfig(
    model_id="chopratejas/kompress-finance",
    chunk_words=200,
    score_threshold=0.45,
    enable_ccr=False
)

compressor = KompressCompressor(cfg)

Summary

• Headroom's Kompress transform utilizes a ModernBERT-based dual-head architecture from HuggingFace (chopratejas/kompress-v2-base) to compress long tool outputs and assistant messages.
• The system supports multiple backends (ONNX, PyTorch, CoreML) through environment variable configuration, enabling deployment flexibility across CPU and GPU environments.
• Chunk-based processing (default 350 words) ensures compliance with the 512-token limit while maintaining semantic coherence through token-to-word mapping.
• Dual inference modes provide either binary keep/discard decisions or ratio-based importance scoring, configurable via target_ratio and score_threshold parameters.
• Privacy-preserving CCR caching stores compression results when efficiency exceeds 20%, using content signatures to avoid recomputing similar texts.
• Batch optimization in PyTorch GPU mode delivers approximately 2× throughput improvements for multi-document workloads.

Frequently Asked Questions

How does Headroom choose between ONNX and PyTorch backends?

The _selected_backend() function in headroom/transforms/kompress_compressor.py reads the HEADROOM_KOMPRESS_BACKEND environment variable to determine execution mode. Valid options include "onnx", "pytorch", "onnx_coreml", and "auto". ONNX provides lighter CPU inference, while PyTorch enables GPU acceleration and batch processing capabilities.

What is the maximum text length the Kompress model can handle?

The underlying ModernBERT model has a 512-token limit per inference pass. To handle longer texts, Headroom implements chunking that splits input into 350-word segments (configurable via chunk_words in KompressConfig). This ensures each chunk fits within model constraints while preserving enough context for accurate importance scoring.

How does the compression ratio target work?

When target_ratio is specified, the system uses get_scores() to rank all tokens by importance, then retains the top-k words to achieve the desired compression percentage. Without a target ratio, the binary get_keep_mask() applies a threshold (default 0.5) to determine token retention. The score_threshold parameter in KompressConfig adjusts this aggressiveness.

What is CCR caching and when does it activate?

CCR (Cache-Controlled Retrieval) stores original-compressed text pairs when the compression ratio drops below 0.8 (indicating >20% reduction). The _store_in_ccr() function uses _kompress_content_signature() to generate privacy-preserving cache keys. This prevents recomputation of frequently occurring long outputs while ensuring sensitive data remains secure through signature-based indexing rather than raw text storage.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}