SmartCrusher vs Kompress-Base vs CodeCompressor: Headroom Compression Methods Explained

SmartCrusher is a Rust-backed JSON array compressor that preserves schema integrity through lossless compaction and targeted row removal, while Kompress-Base (implemented as KompressCompressor and sometimes referred to as CodeCompressor) is a ModernBERT-based neural compressor that scores and eliminates low-importance tokens from plain text; both methods integrate with Headroom’s CCR cache and TOIN telemetry but target fundamentally different payload types.

Headroom provides specialized compression pipelines to reduce token counts before LLM ingestion. Based on the source analysis of chopratejas/headroom, the repository implements two distinct compression strategies—SmartCrusher for structured data and Kompress-Base for unstructured text—each with unique implementation details, configuration options, and fallback behaviors.

SmartCrusher: Schema-Aware JSON Array Compression

SmartCrusher targets large JSON arrays such as tool-output tables and logs, maintaining the original schema while aggressively reducing payload size.

Rust Backend and Python Bridge

The core implementation resides in a Rust crate exposed via PyO3 (headroom._core.SmartCrusher). The Python shim located in headroom/transforms/smart_crusher.py forwards data to the Rust layer and handles TOIN telemetry logging and CCR marker management. This architecture ensures deterministic, high-performance processing of tabular data.

Configuration and Compression Strategy

Configuration is controlled through SmartCrusherConfig, which exposes fine-grained thresholds for variance, uniqueness, similarity, and maximum item counts. The algorithm operates in a lossless-first mode, attempting compact encodings such as CSV-schema serialization. If the resulting compression saves less than 30%, it falls back to a lossy row-drop path that emits a CCR marker (<<ccr:HASH …>>) and stores the dropped rows for later retrieval.

CCR Integration and Telemetry

SmartCrusher appends sentinels ({"_ccr_dropped": …}) to the kept array and mirrors the Rust store into the Python compression_store, enabling the /v1/retrieve endpoint to resolve content hashes. After successful non-passthrough compression, the system records the event to TOIN (Tool-Output Identifier Network), learning from array shape patterns.

Fallback Behavior

If the compiled Rust extension is missing, the import fails loudly—compression cannot proceed without the native crate.

Kompress-Base: Neural Text Compression

Kompress-Base (class KompressCompressor) handles long plain-text outputs such as tool narration and assistant messages, using machine learning to identify and remove semantically redundant tokens.

ModernBERT Implementation and Token Scoring

The compressor utilizes a ModernBERT-style transformer model (ONNX INT8 or PyTorch) auto-downloaded from HuggingFace (chopratejas/kompress-base). The full implementation in headroom/transforms/kompress_compressor.py scores each token to generate a keep-mask. Tokens falling below the configured score threshold are dropped; an optional target_ratio parameter forces a fixed retention percentage.

Configuration and Performance

KompressConfig manages model selection, chunk size, score thresholds, device allocation (auto/CPU/GPU), and CCR flags. ONNX CPU execution provides lightweight inference without batch-dimension speedups, while GPU (PyTorch) delivers 2×–2.3× performance improvements for batched workloads.

CCR and Passthrough Fallbacks

When the compression ratio drops below 0.8, Kompress-Base appends a plain-text CCR marker ([N items compressed … Retrieve more: hash=…]) and stores the original text via _store_in_ccr. If model loading fails at runtime, the compressor silently falls back to passthrough mode, returning the original text unchanged. If neither ONNX nor PyTorch are available, the class raises an informative ImportError during instantiation.

Telemetry Integration

After successful compression, the system records the original text signature to TOIN via _kompress_content_signature, enabling content-aware learning.

Key Differences at a Glance

Feature	SmartCrusher	Kompress-Base (KompressCompressor)
Primary Target	Large JSON arrays (tables, logs)	Long plain-text (narration, messages)
Implementation	Rust via PyO3 (headroom._core)	ModernBERT (ONNX INT8 or PyTorch)
Configuration	SmartCrusherConfig (variance, uniqueness thresholds)	KompressConfig (model ID, chunk size, device)
Strategy	Lossless schema encoding → lossy row drop	Token importance scoring → low-score token removal
CCR Marker	JSON sentinel ({"_ccr_dropped": …})	Plain-text marker with hash
Fallback	Hard fail if Rust missing	Passthrough on model failure; ImportError if no ML backend
Performance	Fast, deterministic	GPU batching 2×–2.3× faster than CPU

Implementation Examples

Compressing JSON Arrays with SmartCrusher

from headroom.transforms.smart_crusher import SmartCrusher, SmartCrusherConfig

cfg = SmartCrusherConfig(
    min_items_to_analyze=10,
    max_items_after_crush=12,
    preserve_change_points=False,
)

crusher = SmartCrusher(config=cfg)
json_array = '[{"a": 1, "b": 2}, {"a": 3, "b": 4}, {"a": 5, "b": 6}]'

result = crusher.crush(json_array, query="summarize")
print("Compressed:", result.compressed)
print("Strategy:", result.strategy)   # "lossless" or "smart_crusher_row_drop"

Compressing Plain Text with Kompress-Base

from headroom.transforms.kompress_compressor import KompressCompressor, KompressConfig

compressor = KompressCompressor()
long_text = """Lorem ipsum ... (very long tool output)"""

result = compressor.compress(long_text)
print("Compressed:", result.compressed)
print("Ratio:", result.compression_ratio)   # e.g., 0.45

print("CCR key:", result.cache_key)         # None if ratio ≥ 0.8

Integrated Pipeline Usage

from headroom.transforms.pipeline import Pipeline
from headroom.transforms.smart_crusher import SmartCrusher
from headroom.transforms.kompress_compressor import KompressCompressor

pipeline = Pipeline(
    compressors=[
        SmartCrusher(),
        KompressCompressor(),
    ]
)

messages = [
    {"role": "tool", "content": json_array},      # Routed to SmartCrusher

    {"role": "assistant", "content": long_text}, # Routed to KompressCompressor

]

compressed = pipeline.apply(messages, tokenizer=my_tokenizer)

The routing logic in headroom/transforms/pipeline.py automatically selects the appropriate compressor based on content-type detection.

Summary

• SmartCrusher (headroom/transforms/smart_crusher.py) is a Rust-powered, schema-preserving compressor for JSON arrays that uses lossless compaction first, then drops rows if savings are insufficient, integrating tightly with CCR via JSON sentinels.
• Kompress-Base (headroom/transforms/kompress_compressor.py) is a neural text compressor using ModernBERT token scoring, targeting plain text with configurable thresholds and GPU acceleration, falling back to passthrough on model errors.
• Both methods support TOIN learning for adaptive compression and CCR storage for content retrieval, but require different dependencies (Rust toolchain vs. ONNX/PyTorch).
• CodeCompressor appears to be an alternative or legacy reference to the Kompress-Base implementation, as no separate CodeCompressor class exists in the analyzed source files.

Frequently Asked Questions

When should I use SmartCrusher versus Kompress-Base?

Use SmartCrusher when processing structured JSON arrays such as log entries, database query results, or tabular tool outputs where preserving column schema is critical. Use Kompress-Base for free-form text such as long assistant responses, documentation, or narrative tool output where semantic token importance varies across the document.

What happens if the required backend dependencies are missing?

SmartCrusher will raise an import error and halt execution if the Rust extension (headroom._core) is not compiled. Kompress-Base raises an ImportError during class instantiation if neither ONNX Runtime nor PyTorch are installed; however, if the model fails to load at runtime, it silently falls back to passthrough mode, returning uncompressed text.

How does CCR integration differ between the two compressors?

SmartCrusher embeds structured JSON sentinels ({"_ccr_dropped": …}) within the compressed array and stores dropped rows in the compression_store. Kompress-Base appends a human-readable plain-text marker containing the content hash and stores the full original text via _store_in_ccr. Both enable retrieval via the /v1/retrieve endpoint using the stored hash.

What is the typical performance difference between CPU and GPU for Kompress-Base?

According to the source implementation, Kompress-Base running on GPU via PyTorch achieves 2×–2.3× speedup over CPU inference when processing batched text. The ONNX CPU backend is lightweight and suitable for single-threaded environments but does not provide batch-dimension optimizations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}