# SmartCrusher vs Kompress-Base vs CodeCompressor: Headroom Compression Methods Explained

> Understand Headroom's SmartCrusher vs Kompress-Base vs CodeCompressor. Learn how these headroom compression methods preserve data integrity and remove low-importance tokens to optimize payloads.

- Repository: [Tejas Chopra/headroom](https://github.com/chopratejas/headroom)
- Tags: deep-dive
- Published: 2026-06-20

---

**SmartCrusher is a Rust-backed JSON array compressor that preserves schema integrity through lossless compaction and targeted row removal, while Kompress-Base (implemented as `KompressCompressor` and sometimes referred to as CodeCompressor) is a ModernBERT-based neural compressor that scores and eliminates low-importance tokens from plain text; both methods integrate with Headroom’s CCR cache and TOIN telemetry but target fundamentally different payload types.**

Headroom provides specialized compression pipelines to reduce token counts before LLM ingestion. Based on the source analysis of `chopratejas/headroom`, the repository implements two distinct compression strategies—**SmartCrusher** for structured data and **Kompress-Base** for unstructured text—each with unique implementation details, configuration options, and fallback behaviors.

## SmartCrusher: Schema-Aware JSON Array Compression

SmartCrusher targets large JSON arrays such as tool-output tables and logs, maintaining the original schema while aggressively reducing payload size.

### Rust Backend and Python Bridge

The core implementation resides in a Rust crate exposed via PyO3 (`headroom._core.SmartCrusher`). The Python shim located in [`headroom/transforms/smart_crusher.py`](https://github.com/chopratejas/headroom/blob/main/headroom/transforms/smart_crusher.py) forwards data to the Rust layer and handles TOIN telemetry logging and CCR marker management. This architecture ensures deterministic, high-performance processing of tabular data.

### Configuration and Compression Strategy

Configuration is controlled through `SmartCrusherConfig`, which exposes fine-grained thresholds for variance, uniqueness, similarity, and maximum item counts. The algorithm operates in a **lossless-first** mode, attempting compact encodings such as CSV-schema serialization. If the resulting compression saves less than 30%, it falls back to a **lossy row-drop** path that emits a CCR marker (`<<ccr:HASH …>>`) and stores the dropped rows for later retrieval.

### CCR Integration and Telemetry

SmartCrusher appends sentinels (`{"_ccr_dropped": …}`) to the kept array and mirrors the Rust store into the Python `compression_store`, enabling the `/v1/retrieve` endpoint to resolve content hashes. After successful non-passthrough compression, the system records the event to TOIN (Tool-Output Identifier Network), learning from array shape patterns.

### Fallback Behavior

If the compiled Rust extension is missing, the import fails loudly—compression cannot proceed without the native crate.

## Kompress-Base: Neural Text Compression

Kompress-Base (class `KompressCompressor`) handles long plain-text outputs such as tool narration and assistant messages, using machine learning to identify and remove semantically redundant tokens.

### ModernBERT Implementation and Token Scoring

The compressor utilizes a ModernBERT-style transformer model (ONNX INT8 or PyTorch) auto-downloaded from HuggingFace (`chopratejas/kompress-base`). The full implementation in [`headroom/transforms/kompress_compressor.py`](https://github.com/chopratejas/headroom/blob/main/headroom/transforms/kompress_compressor.py) scores each token to generate a keep-mask. Tokens falling below the configured score threshold are dropped; an optional `target_ratio` parameter forces a fixed retention percentage.

### Configuration and Performance

`KompressConfig` manages model selection, chunk size, score thresholds, device allocation (auto/CPU/GPU), and CCR flags. **ONNX CPU** execution provides lightweight inference without batch-dimension speedups, while **GPU (PyTorch)** delivers 2×–2.3× performance improvements for batched workloads.

### CCR and Passthrough Fallbacks

When the compression ratio drops below 0.8, Kompress-Base appends a plain-text CCR marker (`[N items compressed … Retrieve more: hash=…]`) and stores the original text via `_store_in_ccr`. If model loading fails at runtime, the compressor silently falls back to passthrough mode, returning the original text unchanged. If neither ONNX nor PyTorch are available, the class raises an informative `ImportError` during instantiation.

### Telemetry Integration

After successful compression, the system records the original text signature to TOIN via `_kompress_content_signature`, enabling content-aware learning.

## Key Differences at a Glance

| Feature | SmartCrusher | Kompress-Base (KompressCompressor) |
|---|---|---|
| **Primary Target** | Large JSON arrays (tables, logs) | Long plain-text (narration, messages) |
| **Implementation** | Rust via PyO3 (`headroom._core`) | ModernBERT (ONNX INT8 or PyTorch) |
| **Configuration** | `SmartCrusherConfig` (variance, uniqueness thresholds) | `KompressConfig` (model ID, chunk size, device) |
| **Strategy** | Lossless schema encoding → lossy row drop | Token importance scoring → low-score token removal |
| **CCR Marker** | JSON sentinel (`{"_ccr_dropped": …}`) | Plain-text marker with hash |
| **Fallback** | Hard fail if Rust missing | Passthrough on model failure; `ImportError` if no ML backend |
| **Performance** | Fast, deterministic | GPU batching 2×–2.3× faster than CPU |

## Implementation Examples

### Compressing JSON Arrays with SmartCrusher

```python
from headroom.transforms.smart_crusher import SmartCrusher, SmartCrusherConfig

cfg = SmartCrusherConfig(
    min_items_to_analyze=10,
    max_items_after_crush=12,
    preserve_change_points=False,
)

crusher = SmartCrusher(config=cfg)
json_array = '[{"a": 1, "b": 2}, {"a": 3, "b": 4}, {"a": 5, "b": 6}]'

result = crusher.crush(json_array, query="summarize")
print("Compressed:", result.compressed)
print("Strategy:", result.strategy)   # "lossless" or "smart_crusher_row_drop"

```

### Compressing Plain Text with Kompress-Base

```python
from headroom.transforms.kompress_compressor import KompressCompressor, KompressConfig

compressor = KompressCompressor()
long_text = """Lorem ipsum ... (very long tool output)"""

result = compressor.compress(long_text)
print("Compressed:", result.compressed)
print("Ratio:", result.compression_ratio)   # e.g., 0.45

print("CCR key:", result.cache_key)         # None if ratio ≥ 0.8

```

### Integrated Pipeline Usage

```python
from headroom.transforms.pipeline import Pipeline
from headroom.transforms.smart_crusher import SmartCrusher
from headroom.transforms.kompress_compressor import KompressCompressor

pipeline = Pipeline(
    compressors=[
        SmartCrusher(),
        KompressCompressor(),
    ]
)

messages = [
    {"role": "tool", "content": json_array},      # Routed to SmartCrusher

    {"role": "assistant", "content": long_text}, # Routed to KompressCompressor

]

compressed = pipeline.apply(messages, tokenizer=my_tokenizer)

```

The routing logic in [`headroom/transforms/pipeline.py`](https://github.com/chopratejas/headroom/blob/main/headroom/transforms/pipeline.py) automatically selects the appropriate compressor based on content-type detection.

## Summary

- **SmartCrusher** ([`headroom/transforms/smart_crusher.py`](https://github.com/chopratejas/headroom/blob/main/headroom/transforms/smart_crusher.py)) is a Rust-powered, schema-preserving compressor for JSON arrays that uses lossless compaction first, then drops rows if savings are insufficient, integrating tightly with CCR via JSON sentinels.
- **Kompress-Base** ([`headroom/transforms/kompress_compressor.py`](https://github.com/chopratejas/headroom/blob/main/headroom/transforms/kompress_compressor.py)) is a neural text compressor using ModernBERT token scoring, targeting plain text with configurable thresholds and GPU acceleration, falling back to passthrough on model errors.
- Both methods support **TOIN learning** for adaptive compression and **CCR storage** for content retrieval, but require different dependencies (Rust toolchain vs. ONNX/PyTorch).
- **CodeCompressor** appears to be an alternative or legacy reference to the Kompress-Base implementation, as no separate `CodeCompressor` class exists in the analyzed source files.

## Frequently Asked Questions

### When should I use SmartCrusher versus Kompress-Base?

**Use SmartCrusher** when processing structured JSON arrays such as log entries, database query results, or tabular tool outputs where preserving column schema is critical. **Use Kompress-Base** for free-form text such as long assistant responses, documentation, or narrative tool output where semantic token importance varies across the document.

### What happens if the required backend dependencies are missing?

SmartCrusher will raise an import error and halt execution if the Rust extension (`headroom._core`) is not compiled. Kompress-Base raises an `ImportError` during class instantiation if neither ONNX Runtime nor PyTorch are installed; however, if the model fails to load at runtime, it silently falls back to passthrough mode, returning uncompressed text.

### How does CCR integration differ between the two compressors?

SmartCrusher embeds structured JSON sentinels (`{"_ccr_dropped": …}`) within the compressed array and stores dropped rows in the `compression_store`. Kompress-Base appends a human-readable plain-text marker containing the content hash and stores the full original text via `_store_in_ccr`. Both enable retrieval via the `/v1/retrieve` endpoint using the stored hash.

### What is the typical performance difference between CPU and GPU for Kompress-Base?

According to the source implementation, Kompress-Base running on GPU via PyTorch achieves 2×–2.3× speedup over CPU inference when processing batched text. The ONNX CPU backend is lightweight and suitable for single-threaded environments but does not provide batch-dimension optimizations.