# How SmartCrusher Compresses JSON Tool Outputs in Headroom: A Technical Deep Dive

> Discover how SmartCrusher dramatically compresses JSON tool outputs in Headroom using advanced statistical analysis and pattern classification, slashing data size by up to 95% while ensuring full data retrieval.

- Repository: [Tejas Chopra/headroom](https://github.com/chopratejas/headroom)
- Tags: deep-dive
- Published: 2026-06-14

---

**SmartCrusher reduces JSON tool outputs by 70-95% using three-stage statistical analysis, pattern classification, and specialized compression strategies while preserving the ability to retrieve full data via the CCR cache system.**

The `chopratejas/headroom` repository implements an intelligent compression system that automatically shrinks large JSON payloads from tool calls before they reach the LLM context window. This **SmartCrusher** transform analyzes array structures, detects statistical patterns like time-series or clusters, and applies targeted reduction strategies to minimize token usage without losing critical information.

## The Three-Stage Compression Pipeline

According to the architecture documentation in [`wiki/ARCHITECTURE.md`](https://github.com/chopratejas/headroom/blob/main/wiki/ARCHITECTURE.md), SmartCrusher operates through a distinct pipeline that preserves semantic meaning while dramatically reducing token count.

### Stage 1: Statistical Analysis of JSON Arrays

SmartCrusher first inspects the structure of the JSON object to compute field-level metrics. For each field in the array, it calculates **unique-ratio**, **variance**, and detects **change points** (statistical spikes) that indicate significant events in the data. This analysis occurs in [`headroom/transforms/smart_crusher.py`](https://github.com/chopratejas/headroom/blob/main/headroom/transforms/smart_crusher.py), where the `SmartCrusher` class implements the statistical profiling logic.

### Stage 2: Pattern Classification

Based on the computed statistics, SmartCrusher classifies the data into one of four distinct patterns:

*   **TIME_SERIES** – Numeric fields with timestamps showing variance spikes, indicating metric data with anomalies.
*   **CLUSTER** – Log-like records where most fields remain identical across items, suggesting repetitive event data.
*   **TOP_N** – Ranked lists (such as search results) where only the highest-scoring entries contain relevant information.
*   **SMART_SAMPLE** – Generic sampling strategy applied when no specific pattern matches.

The classification logic follows the flow described in the "Smart Crusher Deep Dive" section of the architecture documentation.

### Stage 3: Pattern-Specific Compression

Once classified, SmartCrusher applies the appropriate reduction strategy:

*   **Time-series** – Retains items around detected change points, summarizes stable intervals, and factors out constant fields.
*   **Cluster** – Groups similar log messages and retains only representative samples per cluster.
*   **Top-N** – Sorts by a score field and retains only the highest-scoring N items.
*   **Smart-sample** – Performs statistically-guided sampling while extracting constant fields into a separate metadata section.

## The CCR System: Lossless Retrieval for Compressed Data

After compression, SmartCrusher injects a **CCR (Compress-Cache-Retrieve)** marker into the LLM context. This marker stores the full original JSON in an in-memory cache managed by [`headroom/cache/compression_store.py`](https://github.com/chopratejas/headroom/blob/main/headroom/cache/compression_store.py). 

If the model requires additional detail from the compressed data, it can invoke the automatically added `headroom_retrieve` tool, implemented in [`headroom/ccr/tool_injection.py`](https://github.com/chopratejas/headroom/blob/main/headroom/ccr/tool_injection.py), which fetches the original content without additional API costs. This ensures that the 70-95% token reduction achieved during initial compression never results in permanent data loss.

## Implementing SmartCrusher in Your Headroom Pipeline

You can invoke SmartCrusher directly or enable it through the HeadroomClient configuration.

### Direct Usage with SmartCrusher Class

```python
from headroom.transforms.smart_crusher import SmartCrusher

# Initialize with target item count

crush = SmartCrusher(max_items_after_crush=20)

# Example: Compress 60 metric records with a CPU spike at index 45

original = [
    {"ts": i, "cpu": 45 + (i == 45) * 50, "host": "prod-1"} for i in range(60)
]

compressed, meta = crush.crush_array(original, tool_name="metrics")
print(compressed)

# {

#   "__headroom_constants": {"host": "prod-1"},

#   "__headroom_summary": "items 0‑44: cpu stable at ~45",

#   "data": [{"ts": 45, "cpu": 95}, {"ts": 46, "cpu": 95}, …]

# }

```

### Automatic Compression via HeadroomClient

```python
from headroom import HeadroomClient, OpenAIProvider
from openai import OpenAI

base = OpenAI(api_key="...")
client = HeadroomClient(original_client=base, provider=OpenAIProvider())

resp = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "user", "content": "Show recent logs"},
        {"role": "assistant", "tool_calls": [{"type": "search", "id": "1"}]},
        {"role": "tool", "content": huge_json_string},   # >5k tokens

    ],
    headroom_mode="optimize"
)

# The response is produced after SmartCrusher has reduced the JSON

```

## Key Source Files and Architecture

| File | Purpose |
|------|---------|
| [`headroom/transforms/smart_crusher.py`](https://github.com/chopratejas/headroom/blob/main/headroom/transforms/smart_crusher.py) | Implements the statistical analysis, pattern detection, and compression logic for JSON arrays. |
| [`wiki/ARCHITECTURE.md`](https://github.com/chopratejas/headroom/blob/main/wiki/ARCHITECTURE.md) | Provides the high-level description of the SmartCrusher pipeline and CCR integration. |
| [`headroom/cache/compression_store.py`](https://github.com/chopratejas/headroom/blob/main/headroom/cache/compression_store.py) | Stores the original JSON so that the LLM can retrieve it later via the injected tool. |
| [`headroom/ccr/tool_injection.py`](https://github.com/chopratejas/headroom/blob/main/headroom/ccr/tool_injection.py) | Inserts the retrieval marker (`headroom_retrieve`) into the LLM context after compression. |

## Summary

*   **SmartCrusher** analyzes JSON tool outputs using statistical metrics including unique-ratio, variance, and change point detection.
*   The system classifies data into **TIME_SERIES**, **CLUSTER**, **TOP_N**, or **SMART_SAMPLE** patterns before applying specialized compression.
*   The **CCR (Compress-Cache-Retrieve)** system stores original data in memory, allowing the LLM to request full details via the `headroom_retrieve` tool.
*   Typical token reduction ranges from **70-95%** while preserving spikes, anomalies, and unique items.
*   Implementation occurs primarily in [`headroom/transforms/smart_crusher.py`](https://github.com/chopratejas/headroom/blob/main/headroom/transforms/smart_crusher.py) with cache management in [`headroom/cache/compression_store.py`](https://github.com/chopratejas/headroom/blob/main/headroom/cache/compression_store.py).

## Frequently Asked Questions

### What percentage of token reduction does SmartCrusher typically achieve?

SmartCrusher typically achieves **70-95% token reduction** on large JSON tool outputs. The exact percentage depends on the data pattern—time-series data with stable intervals compresses heavily, while highly random data achieves more modest reductions through smart sampling.

### How does the LLM retrieve the full original JSON after compression?

The **CCR (Compress-Cache-Retrieve)** marker stores the complete original JSON in an in-memory cache via [`headroom/cache/compression_store.py`](https://github.com/chopratejas/headroom/blob/main/headroom/cache/compression_store.py). The [`headroom/ccr/tool_injection.py`](https://github.com/chopratejas/headroom/blob/main/headroom/ccr/tool_injection.py) module automatically adds a `headroom_retrieve` tool to the LLM context, allowing the model to fetch the full data on demand without additional API costs.

### Which JSON patterns can SmartCrusher detect and optimize?

SmartCrusher recognizes four distinct patterns: **TIME_SERIES** for metric data with temporal spikes, **CLUSTER** for repetitive log entries, **TOP_N** for ranked search results, and **SMART_SAMPLE** for generic arrays. Each pattern triggers a specific compression strategy optimized for that data structure.

### How do I configure the maximum number of items after compression?

Pass the `max_items_after_crush` parameter when initializing the `SmartCrusher` class, as shown in the `crush_array()` method signature. Setting this to approximately 20 items (as in `SmartCrusher(max_items_after_crush=20)`) provides a balance between context window efficiency and information retention.