How SmartCrusher Compresses JSON Tool Outputs in Headroom: A Technical Deep Dive

SmartCrusher reduces JSON tool outputs by 70-95% using three-stage statistical analysis, pattern classification, and specialized compression strategies while preserving the ability to retrieve full data via the CCR cache system.

The chopratejas/headroom repository implements an intelligent compression system that automatically shrinks large JSON payloads from tool calls before they reach the LLM context window. This SmartCrusher transform analyzes array structures, detects statistical patterns like time-series or clusters, and applies targeted reduction strategies to minimize token usage without losing critical information.

The Three-Stage Compression Pipeline

According to the architecture documentation in wiki/ARCHITECTURE.md, SmartCrusher operates through a distinct pipeline that preserves semantic meaning while dramatically reducing token count.

Stage 1: Statistical Analysis of JSON Arrays

SmartCrusher first inspects the structure of the JSON object to compute field-level metrics. For each field in the array, it calculates unique-ratio, variance, and detects change points (statistical spikes) that indicate significant events in the data. This analysis occurs in headroom/transforms/smart_crusher.py, where the SmartCrusher class implements the statistical profiling logic.

Stage 2: Pattern Classification

Based on the computed statistics, SmartCrusher classifies the data into one of four distinct patterns:

• TIME_SERIES – Numeric fields with timestamps showing variance spikes, indicating metric data with anomalies.
• CLUSTER – Log-like records where most fields remain identical across items, suggesting repetitive event data.
• TOP_N – Ranked lists (such as search results) where only the highest-scoring entries contain relevant information.
• SMART_SAMPLE – Generic sampling strategy applied when no specific pattern matches.

The classification logic follows the flow described in the "Smart Crusher Deep Dive" section of the architecture documentation.

Stage 3: Pattern-Specific Compression

Once classified, SmartCrusher applies the appropriate reduction strategy:

• Time-series – Retains items around detected change points, summarizes stable intervals, and factors out constant fields.
• Cluster – Groups similar log messages and retains only representative samples per cluster.
• Top-N – Sorts by a score field and retains only the highest-scoring N items.
• Smart-sample – Performs statistically-guided sampling while extracting constant fields into a separate metadata section.

The CCR System: Lossless Retrieval for Compressed Data

After compression, SmartCrusher injects a CCR (Compress-Cache-Retrieve) marker into the LLM context. This marker stores the full original JSON in an in-memory cache managed by headroom/cache/compression_store.py.

If the model requires additional detail from the compressed data, it can invoke the automatically added headroom_retrieve tool, implemented in headroom/ccr/tool_injection.py, which fetches the original content without additional API costs. This ensures that the 70-95% token reduction achieved during initial compression never results in permanent data loss.

Implementing SmartCrusher in Your Headroom Pipeline

You can invoke SmartCrusher directly or enable it through the HeadroomClient configuration.

Direct Usage with SmartCrusher Class

from headroom.transforms.smart_crusher import SmartCrusher

# Initialize with target item count

crush = SmartCrusher(max_items_after_crush=20)

# Example: Compress 60 metric records with a CPU spike at index 45

original = [
    {"ts": i, "cpu": 45 + (i == 45) * 50, "host": "prod-1"} for i in range(60)
]

compressed, meta = crush.crush_array(original, tool_name="metrics")
print(compressed)

# {

#   "__headroom_constants": {"host": "prod-1"},

#   "__headroom_summary": "items 0‑44: cpu stable at ~45",

#   "data": [{"ts": 45, "cpu": 95}, {"ts": 46, "cpu": 95}, …]

# }

Automatic Compression via HeadroomClient

from headroom import HeadroomClient, OpenAIProvider
from openai import OpenAI

base = OpenAI(api_key="...")
client = HeadroomClient(original_client=base, provider=OpenAIProvider())

resp = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "user", "content": "Show recent logs"},
        {"role": "assistant", "tool_calls": [{"type": "search", "id": "1"}]},
        {"role": "tool", "content": huge_json_string},   # >5k tokens

    ],
    headroom_mode="optimize"
)

# The response is produced after SmartCrusher has reduced the JSON

Key Source Files and Architecture

File	Purpose
headroom/transforms/smart_crusher.py	Implements the statistical analysis, pattern detection, and compression logic for JSON arrays.
wiki/ARCHITECTURE.md	Provides the high-level description of the SmartCrusher pipeline and CCR integration.
headroom/cache/compression_store.py	Stores the original JSON so that the LLM can retrieve it later via the injected tool.
headroom/ccr/tool_injection.py	Inserts the retrieval marker (headroom_retrieve) into the LLM context after compression.

Summary

• SmartCrusher analyzes JSON tool outputs using statistical metrics including unique-ratio, variance, and change point detection.
• The system classifies data into TIME_SERIES, CLUSTER, TOP_N, or SMART_SAMPLE patterns before applying specialized compression.
• The CCR (Compress-Cache-Retrieve) system stores original data in memory, allowing the LLM to request full details via the headroom_retrieve tool.
• Typical token reduction ranges from 70-95% while preserving spikes, anomalies, and unique items.
• Implementation occurs primarily in headroom/transforms/smart_crusher.py with cache management in headroom/cache/compression_store.py.

Frequently Asked Questions

What percentage of token reduction does SmartCrusher typically achieve?

SmartCrusher typically achieves 70-95% token reduction on large JSON tool outputs. The exact percentage depends on the data pattern—time-series data with stable intervals compresses heavily, while highly random data achieves more modest reductions through smart sampling.

How does the LLM retrieve the full original JSON after compression?

The CCR (Compress-Cache-Retrieve) marker stores the complete original JSON in an in-memory cache via headroom/cache/compression_store.py. The headroom/ccr/tool_injection.py module automatically adds a headroom_retrieve tool to the LLM context, allowing the model to fetch the full data on demand without additional API costs.

Which JSON patterns can SmartCrusher detect and optimize?

SmartCrusher recognizes four distinct patterns: TIME_SERIES for metric data with temporal spikes, CLUSTER for repetitive log entries, TOP_N for ranked search results, and SMART_SAMPLE for generic arrays. Each pattern triggers a specific compression strategy optimized for that data structure.

How do I configure the maximum number of items after compression?

Pass the max_items_after_crush parameter when initializing the SmartCrusher class, as shown in the crush_array() method signature. Setting this to approximately 20 items (as in SmartCrusher(max_items_after_crush=20)) provides a balance between context window efficiency and information retention.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}