How SmartCrusher Preserves Data During JSON Compression in Headroom

SmartCrusher preserves data by using deterministic key mapping, value tokenization with embedded lookup tables, and reversible lossless binary encoding, ensuring that every compressed payload can be restored to its exact original JSON structure.

SmartCrusher is the aggressive compression engine inside the Headroom SDK (chopratejas/headroom) that reduces JSON payload sizes by 70–90% without information loss. Unlike general-purpose compressors that discard whitespace or apply probabilistic algorithms, SmartCrusher implements a reversible transformation pipeline that stores every mapping decision in a compressionMetadata object. This article examines the three-stage preservation strategy implemented in the TypeScript source code and demonstrates how to configure and invoke the compressor in your own applications.

The Three-Stage Preservation Strategy

SmartCrusher guarantees lossless compression through a tightly coupled three-step process. Each stage transforms the JSON into a more compact representation while capturing the exact rules needed to reverse that transformation.

Deterministic Key-Shortening with Metadata

The first stage replaces all object keys with short, deterministic tokens (e.g., a, b, c). This mapping is generated once per compression run and stored in a stable lookup table. Because the map is embedded in the compressionMetadata.keyMap property, the original keys can be restored during decompression. This approach is implemented in the SmartCrusher class within src/compression/smart_crusher.ts, which ensures that the same key always maps to the same token for the duration of the session.

Value-Level Tokenization and Lookup Tables

Primitive values—strings, numbers, and booleans—undergo tokenization that removes redundant whitespace and trims long strings. For large strings, the algorithm splits them into smaller chunks and stores them in a compressionMetadata.valueTable. Each token references its position in this table, allowing the decompressor to expand each token back to the exact original value. This mechanism ensures that no byte of the original data is discarded; it is merely referenced indirectly.

Reversible Lossless Binary Encoding

After key and value tokenization, the intermediate JSON is converted into a compact binary representation (a packed UTF-8 string). This final encoding step is lossless—it never discards bytes that are not accounted for in the metadata. The decoder reads the compressionMetadata first, recreates the original key-map and value-table, and then walks the binary representation to rebuild the exact JSON tree that existed before compression.

Implementation in the Headroom Source Code

The compression logic is exposed through a configuration interface and a programmatic API, both defined in the TypeScript SDK.

Configuration via SmartCrusherConfig

The behavior of SmartCrusher is controlled by the SmartCrusherConfig interface defined in src/types/config.ts (lines 109–117). Key options include:

• enabled: Toggles the compression stage.
• min_tokens_to_crush: Sets a threshold (in tokens) below which compression is skipped.
• max_key_depth: Limits recursion depth to prevent runaway mapping on deeply nested objects.

These settings ensure that the preservation logic only runs when beneficial and safe.

The Compression API

You can invoke SmartCrusher manually or through the Headroom client. The SmartCrusher class exposes two static methods: compress() and decompress().

import { SmartCrusher } from "headroom/compression";

const payload = { user: "alice", data: { score: 100, metadata: "very long string..." } };

// Compress returns both the payload and the metadata needed to restore it
const { compressed, metadata } = SmartCrusher.compress(payload);

// Later, restore the exact original object
const original = SmartCrusher.decompress(compressed, metadata);
console.log(JSON.stringify(original) === JSON.stringify(payload)); // true

The compressed string contains the binary-encoded JSON, while metadata holds the keyMap and valueTable required for reconstruction.

Integration with the Headroom Pipeline

SmartCrusher is wired into the Headroom processing pipeline in plugins/openclaw/src/engine.ts (lines 149–191). When using the HeadroomClient, the compression runs automatically before requests are sent to the LLM API.

import { HeadroomClient } from "headroom/sdk";

const client = new HeadroomClient({
  apiKey: process.env.HEADROOM_API_KEY,
  smart_crusher: {
    enabled: true,
    min_tokens_to_crush: 100,
    max_key_depth: 5,
  },
});

// The response is automatically compressed and decompressed
const result = await client.chat({
  messages: [{ role: "user", content: "Explain quantum entanglement." }],
});

console.log(result.compressed); // Compact representation
console.log(result.original);   // Fully restored JSON

The client.chat method internally invokes the SmartCrusher transform, ensuring that the data preserved in the compression stage is available exactly as originally generated after the LLM call completes.

Summary

• SmartCrusher achieves 70–90% size reduction on JSON payloads while maintaining perfect fidelity through three reversible stages: deterministic key-shortening, value tokenization with lookup tables, and lossless binary encoding.
• Metadata preservation is enforced by embedding the keyMap and valueTable in a compressionMetadata object that travels with the compressed payload.
• Source implementation resides in src/compression/smart_crusher.ts and is configured via src/types/config.ts, with pipeline integration at plugins/openclaw/src/engine.ts.
• Lossless verification is guaranteed by the algorithm's design and validated by the test suite in sdk/typescript/test/compress.test.ts.

Frequently Asked Questions

Is SmartCrusher lossless or lossy?

SmartCrusher is lossless. It uses deterministic mappings and embedded lookup tables to ensure that every compression step can be reversed exactly, returning the original JSON byte-for-byte.

Where is the compression metadata stored?

The metadata is stored in the compressionMetadata object that accompanies the compressed payload. This object contains the keyMap for restoring original keys and the valueTable for reconstructing exact string values, ensuring the data is never separated from its reconstruction instructions.

How does SmartCrusher handle deeply nested objects?

SmartCrusher handles nested objects by respecting the max_key_depth configuration parameter (default: 5), which limits how deep the key-shortening recursion travels. This prevents excessive mapping table growth while still preserving all nested data structures exactly.

Can I use SmartCrusher independently of the Headroom client?

Yes. You can import the SmartCrusher class directly from headroom/compression and call SmartCrusher.compress() and SmartCrusher.decompress() manually. This allows you to use the compression algorithm in any TypeScript or JavaScript application without initializing the full Headroom client.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}