# How SmartCrusher Preserves Data During JSON Compression in Headroom

> Learn how SmartCrusher preserves data during JSON compression using deterministic key mapping, value tokenization, and reversible encoding. Restore your exact original JSON structure.

- Repository: [Tejas Chopra/headroom](https://github.com/chopratejas/headroom)
- Tags: how-to-guide
- Published: 2026-06-16

---

**SmartCrusher preserves data by using deterministic key mapping, value tokenization with embedded lookup tables, and reversible lossless binary encoding, ensuring that every compressed payload can be restored to its exact original JSON structure.**

SmartCrusher is the aggressive compression engine inside the **Headroom** SDK (`chopratejas/headroom`) that reduces JSON payload sizes by 70–90% without information loss. Unlike general-purpose compressors that discard whitespace or apply probabilistic algorithms, SmartCrusher implements a reversible transformation pipeline that stores every mapping decision in a `compressionMetadata` object. This article examines the three-stage preservation strategy implemented in the TypeScript source code and demonstrates how to configure and invoke the compressor in your own applications.

## The Three-Stage Preservation Strategy

SmartCrusher guarantees lossless compression through a tightly coupled three-step process. Each stage transforms the JSON into a more compact representation while capturing the exact rules needed to reverse that transformation.

### Deterministic Key-Shortening with Metadata

The first stage replaces all object keys with short, deterministic tokens (e.g., `a`, `b`, `c`). This mapping is generated once per compression run and stored in a stable lookup table. Because the map is embedded in the `compressionMetadata.keyMap` property, the original keys can be restored during decompression. This approach is implemented in the `SmartCrusher` class within [`src/compression/smart_crusher.ts`](https://github.com/chopratejas/headroom/blob/main/src/compression/smart_crusher.ts), which ensures that the same key always maps to the same token for the duration of the session.

### Value-Level Tokenization and Lookup Tables

Primitive values—strings, numbers, and booleans—undergo tokenization that removes redundant whitespace and trims long strings. For large strings, the algorithm splits them into smaller chunks and stores them in a `compressionMetadata.valueTable`. Each token references its position in this table, allowing the decompressor to expand each token back to the exact original value. This mechanism ensures that no byte of the original data is discarded; it is merely referenced indirectly.

### Reversible Lossless Binary Encoding

After key and value tokenization, the intermediate JSON is converted into a compact binary representation (a packed UTF-8 string). This final encoding step is **lossless**—it never discards bytes that are not accounted for in the metadata. The decoder reads the `compressionMetadata` first, recreates the original key-map and value-table, and then walks the binary representation to rebuild the exact JSON tree that existed before compression.

## Implementation in the Headroom Source Code

The compression logic is exposed through a configuration interface and a programmatic API, both defined in the TypeScript SDK.

### Configuration via SmartCrusherConfig

The behavior of SmartCrusher is controlled by the `SmartCrusherConfig` interface defined in [`src/types/config.ts`](https://github.com/chopratejas/headroom/blob/main/src/types/config.ts) (lines 109–117). Key options include:

- **`enabled`**: Toggles the compression stage.
- **`min_tokens_to_crush`**: Sets a threshold (in tokens) below which compression is skipped.
- **`max_key_depth`**: Limits recursion depth to prevent runaway mapping on deeply nested objects.

These settings ensure that the preservation logic only runs when beneficial and safe.

### The Compression API

You can invoke SmartCrusher manually or through the Headroom client. The `SmartCrusher` class exposes two static methods: `compress()` and `decompress()`.

```typescript
import { SmartCrusher } from "headroom/compression";

const payload = { user: "alice", data: { score: 100, metadata: "very long string..." } };

// Compress returns both the payload and the metadata needed to restore it
const { compressed, metadata } = SmartCrusher.compress(payload);

// Later, restore the exact original object
const original = SmartCrusher.decompress(compressed, metadata);
console.log(JSON.stringify(original) === JSON.stringify(payload)); // true

```

The `compressed` string contains the binary-encoded JSON, while `metadata` holds the `keyMap` and `valueTable` required for reconstruction.

## Integration with the Headroom Pipeline

SmartCrusher is wired into the Headroom processing pipeline in [`plugins/openclaw/src/engine.ts`](https://github.com/chopratejas/headroom/blob/main/plugins/openclaw/src/engine.ts) (lines 149–191). When using the `HeadroomClient`, the compression runs automatically before requests are sent to the LLM API.

```typescript
import { HeadroomClient } from "headroom/sdk";

const client = new HeadroomClient({
  apiKey: process.env.HEADROOM_API_KEY,
  smart_crusher: {
    enabled: true,
    min_tokens_to_crush: 100,
    max_key_depth: 5,
  },
});

// The response is automatically compressed and decompressed
const result = await client.chat({
  messages: [{ role: "user", content: "Explain quantum entanglement." }],
});

console.log(result.compressed); // Compact representation
console.log(result.original);   // Fully restored JSON

```

The `client.chat` method internally invokes the `SmartCrusher` transform, ensuring that the data preserved in the compression stage is available exactly as originally generated after the LLM call completes.

## Summary

- **SmartCrusher** achieves 70–90% size reduction on JSON payloads while maintaining perfect fidelity through three reversible stages: deterministic key-shortening, value tokenization with lookup tables, and lossless binary encoding.
- **Metadata preservation** is enforced by embedding the `keyMap` and `valueTable` in a `compressionMetadata` object that travels with the compressed payload.
- **Source implementation** resides in [`src/compression/smart_crusher.ts`](https://github.com/chopratejas/headroom/blob/main/src/compression/smart_crusher.ts) and is configured via [`src/types/config.ts`](https://github.com/chopratejas/headroom/blob/main/src/types/config.ts), with pipeline integration at [`plugins/openclaw/src/engine.ts`](https://github.com/chopratejas/headroom/blob/main/plugins/openclaw/src/engine.ts).
- **Lossless verification** is guaranteed by the algorithm's design and validated by the test suite in [`sdk/typescript/test/compress.test.ts`](https://github.com/chopratejas/headroom/blob/main/sdk/typescript/test/compress.test.ts).

## Frequently Asked Questions

### Is SmartCrusher lossless or lossy?

SmartCrusher is **lossless**. It uses deterministic mappings and embedded lookup tables to ensure that every compression step can be reversed exactly, returning the original JSON byte-for-byte.

### Where is the compression metadata stored?

The metadata is stored in the `compressionMetadata` object that accompanies the compressed payload. This object contains the `keyMap` for restoring original keys and the `valueTable` for reconstructing exact string values, ensuring the data is never separated from its reconstruction instructions.

### How does SmartCrusher handle deeply nested objects?

SmartCrusher handles nested objects by respecting the `max_key_depth` configuration parameter (default: 5), which limits how deep the key-shortening recursion travels. This prevents excessive mapping table growth while still preserving all nested data structures exactly.

### Can I use SmartCrusher independently of the Headroom client?

Yes. You can import the `SmartCrusher` class directly from `headroom/compression` and call `SmartCrusher.compress()` and `SmartCrusher.decompress()` manually. This allows you to use the compression algorithm in any TypeScript or JavaScript application without initializing the full Headroom client.