How the `detect()` Function Classifies Payloads and Routes Them to the Correct Compressor in Caveman

The detect() function automatically inspects payload types—distinguishing between JSON, source code, and binary data—and routes each to specialized compressors like elision, toon, or squoosh via a catalog lookup.

The detect() function serves as the intelligent routing layer in the Caveman open-source compression engine (JuliusBrussee/caveman). Located in the core compression pipeline, this function eliminates guesswork by analyzing content signatures before delegation, ensuring that text-heavy JSON payloads never waste cycles on binary compression algorithms, while image buffers bypass text-oriented optimizers entirely.

Payload Type Classification

The classification logic begins with a fundamental type check in packages/engine/src/compression/detect.ts. The function first distinguishes between string payloads and Uint8Array buffers, then applies content-specific heuristics to each category.

Textual Content Detection

For string inputs, detect() examines byte patterns and structural markers without fully parsing the content:

• JSON-like structures — If the payload starts with { or [, the function tags it as structured data and routes it to the elision compressor, which strips whitespace and optimizes key-value redundancy.

• Source code identification — The presence of line breaks combined with keyword tokens (e.g., function, const, import) triggers selection of the toon compressor, a code-aware algorithm that preserves semantic structure while removing comments and unnecessary whitespace.

• Pure prose fallback — Text lacking programming syntax markers or JSON delimiters defaults to the generic elision text compressor, optimized for natural language redundancy.

Binary Content Detection

When detect() encounters a Uint8Array, it inspects the first eight bytes for known magic numbers:

• Image signatures — Pixels starting with PNG (0x89 0x50 0x4E 0x47) or JPEG (0xFF 0xD8 0xFF) headers route the payload to the squoosh image compressor.

• Generic binary fallback — Unrecognized binary patterns default to the zstd compressor, which applies high-performance entropy coding suitable for non-structured byte streams.

Routing Logic and Compression Catalog

After classification, detect() performs a lookup against the compression catalog defined in packages/engine/src/compression/catalog.ts. This catalog maps payload categories to compressor identifiers:

// Conceptual structure based on catalog.ts implementation
const compressionCatalog = {
  'structured-text': 'elision',
  'source-code': 'toon', 
  'image-png': 'squoosh',
  'image-jpeg': 'squoosh',
  'generic-binary': 'zstd'
};

The catalog abstraction allows the detection logic to remain agnostic of specific compressor implementations. When detect() returns a compressor name (e.g., 'toon'), the SDK wraps this identifier in the request payload sent to the engine endpoint POST /sdk/v1/compress.

Engine Delegation

The detect() function operates entirely within the preprocessing phase—it never executes compression algorithms itself. Instead, it prepares a CompressRequest object containing:

1. The original payload buffer
2. The selected compressor identifier from the catalog
3. Content-type metadata for engine logging

The Caveman engine (implemented in native binaries referenced by packages/engine/src/index.ts) receives this request, instantiates the concrete compressor class (located in packages/engine/src/compression/compressors/), and streams the result back as a CompressResult containing the compressed payload and token-savings statistics.

Implementation in the SDK

In packages/sdk/typescript/src/index.ts, the public Cave.compress() method wraps this detection pipeline. Developers invoke compression without specifying algorithms:

import { Cave } from '@caveman/sdk';

// JSON payload → automatically routed to 'elision'
const jsonData = JSON.stringify({ user: 'alice', permissions: ['read', 'write'] });
await Cave.compress(jsonData);

// TypeScript source → routed to 'toon' compressor
const codeSnippet = `
function calculateTax(amount: number, rate: number): number {
  return amount * rate;
}
`;
await Cave.compress(codeSnippet);

// Binary image → routed to 'squoosh'
const imageBuffer = new Uint8Array(await fetch('https://example.com/logo.png').then(r => r.arrayBuffer()));
await Cave.compress(imageBuffer);

The SDK transparently calls detect() during the compress() execution, ensuring optimal compressor selection without manual configuration.

Summary

• detect() inspects payload types at the byte level, distinguishing strings from Uint8Array buffers before applying content heuristics.

• Classification heuristics identify JSON structures (via bracket prefixes), source code (via line breaks and keyword tokens), and image binaries (via magic number signatures).

• Catalog-based routing maps detected categories to compressor identifiers in packages/engine/src/compression/catalog.ts, decoupling detection logic from compression implementations.

• Zero-configuration delegation allows the SDK to automatically select between elision, toon, squoosh, and zstd compressors based on payload content rather than file extensions or MIME types.

Frequently Asked Questions

How does detect() determine if a string contains source code?

The function scans for synchronized line-break patterns paired with programming keywords (such as function, class, or const) and punctuation characteristic of code blocks. When these markers exceed a threshold frequency, detect() categorizes the payload as source code and routes it to the toon compressor, which preserves syntax tree structure during compression.

Can I override the compressor selected by detect()?

According to the current implementation in packages/sdk/typescript/src/index.ts, the compress() method accepts an optional compressor parameter that bypasses automatic detection. When provided (e.g., Cave.compress(payload, { compressor: 'zstd' })), the SDK skips the detect() call and routes directly to the specified engine compressor, though this disables content-specific optimizations.

What happens when detect() encounters an unknown binary format?

Binary payloads lacking recognizable magic numbers (non-image, non-executable buffers) default to the zstd compressor via the catalog fallback entry. This ensures that arbitrary binary data receives efficient entropy compression without risking corruption from format-specific algorithms like squoosh that expect valid image headers.

Where is the compressor catalog defined and how can it be extended?

The compression catalog resides in packages/engine/src/compression/catalog.ts as a static mapping object. To extend support for new payload types (e.g., PDF documents or WebAssembly modules), developers add new detection heuristics to detect() and register corresponding compressor entries in the catalog, then implement the compressor class in packages/engine/src/compression/compressors/.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}