# How the `detect()` Function Classifies Payloads and Routes Them to the Correct Compressor in Caveman

> Discover how Caveman's detect() function classifies payloads by type and routes them to the right compressor. Learn about JSON, source code, and binary data handling.

- Repository: [Julius Brussee/caveman](https://github.com/JuliusBrussee/caveman)
- Tags: internals
- Published: 2026-09-04

---

**The `detect()` function automatically inspects payload types—distinguishing between JSON, source code, and binary data—and routes each to specialized compressors like `elision`, `toon`, or `squoosh` via a catalog lookup.**

The `detect()` function serves as the intelligent routing layer in the **Caveman** open-source compression engine (JuliusBrussee/caveman). Located in the core compression pipeline, this function eliminates guesswork by analyzing content signatures before delegation, ensuring that text-heavy JSON payloads never waste cycles on binary compression algorithms, while image buffers bypass text-oriented optimizers entirely.

## Payload Type Classification

The classification logic begins with a fundamental type check in [`packages/engine/src/compression/detect.ts`](https://github.com/JuliusBrussee/caveman/blob/main/packages/engine/src/compression/detect.ts). The function first distinguishes between **string** payloads and **Uint8Array** buffers, then applies content-specific heuristics to each category.

### Textual Content Detection

For string inputs, `detect()` examines byte patterns and structural markers without fully parsing the content:

- **JSON-like structures** — If the payload starts with `{` or `[`, the function tags it as structured data and routes it to the **`elision`** compressor, which strips whitespace and optimizes key-value redundancy.

- **Source code identification** — The presence of line breaks combined with keyword tokens (e.g., `function`, `const`, `import`) triggers selection of the **`toon`** compressor, a code-aware algorithm that preserves semantic structure while removing comments and unnecessary whitespace.

- **Pure prose fallback** — Text lacking programming syntax markers or JSON delimiters defaults to the generic **`elision`** text compressor, optimized for natural language redundancy.

### Binary Content Detection

When `detect()` encounters a `Uint8Array`, it inspects the first eight bytes for known magic numbers:

- **Image signatures** — Pixels starting with PNG (`0x89 0x50 0x4E 0x47`) or JPEG (`0xFF 0xD8 0xFF`) headers route the payload to the **`squoosh`** image compressor.

- **Generic binary fallback** — Unrecognized binary patterns default to the **`zstd`** compressor, which applies high-performance entropy coding suitable for non-structured byte streams.

## Routing Logic and Compression Catalog

After classification, `detect()` performs a lookup against the **compression catalog** defined in [`packages/engine/src/compression/catalog.ts`](https://github.com/JuliusBrussee/caveman/blob/main/packages/engine/src/compression/catalog.ts). This catalog maps payload categories to compressor identifiers:

```typescript
// Conceptual structure based on catalog.ts implementation
const compressionCatalog = {
  'structured-text': 'elision',
  'source-code': 'toon', 
  'image-png': 'squoosh',
  'image-jpeg': 'squoosh',
  'generic-binary': 'zstd'
};

```

The catalog abstraction allows the detection logic to remain agnostic of specific compressor implementations. When `detect()` returns a compressor name (e.g., `'toon'`), the SDK wraps this identifier in the request payload sent to the engine endpoint `POST /sdk/v1/compress`.

## Engine Delegation

The `detect()` function operates entirely within the preprocessing phase—**it never executes compression algorithms itself**. Instead, it prepares a `CompressRequest` object containing:

1. The original payload buffer
2. The selected compressor identifier from the catalog
3. Content-type metadata for engine logging

The Caveman engine (implemented in native binaries referenced by [`packages/engine/src/index.ts`](https://github.com/JuliusBrussee/caveman/blob/main/packages/engine/src/index.ts)) receives this request, instantiates the concrete compressor class (located in `packages/engine/src/compression/compressors/`), and streams the result back as a `CompressResult` containing the compressed payload and token-savings statistics.

## Implementation in the SDK

In [`packages/sdk/typescript/src/index.ts`](https://github.com/JuliusBrussee/caveman/blob/main/packages/sdk/typescript/src/index.ts), the public `Cave.compress()` method wraps this detection pipeline. Developers invoke compression without specifying algorithms:

```typescript
import { Cave } from '@caveman/sdk';

// JSON payload → automatically routed to 'elision'
const jsonData = JSON.stringify({ user: 'alice', permissions: ['read', 'write'] });
await Cave.compress(jsonData);

// TypeScript source → routed to 'toon' compressor
const codeSnippet = `
function calculateTax(amount: number, rate: number): number {
  return amount * rate;
}
`;
await Cave.compress(codeSnippet);

// Binary image → routed to 'squoosh'
const imageBuffer = new Uint8Array(await fetch('https://example.com/logo.png').then(r => r.arrayBuffer()));
await Cave.compress(imageBuffer);

```

The SDK transparently calls `detect()` during the `compress()` execution, ensuring optimal compressor selection without manual configuration.

## Summary

- **`detect()`** inspects payload types at the byte level, distinguishing strings from `Uint8Array` buffers before applying content heuristics.

- **Classification heuristics** identify JSON structures (via bracket prefixes), source code (via line breaks and keyword tokens), and image binaries (via magic number signatures).

- **Catalog-based routing** maps detected categories to compressor identifiers in [`packages/engine/src/compression/catalog.ts`](https://github.com/JuliusBrussee/caveman/blob/main/packages/engine/src/compression/catalog.ts), decoupling detection logic from compression implementations.

- **Zero-configuration delegation** allows the SDK to automatically select between `elision`, `toon`, `squoosh`, and `zstd` compressors based on payload content rather than file extensions or MIME types.

## Frequently Asked Questions

### How does `detect()` determine if a string contains source code?

The function scans for synchronized line-break patterns paired with programming keywords (such as `function`, `class`, or `const`) and punctuation characteristic of code blocks. When these markers exceed a threshold frequency, `detect()` categorizes the payload as source code and routes it to the `toon` compressor, which preserves syntax tree structure during compression.

### Can I override the compressor selected by `detect()`?

According to the current implementation in [`packages/sdk/typescript/src/index.ts`](https://github.com/JuliusBrussee/caveman/blob/main/packages/sdk/typescript/src/index.ts), the `compress()` method accepts an optional `compressor` parameter that bypasses automatic detection. When provided (e.g., `Cave.compress(payload, { compressor: 'zstd' })`), the SDK skips the `detect()` call and routes directly to the specified engine compressor, though this disables content-specific optimizations.

### What happens when `detect()` encounters an unknown binary format?

Binary payloads lacking recognizable magic numbers (non-image, non-executable buffers) default to the `zstd` compressor via the catalog fallback entry. This ensures that arbitrary binary data receives efficient entropy compression without risking corruption from format-specific algorithms like `squoosh` that expect valid image headers.

### Where is the compressor catalog defined and how can it be extended?

The compression catalog resides in [`packages/engine/src/compression/catalog.ts`](https://github.com/JuliusBrussee/caveman/blob/main/packages/engine/src/compression/catalog.ts) as a static mapping object. To extend support for new payload types (e.g., PDF documents or WebAssembly modules), developers add new detection heuristics to `detect()` and register corresponding compressor entries in the catalog, then implement the compressor class in `packages/engine/src/compression/compressors/`.