# How the Caveman Compression Engine Performs Semantic Condensation in OmniRoute

> Discover how the Caveman compression engine in OmniRoute achieves semantic condensation via rule-based transformation, structure-aware preservation, and whitespace normalization. Learn more now!

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: internals
- Published: 2026-08-03

---

**The Caveman engine in OmniRoute performs semantic condensation through three deterministic stages: rule-based lexical transformation using regex patterns, structure-aware preservation of code blocks and URLs, and whitespace normalization with sentence recapitalization.**

The **Caveman** engine serves as OmniRoute's standard compression mode, shrinking prompt and response payloads while preserving semantic meaning. This TypeScript-based implementation runs entirely in-process, making it suitable for high-throughput request paths that include `caveman` in their compression pipeline. According to the OmniRoute source code, the engine achieves **deterministic, language-agnostic compression** through a carefully orchestrated three-stage workflow.

---

## Rule-Based Lexical Transformation

The core of Caveman compression resides in [`cavemanRules.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/cavemanRules.ts)【CAVEMAN_RULES】, which defines a catalogue of **CavemanRule** objects. Each rule specifies:

- **`pattern`** — A `RegExp` matching target phrasing
- **`replacement`** — Static string or function computing the substitute
- **`context`** — Scope restriction (`"all"`, `"user"`, or `"assistant"`)
- **`category`** — Logical grouping (`filler`, `context`, `structural`, `dedup`, `ultra`)
- **`minIntensity`** — Minimum engine intensity (`lite`, `full`, `ultra`) for activation

Rules are organized into five categories: **Filler Removal**, **Context Condensation**, **Structural Compression**, **Multi-Turn Dedup**, and **Ultra Abbreviations**.

### Rule Selection and Application

The function `getRulesForContext()`【GET_RULES】 filters the rule set based on requested intensity and language pack. Selected rules pass through `applyRulesToText()`【APPLY_RULES】, which implements **keyword pre-filtering** via `shouldAttemptRule()` to skip irrelevant patterns, then records applied rules in `appliedRules` for telemetry.

Example rule from [`cavemanRules.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/cavemanRules.ts):

```typescript
{
  name: "redundant_phrasing",
  pattern: /\b(?:make sure to|be sure to|due to the fact that|the reason is because)\b\s*/gi,
  replacement: (m) => ({
    "make sure to": "ensure ",
    "be sure to": "ensure ",
    "due to the fact that": "because ",
    "the reason is because": "because ",
  })[m.trim().toLowerCase()] ?? "",
  context: "all",
  category: "structural",
  minIntensity: "full",
}

```

---

## Structure-Aware Preservation

Before any rule executes, Caveman identifies **protected structures** using `PROTECTED_STRUCTURE_RE` in [`caveman.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/caveman.ts)【PROTECTED_RE】. This regex detects:

- Code fences and markdown blocks
- URLs and environment variable references
- Custom `preservePatterns` supplied by the user

### Extraction and Restoration Pipeline

The preservation workflow in [`preservation.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/preservation.ts) operates as follows:

1. **`extractPreservedBlocks()`** — Removes protected content, stores in **blocks** array
2. **Rule application** — Runs lexical transformation on stripped text
3. **`restorePreservedBlocks()`** — Re-inserts protected content unchanged
4. **`validateCompression()`** — Ensures preservation integrity; on failure, returns original text with fallback flag

This guarantees **syntax-critical fragments remain untouched** while surrounding prose undergoes condensation.

---

## Cleanup and Recasing

After rule transformation, `cleanupArtifacts()`【CLEANUP_ARTIFACTS】 applies deterministic normalization:

| Function | Effect |
|----------|--------|
| `collapseHorizontalWhitespaceRuns` | Collapses space/tab runs to single space |
| `removeHorizontalWhitespaceBeforePunctuation` | Drops stray spaces before `,.!?:;` |
| `collapseRepeatedSentencePunctuation` | Reduces `!!!` / `???` / `...` to single character |
| `stripLineTrailingHorizontalWhitespace` | Trims line-ending spaces |
| `collapseExcessNewlines` | Limits consecutive newlines to exactly two |
| `recapitalizeSentences` | Uppercases first letter after sentence terminators or line start |

These composable helpers produce **compact, well-formed output** without introducing ambiguity.

---

## Telemetry and Performance Metrics

The entry point `cavemanCompress()` (line 445 of [`caveman.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/caveman.ts)) aggregates comprehensive statistics:

- **Token savings** via `estimateCompressionTokens`
- **Techniques applied** — always includes `"caveman-rules"` plus specific rule names
- **Duration** in milliseconds
- **Warnings/errors** from validation failures

The resulting `CompressionStats` populates OmniRoute's analytics database, feeding **engine breakdown visualizations** and **cost-saving dashboards**.

---

## Usage Examples

### Direct Node.js Integration

```typescript
import { cavemanCompress } from
  "https://raw.githubusercontent.com/diegosouzapw/OmniRoute/release/v3.8.50/open-sse/services/compression/caveman.ts";

const request = {
  messages: [
    {
      role: "user",
      content:
        "Hi there! I was wondering if you could please **explain in detail** why the function appears to be handling the database configuration.\n\n```ts\nconst config = { db: process.env.DB_URL };\n```",
    },
  ],
};

const result = cavemanCompress(request, {
  intensity: "full",
  language: "en",
  autoDetectLanguage: false,
});

console.log("Compressed:", result.compressed);
console.log("New text:", result.body.messages?.[0].content);

```

**Output behavior:** Greetings and verbose phrasing stripped, article "the" before "database" removed, code block preserved intact.

### API Pipeline Configuration

```json
POST /api/v1/chat/completions
{
  "model": "gpt-4o-mini",
  "messages": [
    { "role": "user", "content": "Can you explain why I need to configure the authentication token?" }
  ],
  "pipeline": ["caveman"],
  "compression": { "caveman": { "intensity": "full" } }
}

```

OmniRoute's `applyStackedCompression` service resolves the pipeline and invokes `cavemanCompress` before upstream LLM forwarding.

### Language Packs and Custom Preservation

```typescript
cavemanCompress(request, {
  intensity: "full",
  language: "pt",
  preservePatterns: ["\\bAPI_KEY\\b"],
});

```

Portuguese rules load via `loadAllRulesForLanguage()` from [`ruleLoader.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/ruleLoader.ts), with `API_KEY` literals protected throughout processing.

---

## Key Source Files

| File | Purpose |
|------|---------|
| [`open-sse/services/compression/caveman.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/compression/caveman.ts) | Main compressor, token accounting, orchestration |
| [`open-sse/services/compression/cavemanRules.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/compression/cavemanRules.ts) | Canonical rule catalogue |
| [`open-sse/services/compression/cavemanAdapter.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/compression/cavemanAdapter.ts) | Pipeline engine adapter |
| [`open-sse/services/compression/preservation.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/compression/preservation.ts) | Protected block extraction/reinsertion |
| [`open-sse/services/compression/ruleLoader.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/compression/ruleLoader.ts) | Language pack loader |
| [`open-sse/services/compression/types.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/compression/types.ts) | Type definitions (`CavemanConfig`, `CompressionResult`) |

---

## Summary

- **Caveman semantic condensation** operates through three deterministic stages: lexical transformation, structure preservation, and cleanup normalization
- **Rule-based engine** applies intensity-filtered regex patterns from [`cavemanRules.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/cavemanRules.ts), with keyword pre-filtering for performance
- **Structure-aware preservation** extracts code blocks, URLs, and custom patterns before transformation, validated post-process
- **Cleanup pipeline** collapses whitespace, normalizes punctuation, and recapitalizes sentences for compact output
- **Full telemetry** tracks token savings, applied techniques, and timing via `CompressionStats`
- **Language-agnostic design** supports extensible packs through [`ruleLoader.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/ruleLoader.ts), defaulting to English with auto-detection capability

---

## Frequently Asked Questions

### How does Caveman compression ensure code blocks remain intact?

Caveman uses `extractPreservedBlocks()` from [`preservation.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/preservation.ts) to remove code fences, markdown structures, URLs, and environment references before rule application. These **protected blocks** are stored separately and restored unchanged via `restorePreservedBlocks()`. The `validateCompression()` step verifies integrity, falling back to original text if corruption is detected.

### What intensity levels does the Caveman engine support?

The engine recognizes three intensities: **lite**, **full**, and **ultra**. Each `CavemanRule` specifies `minIntensity` — rules only execute when the requested intensity meets or exceeds this threshold. The `getRulesForContext()` function filters accordingly, allowing progressive aggressiveness from basic filler removal (lite) through aggressive abbreviation (ultra).

### Can Caveman compression handle non-English languages?

Yes. The engine loads language-specific rule packs via `loadAllRulesForLanguage()` in [`ruleLoader.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/ruleLoader.ts). Set `language` in `CavemanConfig` or enable `autoDetectLanguage` for automatic selection. Portuguese (`pt`) packs exist in the repository; additional languages follow the same pattern under `open-sse/services/compression/rules/`.

### What happens if Caveman compression corrupts the output?

The `validateCompression()` function checks post-transformation integrity. On validation failure, the engine **records a fallback event** and returns the original unmodified text. This deterministic safety mechanism ensures prompt reliability even when edge cases bypass protection regexes.