OmniRoute Prompt Compression Techniques: Lite, Caveman, and RTK Explained

OmniRoute implements three distinct prompt compression techniques—Lite, Caveman, and RTK—that can be stacked or used individually to reduce token count before sending requests to upstream LLMs, each optimized for different latency and fidelity requirements.

OmniRoute, an open-source request routing layer for LLM applications, provides a modular prompt compression pipeline designed to minimize token costs and latency. The system supports multiple compression modes that can be layered according to performance requirements. These prompt compression techniques are implemented in the open-sse/services/compression/ directory and orchestrated through a priority-based stacking system.

How the Compression Pipeline Works

The compression system uses a stacked architecture where multiple engines can be applied sequentially. The function selectCompressionMode in open-sse/services/compression/strategySelector.ts examines the user-provided configuration, combo-level overrides, and auto-trigger thresholds to determine which engines to execute.

Engines run in order of their stackPriority values:

• Lite (priority 5)
• RTK (priority 10)
• Caveman (priority 20)
• Ultra (priority 40)

This deterministic ordering allows earlier, faster steps to reduce token count sufficiently so that later, more expensive engines run less frequently.

Lite Compression: Fast Whitespace and Structure Cleanup

The Lite mode provides the fastest compression through pure JavaScript transformations in open-sse/services/compression/lite.ts. This engine is optimized for low-latency scenarios and runs first in the stacked pipeline.

Core techniques include:

• Whitespace collapse: Removes excess newlines and trailing spaces from message content
• System prompt deduplication: Identifies and removes identical system prompts
• Tool-result truncation: Shortens overly long tool result messages
• Duplicate message removal: Eliminates consecutive duplicate messages
• Image URL placeholder replacement: Substitutes inline base64 image URLs with placeholder tokens when the target model does not support vision

Caveman Compression: Semantic Rule-Based Reduction

Caveman mode applies rule-based semantic reduction through the cavemanEngine implemented in open-sse/services/compression/caveman.ts. This technique strips filler content while preserving high-value tokens.

Key characteristics:

• Applies curated "caveman rules" that remove filler words, articles, and polite greetings
• Compresses selected roles (user by default) on a per-message basis
• Can prune across the entire transcript when needed
• Guarantees preservation of code blocks, URLs, identifiers, and other high-value tokens

The rule definitions are loaded from open-sse/services/compression/cavemanRules.ts, which provides JSON-driven language-specific packs (e.g., en/filler.json, en/dedup.json).

RTK Compression: Smart Content Filtering and Truncation

RTK (Rule-Based Truncate-Keep) offers intelligent truncation that preserves semantic structure. The engine resides in open-sse/services/compression/engines/rtk/index.ts and loads language-specific filter packs from JSON files under rules/*/.

Capabilities include:

• Smart truncation: Preserves commands, JSON structures, and terminal output while discarding noise
• Custom renderers: Supports specialized handling for Terraform plans, Git diffs, and structured tables
• Filter packs: Loads configurations like filters/wget.json to keep specific command outputs
• Stack compatibility: Can be combined with other engines in sequences like [rtk, caveman, lite]

Implementation Examples

Applying Lite Compression Manually

To apply Lite compression independently (useful in unit tests or custom middleware):

import { applyLiteCompression } from '@omniroute/open-sse/services/compression/lite.ts';

const rawBody = {
  messages: [
    { role: 'user', content: '   Hello   \n\n\nworld!   ' },
    { role: 'assistant', content: 'Sure.\n\n\n\nHere is the result.' }
  ]
};

const { body, compressed, stats } = applyLiteCompression(rawBody, {
  preserveSystemPrompt: false,
});

Triggering Standard (Caveman) Compression via API

To use the Caveman compression through the API:

await fetch('/api/v1/chat/completions', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    model: 'gpt-4o-mini',
    compressionMode: 'standard',   // resolves to caveman
    messages: [{ role: 'user', content: 'Can you please give me a detailed explanation of …' }]
  })
});

Running RTK with Custom Filters

To execute RTK compression with specific filter packs:

import { applyCompressionAsync } from '@omniroute/open-sse/services/compression/index.ts';
import { loadRtkFilters } from '@omniroute/open-sse/services/compression/engines/rtk/filterLoader.ts';

const rtkFilters = await loadRtkFilters(['filters/wget.json']);
const result = await applyCompressionAsync({
  messages: [{ role: 'assistant', content: '... long shell output ...' }]
}, {
  mode: 'rtk',
  rtkConfig: { filters: rtkFilters }
});

Summary

• OmniRoute provides three distinct prompt compression techniques: Lite, Caveman, and RTK, each serving different optimization needs.
• The Lite engine in lite.ts handles fast whitespace cleanup and deduplication with priority 5.
• Caveman compression in caveman.ts performs semantic rule-based reduction using JSON rule packs while preserving code and URLs.
• RTK compression in engines/rtk/index.ts offers intelligent truncation with language-specific filters and custom renderers for structured outputs.
• The selectCompressionMode function in strategySelector.ts orchestrates these engines in priority order (Lite → RTK → Caveman → Ultra) for layered optimization.

Frequently Asked Questions

What is the difference between Caveman and RTK compression?

Caveman compression focuses on semantic reduction by stripping filler words and polite greetings from natural language while preserving technical tokens like code and URLs. RTK compression specializes in structured content truncation, using filter packs to preserve specific command outputs, JSON structures, and terminal logs while removing noise. Caveman is ideal for conversational text, whereas RTK excels at technical outputs like shell logs and diffs.

How does the compression pipeline decide which engines to run?

The selectCompressionMode function in open-sse/services/compression/strategySelector.ts evaluates the request's compressionMode parameter, combo-level overrides, and auto-trigger thresholds. It returns an ordered list of engines based on their stackPriority values, ensuring deterministic execution from lowest latency (Lite) to highest compression (Caveman/Ultra).

Can I combine multiple compression techniques in a single request?

Yes, OmniRoute supports stacked compression where multiple engines run sequentially. For example, setting compressionMode to use a combo like [rtk, caveman, lite] applies RTK filtering first, then Caveman semantic reduction, and finally Lite cleanup. The order respects each engine's stackPriority regardless of the array order passed in the configuration.

Where is the entry point for applying compression programmatically?

The main entry point is applyCompressionAsync exported from open-sse/services/compression/index.ts. This function wires together the selected engines based on the configuration provided. For Lite-only transformations, you can import applyLiteCompression directly from open-sse/services/compression/lite.ts for use in middleware or testing scenarios.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}