How OmniRoute Implements RTK + Caveman Prompt Compression to Save Tokens

OmniRoute reduces upstream LLM token costs by running requests through a stackable compression pipeline that combines the RTK (Rapid-Tool-Kit) engine with the Caveman engine to strip redundant content while preserving semantic meaning.

The OmniRoute repository implements a sophisticated prompt compression system that intercepts request bodies before they reach expensive LLM providers. By default, every request passes through a sequential pipeline where the RTK engine handles tool-output compression, followed by the Caveman engine applying rule-based text transformations. This dual-stage approach can reduce token counts by over 30% in multi-tool conversations.

The Compression Pipeline Architecture

The pipeline is assembled by strategySelector.ts, which reads the combo configuration and creates an ordered list of engine specifications based on their stackPriority values. Engines marked as stackable: true execute sequentially, with RTK assigned priority 10 and Caveman assigned priority 20, resulting in the default stacked pipeline [rtk, caveman]【/cache/repos/github.com/diegosouzapw/OmniRoute/main/open-sse/services/compression/strategySelector.ts#L138-L150】.

This ordering ensures that command-aware filtering happens first, followed by general text compression rules that operate on the already-reduced content.

RTK Engine: Command-Aware Tool Output Compression

The RTK engine, implemented in rtk/index.ts, specializes in compressing tool execution outputs and code blocks while preserving error lines and semantic structure.

Entry Point and Message Filtering

The applyRtkCompression function serves as the entry point, receiving the request body and a configuration object【/cache/repos/github.com/diegosouzapw/OmniRoute/main/open-sse/services/compression/engines/rtk/index.ts#L25-L38】. For each message, shouldCompressMessage evaluates whether compression applies based on role types, tool-result flags, and the presence of code fences【/cache/repos/github.com/diegosouzapw/OmniRoute/main/open-sse/services/compression/engines/rtk/index.ts#L42-L46】.

Tool Metadata and Command Detection

Before processing, RTK builds a tool-metadata lookup from preceding assistant messages to determine whether a tool result represents a shell command (which receives filtering) or a non-shell operation (which skips filters)【/cache/repos/github.com/diegosouzapw/OmniRoute/main/open-sse/services/compression/engines/rtk/index.ts#L49-L66】. The detectCommandType function classifies the command type, while matchRtkFilter selects appropriate declarative filters unless the output resembles a "document-like read" that requires preservation【/cache/repos/github.com/diegosouzapw/OmniRoute/main/open-sse/services/compression/engines/rtk/index.ts#L40-L45】.

Core Processing Pipeline

The processRtkText function executes a six-step transformation:

1. Command Detection: Identifies the command type using detectCommandType.
2. Filter Matching: Applies declarative filters via matchRtkFilter, unless the output is document-like.
3. Rendering and Code Stripping: Optionally applies applyRenderer and strips fenced code blocks.
4. Deduplication: Removes repeated lines via deduplicateRepeatedLines and optionally groups similar lines via groupSimilarLines.
5. Smart Truncation: Truncates based on effectiveMaxLines intensity while preserving error lines【/cache/repos/github.com/diegosouzapw/OmniRoute/main/open-sse/services/compression/engines/rtk/index.ts#L28-L34】.
6. Raw Output Retention: Calls maybePersistRtkRawOutput to keep raw pointers when compressed text is shorter, enabling later inspection.

The engine reports its work through createCompressionStats, which tracks techniques used (such as rtk-filter and rtk-raw-output-retention) and calculates token estimates via estimateCompressionTokens【/cache/repos/github.com/diegosouzapw/OmniRoute/main/open-sse/services/compression/engines/rtk/index.ts#L44-L48】.

Caveman Engine: Rule-Based Message Compression

Following RTK, the Caveman engine applies language-aware text rules to further reduce token count. The engine is implemented in cavemanAdapter.ts and exported as cavemanEngine with stackPriority: 20【/cache/repos/github.com/diegosouzapw/OmniRoute/main/open-sse/services/compression/engines/cavemanAdapter.ts#L73-L81】.

Configuration and Adaptation

The apply method first adapts the request body via adaptBodyForCompression, then constructs a caveman configuration that merges user-provided settings, step-level overrides, and language-pack configurations【/cache/repos/github.com/diegosouzapw/OmniRoute/main/open-sse/services/compression/engines/cavemanAdapter.ts#L92-L108】. Notably, the wrapper automatically enables the engine ({ enabled: true }) when no explicit flag exists in the user configuration.

Text Transformation Rules

The actual compression delegates to cavemanCompress (located in caveman.ts), which executes text-rules loaded from cavemanRules.ts. These rules perform operations including:

• Removing redundant system-prompt tokens
• Collapsing whitespace and duplicate lines
• Preserving user-specified language packs
• Stripping non-essential formatting

Because Caveman runs after RTK in the stacked pipeline, it operates on already-filtered content, achieving cumulative compression rather than redundant processing.

Tracking Token Savings and Statistics

After each engine completes, createCompressionStats (in stats.ts) records the originalTokens, compressedTokens, list of techniquesUsed, and per-engine breakdowns【/cache/repos/github.com/diegosouzapw/OmniRoute/main/open-sse/services/compression/stats.ts#L1-L30】. The final response includes a stats object where compressedTokens < originalTokens confirms successful token reduction.

This telemetry enables developers to verify that prompt compression is actively reducing costs and to identify which specific techniques (RTK filtering versus Caveman rules) contributed most to the savings.

Configuring the Compression Pipeline

You can interact with the compression system via the API. The following example demonstrates how requests automatically flow through the default RTK → Caveman pipeline:

import { fetch } from 'node-fetch';

const body = {
  model: 'gpt-4o',
  messages: [
    { role: 'user', content: 'Show me the logs of my recent build.' },
    // Tool results and code blocks follow...
  ],
  // Optional: Override the default pipeline order
  // pipeline: ['caveman', 'rtk'],
};

await fetch('https://my.omniroute.dev/api/v1/chat/completions', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify(body),
})
  .then(r => r.json())
  .then(res => {
    console.log('Compressed?', res.stats.compressed);
    console.log('Original tokens:', res.stats.originalTokens);
    console.log('Compressed tokens:', res.stats.compressedTokens);
    console.log('Techniques used:', res.stats.techniquesUsed);
  });

The request automatically passes through strategySelector.ts, which assembles the stacked pipeline [rtk, caveman] based on the default priorities. The response includes detailed statistics showing exactly how many tokens were saved and which compression techniques were applied.

Summary

• Stackable Pipeline: OmniRoute uses strategySelector.ts to assemble a prioritized pipeline where RTK (priority 10) runs before Caveman (priority 20), ensuring command-aware filtering precedes general text compression.
• RTK Engine: Located in rtk/index.ts, this engine uses detectCommandType, matchRtkFilter, and processRtkText to deduplicate lines, truncate low-value content, and preserve error lines in tool outputs.
• Caveman Engine: Implemented in cavemanAdapter.ts and caveman.ts, this engine applies rule-based transformations including whitespace collapse and system-prompt reduction.
• Token Tracking: The createCompressionStats function in stats.ts records original versus compressed token counts and lists specific techniques applied by each engine.
• Cumulative Savings: Running both engines sequentially typically achieves 30%+ token reduction by removing repetitive log lines, fenced code comments, and redundant whitespace before the request reaches the LLM provider.

Frequently Asked Questions

How does the RTK engine decide which messages to compress?

The RTK engine uses the shouldCompressMessage function in rtk/index.ts to evaluate each message based on role type, tool-result flags, and the presence of code fences【/cache/repos/github.com/diegosouzapw/OmniRoute/main/open-sse/services/compression/engines/rtk/index.ts#L42-L46】. It specifically targets tool outputs and code blocks while preserving user instructions and system prompts that require full context.

Can I change the order of the compression engines?

Yes, the pipeline order is configurable through the strategySelector.ts configuration. While the default stacked pipeline is [rtk, caveman] based on their stackPriority values (10 and 20 respectively), you can override this by specifying a custom pipeline array in your request, such as ['caveman', 'rtk']【/cache/repos/github.com/diegosouzapw/OmniRoute/main/open-sse/services/compression/strategySelector.ts#L138-L150】.

What happens if the compressed text ends up longer than the original?

The RTK engine includes a maybePersistRtkRawOutput step that compares compressed versus original length and retains raw output pointers when compression increases token count【/cache/repos/github.com/diegosouzapw/OmniRoute/main/open-sse/services/compression/engines/rtk/index.ts#L44-L48】. Additionally, the final CompressionStats record tracks both values, allowing the system to prefer uncompressed content when it yields better efficiency.

Which specific techniques does the Caveman engine use to reduce tokens?

The Caveman engine applies rules loaded from cavemanRules.ts that remove redundant system-prompt tokens, collapse whitespace and duplicate lines, and enforce language-pack settings【/cache/repos/github.com/diegosouzapw/OmniRoute/main/open-sse/services/compression/engines/cavemanAdapter.ts#L92-L108】. These transformations specifically target the prose and formatting of messages rather than tool-specific output structures handled by RTK.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}